Manual

Course Developer

The course developer creates the template for each course with detailed attention to instructional design and pedagogy. A complete template will include:

• a brief overview of the course (for announcements, etc.)
• learning objectives
• videos, course slides
• suggested readings
• preparatory exercises
• assignments
• quizzes
• a study guide that includes a list of potential exam questions

• grading guidelines, and
  

• a template for collecting student assessment data using the ''Student Assessment of Learning Gains''.

The resulting course template is duplicated for each section of the course that is offered and provided to the lead instructor who takes responsibility for the next set of tasks.

Lead Instructor

AST courses will typically have at a lead instructor and at least one associate instructor. The associate instructor is a trainee and part of the lead instructor’s responsibility is to mentor the associate.

Dividing tasks among instructors will necessitate frequent communication to ensure all necessary tasks are completed in a timely fashion. See the sample task tracking sheet for BBST Foundations for an idea of how you might set up a tracking sheet. Details to complete the tasks and a suggested timeline are available in the Nuts and Bolts section of this manual. Sample letters and postings are provided in the Appendix.

The lead instructor is responsible for all aspects of the course, including the quality of all work by associates.

The following tasks are typically done by the lead instructor, although they may be delegated to an experienced associate who will soon become a lead for this course:

• Work with the AST Administrative Assistant to schedule and announce the course and to register participants in the course.
• Check links in the course and fix any that are broken. (Notify the course creator so these can also be fixed in the course template provided to other instructors)
• Determine whether the course objectives are appropriate or require customization for this Section. Normally, if the course objectives are appropriate, the videos, slides and readings will not need revision.
• Determine whether additional videos, slides, or readings are needed and add them as necessary. Be careful not to impose a too heavy workload on course participants.
• Familiarize yourself with exercises, assignments, quiz questions, and the study guide. Revise as necessary. Select the final exam questions from the study guide pool.
• Familiarize yourself with the course timeline. Schedule enough time to complete the necessary communication and feedback tasks your participants will expect. These tasks include welcoming students to the class, enforcing course policies, providing weekly feedback on assignments, and monitoring student progress.
• The software testing field is filled with controversy. Encourage students who use different definitions in their work place to post to the glossary.
• Determine which instructor(s) will be responsible for each task on the timeline.
• Report bugs with Moodle to Tech Support
• Create or improve grading guidelines or rubrics for assignments and essay questions and add these to the Instructor Forum for this course, for reuse by later instructors.

Associate Instructors

The Associate Instructor’s main role is to assist the Lead Instructor in offering a successful experience to course participants. Any task on the Lead Instructor task list may be delegated to an Associate Instructor.
It is important that the associate be familiar with every task that will be given to students:

• Watch lectures and take quizzes at least a week before the students will work through them. Point out any bugs to the lead instructor or repair them directly (with the approval of the Lead).
• Review, and preferably try, the labs and assignments before the students. This will prepare you to respond to student questions and challenges.
• If there is any question about what will be acceptable work on a lab or assignment, discuss that with the Lead Instructor. You will normally use the Instructor’s Forum on Moodle for these discussions so other instructors can benefit from your discussions and decisions.
• Set up the final Student Assessment of Learning Gains and announce its availability.
• Facilitate the final discussion among the instructors of passing/failing student performance.
• Communicate final results to students.
• Answer final questions from students, especially students who failed the course, about their performance.

Tasks for All Instructors

• Introduce yourself to the participants in the Meet & Greet Forum. Be sure that at least one instructor greets each student in the course.
• Offer encouragement to students.
• The software testing field is filled with controversy. Encourage students who use different definitions in their work place to post to the glossary. Encourage students who disagree with quiz questions to post their comments on the quiz forum.
• Encourage students to learn from each other and Internet resources and not rely solely on instructors. (This is important and the course will be unsustainable with volunteer instructors faced with attention-needy students).
• Remind students to participate in all aspects of the course as needed. If a student is inactive for a sustained period of time (probably one section of the course content), check with them to determine their status and consider terminating them from the course entirely. See the Fieldstones project for suggested letters for these purposes.
  
• For each ''quiz'', one instructor must take a leadership role:
• reviewing student answers for error patterns that might indicate a need for improvement to the questions or discussion in the question forum
• posting a summary  discussion of performance on the quiz
• responding, as needed, to student comments
• For each ''preparatory task'', one instructor must take a leadership role:
• skimming student answers before the deadline and posting a few comments (perhaps 1 to 3 across the full class, to show that someone’s paying attention). A typical comment might ask a follow-up question or clarify the question that was asked. It should not address whether the answer provided was good or not.
• Skimming student answers after the deadline, identifying particularly good responses and particularly common or interesting mistakes. These are preparatory exercises and so we don’t expect students to know this material or get the answers “right.” The lecture and reading material will provide most of the content feedback. The answers to be held up as exemplars should be answers that took a good approach to the question, not necessarily the ones that provided the right answer. Identify specifically what you are praising. Your feedback does not have to be comprehensive.
• For each ''assignment'', one instructor must take a leadership role:
• assigning students to groups
• encouraging students to get started
• skimming student answers before the deadline and posting a few comments (perhaps 1 to 3 across the full class, to show that someone’s paying attention). A typical comment might ask a follow-up question or clarify the question that was asked. It should not address whether the answer provided was good or not.
• reminding people of the transition from initial contribution to peer review (feedback on other assignments)
• writing a summary review of the initial work and the  feedback. It is not necessary that the instructor provide feedback on each submission (initial response or peer review). The summary should highlight strengths and weaknesses, complimenting some students by name and never naming the authors of work that is criticized.
• For the ''final exam'', one instructor must take a leadership role:
• encouraging students to get started on the exam cram forum
• making occasional comments on the exam cram forum (none of them provide the answer, some of them ask insight-triggering questions)
• assigning students to peer review
• preparing grading instructions for peer review
• preparing grading notes for final feedback
• writing a summary review for each question that comments on the initial answers and peer review feedback. In practice, different instructors might take different questions.
• Collect the final list of students whose names should be posted on the AST website and forward them to the AST Administrative Assistant.

Administrative Assistant

The Administrative Assistant works with the Lead Instructor to schedule and announce the course and to conduct registration for the course. The following list of tasks should be handled by the Administrative Assistant:

• Post the course announcement to AST lists
• Accept registrations and check membership status.
• Work with members, AST Executives, and instructors to resolve problems that arise in course registration.
• Send welcome notes and log-in instructions to course participants
• Advise the instructors of the list of course participants.
• Submit names of participants who passed the course to be posted on the AST website.

Tech Support

The Tech Support administrator will focus his or her attention on Moodle, server, and compatibility issues. General tasks include:

• Backup courses on a regular basis. Until further notice, backups should not include quiz questions or quiz results.
• Update Moodle, php, and other software as needed
• Act on bug reports related to Moodle that instructors and AST executives report.
• The Moodle community is an excellent resource for troubleshooting advice for problems related to.
• Where possible, make the necessary fixes. Otherwise, make workaround instructions as needed.
• Report back to the person who reported the bug.

Principal Investigators

The AST BBST courses are supported, in part, by funding from the National Science Foundation. Cem Kaner and Rebecca Fiedler are coordinating the NSF-related portions of this work. Dr. Kaner’s efforts focus primarily on the subject matter expertise related to software testing.  Dr. Fiedler works on questions related to instructional design and pedagogy. Either can answer questions about the NSF research efforts.

Participant Responsibilities

The AST BBST courses require active, engaged involvement from all participants to provide a successful learning experience for those enrolled in the course. To that end, enrollees should:

• Expect to spend a minimum of eight hours per week for a total of four weeks (three weeks for the course and one week for the exam) for each course. This reflects time spent watching video lectures, taking quizzes, completing homework of various kinds, and a taking final exam. Specifically, course participants must:
• log in to the course several times a week;
• read and respond to course-related emails in a timely manner;
• participate in the course discussions as outlined by the instructor;
• complete all assigned work on time;
• participate in the peer review process for homework and exams, providing feedback and suggesting grades;
• complete the Student Assessment of Learning Gains, and
• ask for help when needed.
• Agree to three policies:
• The AST Course Site Acceptable Use Policy at [http://www.associationforsoftwaretes...eptableUse.htm]
• The Informed Consent for participating in the Black Box Software Testing Courses, at [http://www.associationforsoftwaretes...medConsent.htm]
• The Intellectual Property Policy for the Black Box Software Testing Courses, at [http://www.associationforsoftwaretes...alProperty.htm]

The Informed Consent and Intellectual Property policy reflects the fact that while we teach these courses, researchers are doing detailed research on how to improve the teaching of software testing. In joining the course, enrollees and instructors alike are agreeing to participate in an experiment (the course).

Getting access to the course

If you don’t already have an account on the AST Moodle server, sign up for one at http://www.associationforsoftwaretesting.org. Ask your co-instructors, if any, to do the same.

Once you have done that, contact XXXXX to arrange access to the course you will be teaching and the Instructors’ Forum for yourself and all co-instructors who will be working with you.

Each account holder must agree to three policies to serve as instructors. Those policies are Informed Consent; Intellectual Property Policy; and the AST Moodle Site Acceptable Use Policy. The Informed Consent Policy is your way to acknowledge and give permission for your involvement in the NSF-sponsored research being conducted via these courses. The other two policies govern instructor and participant interactions with each other. You must agree to these policies in each new course that you join and in the Instructors’ Forum. If you have questions about the terms of either of these policies, please feel free to contact one of the Principal Investigators for the NSF research. They are Dr. Cem Kaner ([mailto:kaner@kaner.com]) and  Dr. Rebecca Fiedler ([mailto:becky@beckyfiedler.com]). If you decide you do not agree to participate in the research, you may not teach the AST courses. However, we invite you to view and use many of these same materials at [http://www.testingeducation.org].

Be sure to request access to the course you will teach and the Instructors’ Forum well before the first day of class. Allow plenty of time to review the course videos, readings, activities, quizzes, and exams and to make changes as necessary. Use the Instructors’ Forum as a place to communicate with your co-instructors and to see what other instructors have done as they’ve taught AST courses.

Familiarize yourself with course objectives and instructional strategies

The ''Common Resources'' section of the course contains a wealth of useful information that will help you teach this course. You can find information about the pedagogy underlying the course, videos on how exams are graded, and explanations of why assignments are designed as they are. Be sure to review these resources as you prepare to teach an AST course.

Review and modify the course

Good teachers prepare their classrooms before students arrive. Commercial instructors make sure technical equipment is working, arrange for lunch, and familiarize themselves with the locations of restrooms and water fountains so they can keep attendees comfortable. Kindergarten teachers make many of the same arrangements and also decorate bulletin boards, set up play areas, and make nametags for the little ones. In their own ways, both instructors are making their classrooms a welcoming and comfortable space for the students they will have. Similarly, online instructors have a set of tasks to prepare their online classrooms for their students.

Once you have access to your course, familiarize yourself with everything in the course. It is particularly important to check all links in the course to be sure none have broken as the Internet continues to grow and evolve.

The AST series of courses is standardized. There will always be some variation—student work will vary from class to class and that will drive the discussions. Final exams will differ from section to section. And an instructor might add another reading or make some other relatively minor change. However, if you wish to make more significant changes, you must discuss this with the course creator.

Late Submission Policy

Instructors should determine their policies for accepting and reviewing date submissions from students and post the policy before the course starts.

Late submissions inconvenience peers and instructors who must give feedback; reduce the quality of feedback as reviewers rush to complete the task; deprive class members of the opportunity to learn from the late submission; and, in the case of group projects, block other group members from moving forward on their tasks. In short courses such as those AST offers, lateness too often leads to a failure to complete the course. Communicating your expectations at the outset will make it easier to enforce your policy when the need arises.

Here are the policies that Kaner uses when he is lead instructor:

• All deadlines are either Wednesday midnight or Saturday midnight. Moodle enforces deadlines at a single time, so if Moodle blocks access to a resource at midnight Eastern time (GMT-5:00), that access is blocked to everyone, including to people in California, even though it is only 9 p.m. Pacific time. Because students work around the world, we adopt GMT as the server clock and enforce deadlines at 8 a.m. GMT. Thus a task to be completed by the end of Wednesday is locked out on Thursday 8 a.m. GMT. Some students in Asia think this means that their deadline is not midnight but 8 a.m. GMT (1:30 p.m. in Chennai). The intended deadline is midnight local time.
• We will sometimes not block access to a task, even after the deadline. For example, some instructors prefer to allow students to look at the quiz after the deadline. The fact that access is not blocked is not an extension. We can tell when students submitted work. Students should not submit anything after an 8 a.m. GMT deadline without the explicit permission of the instructor.
• Orientation exercise: should be completed before the student watches the lecture. Must not be submitted after the deadline.
• Quiz: should be completed in parallel with watching the lecture. Must not be submitted after the deadline.
• A student who skips two orientation exercises or an exercise and a quiz will typically be removed from the course and encouraged to enroll in a subsequent section, when s/he has more time.
• Assignments: must be completed by their submission deadline. Late submissions are not accepted. If this is a group project, all students are expected to participate in a way that makes a meaningful contribution to the group. Late submissions are not accepted.
• Feedback: must be completed by the submission deadline. Late submissions are not accepted. All students are expected to provide meaningful feedback.
• Final exam: must be completed by the submission deadline. Late submissions are not accepted.
• Final exam peer review. must be completed by the submission deadline. Late submissions are not accepted.

Controversy

Some professional testers have found controversy in a few of the activities, definitions, and exam questions. Given the variety of work contexts for software testing professionals, it is no surprise to find disagreements in the field. Embrace these controversies or disagreements as potential learning experiences. The AST BBST courses require students to learn some facts, definitions and concepts, but they do not require students to accept those materials as “correct.” It is important to foster tolerance, understanding, and intellectual engagement with diverse perspectives.

Co-instructors should try to identify areas where they disagree with each other – or the video lecturer or readings—ahead of time. There are many excellent ways to handle such disagreements in ways that consider the learning implications for those who are taking the class. For example, co-instructors might honestly disagree with each other about a topic or technique introduced in the course and decide it will be useful for the students to “see” a debate about the relative merits of each instructor’s position. Properly framed, this debate can be an excellent way for students to learn more about the different perspectives held by their instructors. Improperly framed, it might appear that the instructors don’t have their act together; don’t know the material presented in the course; and don’t get along with each other all that well. Students are unlikely to learn anything useful in the second circumstance.

Scheduling the course

If you are interested in teaching a course, contact XXXXX to schedule a course offering and advertise it to AST members or others as appropriate.

Participants’ names

Registration for AST-specific courses is handled by the AST Administrative Assistant ([mailto:secretary@associationforsoftwaretesting.org]). Approximately one week before your course is scheduled to begin, contact the AST Secretary for a list of those who have signed up for your course. Ensure that the AST Secretary has sent a welcome letter with login information to registrants. (See Appendix B.) Optionally, you may elect to send a welcome letter of your own. In it, be sure to remind registrants to agree to the course policies before the class begins.

If you are not teaching the course under the auspices of AST, work with the sponsoring organization to determine how registration will be handled.

Posting introductions

Just as commercial instructors and Kindergarten teachers arrive early to welcome their students, online instructors should prepare their classrooms to be a welcoming space. Be sure to post a friendly introduction to the “Meet & Greet” forum. In it, include your name, your location, your employment, hobbies or interests, and a digital photo. Be sure to check back regularly to greet students as they join the class and post their introductions. Try to have at least one or two co-instructors greet each student.

Agreeing to course policies

To participate in the AST sponsored courses, students must agree to three course policies described elsewhere in this manual. Several days before the course begins, send a reminder to students who have not yet logged in reminding them of this important obligation and setting a deadline of midnight before the course start date. See Appendix C for a sample message you can use.

Almost immediately after the deadline for course policies passes, remove any enrollees who have not agreed to the policies from the Moodle course. Notify them of this and invite them to return for a course at a later time. The timing is important because students who have agreed to the policies are entitled to expect that the only people who will see their work are people who have agreed to the policies. See Appendix D for a sample notification.

Establishing instructor presence

Having a sense that an instructor is actively participating in an online class helps to prevent online students from feeling isolated. Good online instructors use several strategies to establish and maintain their presence in the classroom without taking on an unmanageable workload. These include:

• Send a welcome letter before the course begins
• Post an instructor greeting in the Meet & Greet forum
• Welcome each student as they introduce themselves to the class to establish and early connection with the students. (Responding to each discussion board posting throughout the course is unmanageable, but responding to the first one is important)
• Set student expectations for the frequency of instructor messages and the timeliness of instructor responses. For example, an instructor might tell students “I log in to the course every day except Sunday” or that students can expect to wait 24-48 hours for a response to an email message.
• Encourage students to communicate in public forums for non-confidential issues. Monitor those forums – responding as necessary. The important advantage over email is that all students benefit from the questions and answers shared there.  
• Use email messages to establish and maintain a predictable rhythm for the class as described in the next section.

Setting the rhythm of the course

The AST courses are designed using a weekly structure. Each week has two parts: Sunday to Wednesday and Thursday to Saturday. Each section of the week includes a weekend day, so students who can only work on the weekend can get the week's work done. Instructors can reinforce this segmentation with well-timed messages.

The student week looks like this:

Sunday through Wednesday – Students work on the first part of the assignment with their initial posting due no later than Wednesday.

Thursday to Saturday – Students review and respond to other students. If assigned, students complete peer review.

The instructor might make some posts during this period, but the goal is to acknowledge and encourage progress, rather than to make a substantial contribution. The course is designed so that much of the learning comes from discussing course content with other students, giving and receiving peer review, and attempting quizzes and assignments. Reading and receiving instructor feedback also contributes to student learning but is also helpful, but not necessarily individualized. An online course is not individual tutoring.

The instructor’s week proceeds as described here:

If possible, log in to the course every day. There is no need to respond to students each day – or to spend a lot of time reading the daily posts. However, a daily login allows instructors to monitor student progress and to intervene if a discussion thread goes off track.

We recommend AST instructors make at least two posts each week to maintain a sense of instructor presence in the minds of the students and to reinforce the rhythm of the class. These posts should go in the Course Announcements forum with all course participants subscribed via email.  

''Highlights of the Coming Week.'' Posted on Saturday or Sunday, this instructor posting gives class participants an overview of the coming week and highlights particularly interesting or particularly difficult areas about the content they will encounter in the coming week. It the workload is a bit heavier than usual, this post is a good opportunity to warn students of that fact. Making this post also gives the instructor a chance to remind students how to correct persistent problems the instructor has noticed in previous weeks or to tell students what aspects of their performance the instructor will be watching closely. An instructor may also share additional resources that may be particularly relevant to student efforts for the coming week.

''Weekly Feedback.'' The weekly feedback post is exactly what it sounds like and should generally be posted on Sunday or Monday. In this post, the instructor can point out broad themes identified over the course of the past week; draw attention to individual contributions that are particularly insightful or noteworthy; or invite further participation in an especially engaging thread. An instructor should use this post to identify and correct any broadly held misunderstandings they’ve noticed (but never in a way that would embarrass individual students). To correct misunderstandings, you can ask students to review a particular chunk of course content; point them to additional resources; or create another resource for them. If at all possible, start and end this post on a positive note.

For ideas on what to say about specific activities and exercises, be sure to review the appropiate section of the Fieldstones Project.

Monitoring participant progress

The timeliness of student assignments is very important to the success of the AST online courses. One of the instructors should monitor student submissions. If a student is late, depending on the late policies announced to the class, the instructor can make contact:

• to encourage the student
• to see if there are special circumstances requiring emotional or logistic support
• to make them aware that they are late and that it matters
• to figure out whether this student has actually dropped out

Personal circumstances and work circumstances can give good cause for a course participant to fall late. When that happens, it’s important an instructor intervenes as soon as possible to salvage the course for that student. Often, the circumstances that led to falling behind will be too complicated to resolve for a short course. If that is the case, the instructor should let the students go with a smile. After all, these might be our students but in AST, they are our colleagues, too. How an instructor terminates students could have long-term interpersonal consequences.

Managing quizzes

In the AST courses, quizzes are used formatively to focus students' attention on important information presented in readings and lectures. Please encourage students to complete the quizzes as they watch the videos as recommended in the course instructions. Some students are dismayed by their performance on the quizzes. Consider sharing the Philosophy of Quizzes (make link to Fieldstone) with your class to help them understand how the quizzes are used and why they are used that way.

Although the quizzes provide automatic feedback for quiz answers, you can invite students to visit the Quiz Discussion Forum where they can discuss and/or challenge the questions from the quizzes. Here's how we suggest you manage that process:

For each quiz, begin a Discussion Topic on the Quiz Discussion Forum. Label your topic "Quiz N Answers" or something clever if you'd prefer.

Your first post in the discussion can let students know that you are starting separate threads for each quiz question. Invite them to comment on any of the threads and let them know that instructor comments won't come until later in the week.

Replying to your first message in the Quiz N discussion forum, begin a separate thread for each quiz question. Edit the Subject line to make it descriptive. Paste one quiz question in the message and highlight the correct answer. Do the same for each quiz question. (This is an excellent job for junior instructors - allowing them to become familiar with course content and freeing up the lead instructor for supervisory tasks.) Students can then reply to the threads they're interested in discussing.

One of the instructors should monitor the unfolding discussion in the quiz discussion forum. Once the quiz closes, one of the instructors can post comments to the discussion forum. Most instructors like to make comments on the questions students answered poorly. To find how the group did on specific quiz questions, go to Quizzes -> Quiz 1 --> Results --> Item Analysis.

Preparing students for the exam

Each AST course has an Exam Cram forum for students to discuss study guide questions with their peers and the instructors. This forum is for students to engage with their peers and the course materials to build their own answers to potential exam questions. The Exam Cram Forum should have been pre-populated with a post titled "Using This Study Guide." Following that, it should have one discussion thread for each question. Each thread should have titles like "Long 1" (referring to the first Long Answer question in the study guide). If the forum wasn't already set up in the Master course, set it up in your course and then either update the Master (if you have the privileges) or contact the course developer. You can find "Using This Study Guide" in the Fieldstones Project.

Instructors should monitor discussions on the Exam Cram forum but refrain from providing answers to questions that might appear on the exam. Several types of feedback to students are useful:

• If a sample exam question has multiple parts, but the student-supplied answer(s) skip some of those parts, the instructor could post a question on the forum, asking "Where is X?" or "Have you addressed all of the parts of this question?"
• Similarly, if the question asks for multiple arguments or examples, and the student answer provides only one, the instructor could post a comment like, "This gives one example; the question asks for 3."
• Sometimes a question uses material from the slides but misses key relevant material from the lecutre or the assigned readings. The instructor could post, "Does this use the necessary source material? What about the readings?"
• If students are leaving a question unanswered and the relevant lecture(s) has passed, the instructor might post a very short note, like "No answer?"
• If an answer is going in the wrong direction, the instructor might query, "Does anyone have a comment on this?"

The instructor should not post a comment on every question. Don't give the impression that you'll catch every big mistake or bad direction. 

Assigning group members for group assignments

There are a variety of ways an instructor might assign class members to groups. The following process describes Cem Kaner's approach. (Please suggest alternate approaches in the [http://www.bbstinstructors.org/forums/ BBST Instructors' Forum])

Start by laying out names in time zones and first languages. The goal is to maximize diversity within the groups.
Next, check participation in the class and group people into three very rough categories:

• very active, leader
• not so active, or very new
• not likely to participate

You can determine the number of groups by ensuring each group has one leader and at least two people who are actively engaged in the course.

To assign tasks to each group, try to select a task based on what would be specifically interesting or challenging for one of the people in the group. While you probably won't know everyone well enough to do this for everyone, you are likely to have good ideas for some of the participants. For low-participations, try to assign a task that tempts them to participate. In other cases, assign tasks to stretch them.

Scott Barber suggests assigning students by distributing the strongest students across the groups. After that, distribute the weakest students across the group. After the strongest and weakest students have been assigned, Scott suggests using the remaining students to make groups diverse with respect to time zones, culture, and gender. If you have several participants from the same company, you should also split them across groups to maximize diversity.

Once the initial group task is completed, you will often have individuals review work by other groups. Try to make sure each group's work receives roughly the same amount of review. Within that, try to assign reviewers with relevant experience to the task. For example, if Susan was a game tester, she can review work by a group that focused on a game testing task (assuming Susan wasn't already in the gaming group). It won't always be obvious where to place a reviewer. In those cases, review assignments must be made arbitrarily with the goal of even coverage. By the time you are assigning reviews, you may have have good reason to believe one of your participants is unlikely to complete the course. If that is the case, disregard those students as you try to balance the group reviews.

Finally, consider sorting your group assignment list by last name before posting to the course. This will make it more difficult for students to see patterns (real or imagined) in how you've assigned them to groups, thus minimizing student dissatisfaction with the resulting assignments.

Final Exam

 A few days before the start of exam week, the lead instructor, with the assistance of the instructor team, should complete the following three tasks:

1. Select questions for the exam from the study guide. The typical exam has 6 questions, some short and some long. Students are much more likely to do well on the short answer questions. The long answer questions reveal much more about students' understanding of the material.
• Choose among the available questions in the exam pool to maximize coverage of the course and maximize diversity among the questions. Some questions overlap in their coverage of the course material and some overlapping may be acceptable, but it is generally undesirable to choose similar questions for the same exam.
• If you are teaching BBST-Foundations or BBST-Bug Advocacy, you are not just managing your own course. You are teaching one of the core BBST prerequisites, laying down expectations for all future courses. Your choice of exam questions will influence study strategies for the rest of the series.
• Especially in Foundations and Bug Advocacy, it is desirable to choose a question that no one has attempted and another question that the students addressed poorly in the Exam Cram forum and that you posted a caution/question about.
• Do not choose a question if instructor comments (feedback on the forum) make the question ambiguous or would lead students to a weak answer.
• Every question in the study guide is fair game for the final exam. If any question is NOT appropriate for the exam, please raise that discussion in BBST Instructors Forum so we can discuss it and consider fixing it or dropping it from the study guide. 
• Please choose questions to have some overlap with questions from previous exams so that we may compare performance across groups. However, this redundancy should be limited. As a general guideline, we suggest an overlap of no more than three questions with either of the previous two final exams. 
2. Create a discussion forum for each participant to post exam responses. Call that discussion forum <LAST NAME> Exam.
3. Determine which two exams each student will review and prepare a chart communicating that information to course participants. Do not post that chart until the period for writing the exam is nearly done.

A few days before the exam starts, remind students of it and give them an idea of how it will work. You can use this Fieldstone if you wish.

At the beginning of final exam week, post the questions you have selected for the exam and instructions for completing the exam to the Moodle site. Figure 2 illustrates sample instructions used in the Foundations course:

FINAL EXAMINATION

• Submit your final exam by midnight on Wednesday. Once you post yours, you will see those that have already been posted.
• Use the provided grading chart to assess the submissions of yourself and two of your peers.
• Follow your instructors' instructions to submit your feedback.
• This final piece is due by midnight on Saturday.

• open the final exam forum with your name;
• create a new discussion for each question;
• copy the question itself as the first post in your discussion; and
• enter your answer as the second post in the discussion

 This is a CLOSED BOOK EXAM. When developing and writing your answer, please do not consult with
 other students, other humans, other non-humans, computers, videos, books, tarot cards, or
 other real or potential sources of information. DO NOT copy and paste an answer you have
 already drafted before the exam started. We are not formally supervising this exam (for example
 by forcing you to write it under the supervision of an exam proctor). Instead, we are trusting
 in your integrity and professionalism that you will not cheat on the exam.

Figure 2: Sample instructions for completing final exams

Instructors should monitor the exam and peer review discussion as it unfolds. However, they should refrain from contributing their own ideas regarding exam content or provide feedback on answers until the review process is completed.

Peer Review

Some students may express discomfort with the exam review task:

• Some students have been socialized to never speak critically of the work of their peers. (What are these people doing as testers?)
• Some people think that all assessment should be done by teachers, because that’s the job of the teacher, and a teacher who doesn’t do it is lazy.
• Some people find the task intimidating and look for any rationale to avoid it.

• People learn a lot from doing peer reviews. Some report that they learned more from reading and critiquing their peers’ answers than they did from studying to prepare their own responses.
• It is completely impractical to have volunteer instructors do the majority of assessment in the course. This project will be unsustainable if the students don’t do most of the assessment.
• The model of expert teacher and inadequate, inexpert, naïve student was a good one back when these students were children. However, they are now adult professionals. The AST BBST courses are not beyond their reach. The point of the class is to help these students develop as professional-level critical thinkers in the field. If they won’t attempt that, they are wasting their time and ours.

Encourage all participants to complete the review task to the best of their ability, pointing out that they may review other people’s exams for additional perspectives, draw on their own experiences, and refer to the course materials to complete this task. Providing critical review in the “safe” environment of the online class is good practice for the software tester’s work world where testers must frequently write reports and critically respond to others. In the work place, the stakes are often much higher than they are in the online classroom.

Student evaluation of course

All AST-sponsored courses will use the Student Assessment of Learning Gains (SALG) for students to report on how the various aspects of the course contributed to their learning. The results from the SALG guide further development of the courses and help explain the strengths and weaknesses to this approach for teaching for the NSF-sponsored research. Please make the SALG information available to your students at the end of the course. Contact XXXXX for the course ID and password if you do not already have it.  

Wrap-up message

The wrap-up message will often be the final official contact you have with a student in your course. In the message, be sure to convey a friendly upbeat tone. The wrap-up message should make the following points:

• A reminder to complete the course evaluation if not already completed
• Notification of Pass/Fail results
• Information on how to opt-in to the AST published list of students who’ve passed the course just completed
• An invitation to sign up for other AST courses in the future
• A warm thank you for participation and contributions throughout the course

Pass/Fail results

Collaborate with co-instructors to determine pass/fail results. Notify each student of their results. Passing students may opt to have their name published on the AST website. Instruct students who wish to have their name published how to advise you of that and set a deadline for their decision.  Invite students who failed the class or failed to complete the class that they are welcome to sign up for the same course another time. Encourage all students to monitor the AST website for new course offerings. Review this document for more details about the Pass/Fail task.

Notifying AST

Once the course is completed, notify AST that the course has finished and provide a list of students who passed the course. Indicate which of the passing students has opted to have their name published on the AST website. This is due no later than two weeks from the end of the course.  WHAT ELSE SHOULD GO HERE?

Instructors’ perceptions of the course

Your opinion counts.  Just as it’s important to gauge the effectiveness of course materials with students, AST Executives and the NSF are interested in instructors’ opinions. Shortly after you notify AST that your course has finished, you will receive a link to the Instructor Perceptions survey. We hope you will find time to offer your feedback – both complimentary and critical – to help us improve the instructor experience of offering the course.