Classroom assessment techniques (CATs) are simple, non-graded, usually anonymous, in-class activities designed to give you and your students useful feedback on the teaching-learning process as it is happening.  The standard reference on CATs is Classroom Assessment Techniques: A Handbook for College Teachers by Thomas A. Angelo and K. Patricia Cross (Jossey-Bass, 1993). This book includes 50 CATs, each described in detail with examples from a variety of disciplines.  You’ve probably heard of a few of these, such as the minute paper, muddiest point exercise, and background knowledge probe.

CATs provide what is known as formative assessment, something I’ve frequently blogged about.  This is assessment of student learning intended to inform future teaching.  Formative assessment is often contrasted with summative assessment, which is performed in order to evaluate student performance.  Summative assessment comes at the end of a learning experience; formative assessment happens as the students are learning.  Feedback from formative assessment can provide instructors with useful insight into what students are understanding, what they are not understanding, and how they might target their teaching to their particular students.

At the recent EDUCAUSE Learning Initiative conference, Charlotte Briggs (University of Illinois-Chicago) and Deborah Keyek-Franssen (University of Colorado-Boulder) presented the results of a very useful study.  They combed through all 50 CATs in the Angelo and Cross book and determined that 23 of them could be used with clickers.  I’ve long thought of classroom response systems as a sort of “technoCAT,” a technology-enhanced classroom assessment technique, since they provide such useful formative assessment of student learning.  Charlotte and I connected via Twitter some time ago, and she had let me know that this analysis of the Angelo and Cross book was in the works.  I was very excited to see her work presented at the ELI meeting!

Charlotte and Deborah’s PowerPoint slides are available, as is their handout listing all 23 CATs that can be performed with clickers.  In their slides, they provide the following example of a CAT that can be used “as is” with clickers.

Background Knowledge Probe: Which of the following is NOT a characteristic of Romanticism?

1. Attention to “the natural”
2. Valued “folk” literature, such as fairy tales
3. Had a strong geographical center in Düsseldorf
4. Referred to “the blue flower” as a central symbol for longing
5. Valued medieval literature and art.

You can imagine asking this kind of clicker question at the start of a unit on Romanticism–or a unit that referenced Romanticism but didn’t focus on it.  If knowledge of Romanticism is important for participating in the discussion that followed, then this question will let instructors know how much time they need to spend reviewing Romanticism at the start of the unit.

The background knowledge probe CAT is one that I referenced in my book since it’s such a common use of clickers.  Where Charlotte and Deborah’s work gets more interesting is in their analysis of the other 49 CATs in the Angelo and Cross book!  For instance, they identify 12 other CATs that can be used “as is” with clickers, including such CATs as approximate analogies, problem recognition tasks, self-confidence surveys, and goal ranking and matching.  They also identify 10 CATs that can be modified to work with clickers.

For example, Angelo and Cross describe the “one sentence summary” CAT, in which students are asked to write a one-sentence summary of a reading or lecture using the WDWWWWHW format: Who Does What to Whom When Where How and Why.  Charlotte and Deborah note that students aren’t able to construct and submit these sentences using clickers.  However, they can be given a potential one-sentence summary and asked to identify its flaws.  The example they share in their slides is this one:

One-Sentence Summary: Find the errors in WDWWWWHW: A grand jury is a panel of judges (who) that decides if someone should be charged with a crime (does what to whom) when the offense might be a felony carrying prison time (when) if federal courts and most state courts (where) by listening to arguments by attorneys from both sides (how) so common sense and community perspectives are part of the criminal justice system (why).

1. Who and Why
2. When and Where
3. How and Why
4. Who and How
5. Does What to Whom and How

This clicker serves much the same purpose as a “traditional” one-sentence summary, in part because it’s a “multiple mark” style of question, asking students to identify not one, but two things wrong with the given summary.  If your clicker system allows actual multiple-mark questions, allowing students to select as many incorrect elements as they wish, the question becomes even more complex–and thus closer in usage to the one-sentence summary described by Angelo and Cross.

Charlotte and Deborah make a few very good points about modifying CATs to work with clickers.  They note that doing so “tends to down-grade the complexity” of the CAT itself.  With the one-sentence summary, for instance, you lose the ability to see what surprising things students might say in their constructed sentences.  However, Charlotte and Deborah point out that class discussion of the clicker question can restore that complexity.  As they write, “Instructors often get the most out of clickers when they are used to prompt discussion,” which is a point I always make when I talk about teaching with clickers.

Here’s one more great example along those lines.  Instead of asking students to write down the “muddiest point” of a lecture at the end of class, Charlotte and Deborah suggest in their handout the following:

List potential topics on slide and include an “other” option. Ask students to indicate the topic with which they had the most difficulty. If a significant proportion of the class selects “other”, probe the class to identify other “muddy” issues.

For other ideas on adapting CATs for use with clickers, take a look at their handout.  The Angelo and Cross CATs book is well-known in some educational circles (not so much in others, unfortunately), and Charlotte and Deborah’s work serves as a nice introduction to teaching with clickers for those familiar with the book.  Conversely, those already teaching with clickers are likely to find a few new ideas for using them as they explore the CATs framework.

Reference: DeBourgh, G. A. (2008). Use of classroom “clickers” to promote acquisition of advanced reasoning skills. Nurse Education in Practice, 8(2), 76-87.

Summary: Gregory DeBourgh provides a useful introduction to using clickers in nursing education, focusing on pedagogical strategies that use clickers to promote critical thinking.  His exploration of critical thinking in the context of nursing education is particularly interesting.  Here’s a sample:

“Reasoning is about using intellectual power to draw conclusions, form judgments, and make inferences based on evidence, education, and experience… The practical significance of acquiring skill in advanced reasoning is to move to the level of predictive clinical reasoning which enables one to anticipate both ideal and likely outcomes given a set of data.”

DeBourgh argues that using classroom response systems to engage students in high-level questions is an effective strategy for developing their critical thinking skills.  He supports this assertion by drawing on the literature on the roles of feedback and questioning in learning and by sharing concrete examples of clicker uses in nursing education.

Included are three sample questions, including a “one-best-answer” question that asks students to identify the likely cause of a particular symptom shown by a patient in a case study.  DeBourgh endorses the use of such questions since they better represent situations students are likely to encounter in clinical settings where they must deal with ambiguity.  He also suggests asking question sequences based around patient cases that “change the focus to add new variables,” noting that doing so also reduces the cognitive load students experience when familiarizing themselves with a new case.

DeBourgh makes a good argument for using clicker questions to model critical thinking skills for students:

“Anticipate likely incorrect responses and prepare ‘talking points’ for discussion as this facilitates ‘thinking on your feet’ and makes more visible to students how an expert uses heuristics, reasoning, and refined problem-solving skills to gain command of a clinical situation.”

Asking questions designed to provide an opportunity for the instructor to model critical thinking is one instance of many DeBourgh describes of crafting questions to meet particular teaching and learning objectives.  In doing so, DeBourgh draws on articles by Ian Beatty on good question design, transferring Ian’s advice to the context of nursing education.

DeBourgh also points out that clicker questions embedded in PowerPoint can be particularly useful in nursing, a field which frequently uses pictures, diagrams, sound clips, and video–media that can also be embedded in PowerPoint.  He also notes that nursing courses often involve discussion of nursing ethics and student opinions about ethical decisions, topics that lend themselves well to clicker questions.

The article also includes results from a study survey about clicker use.  Student responses to rating questions are summarized, and student responses to open-ended questions are presented, as well.

DeBourgh ends with a few challenges involved in teaching with clickers, two of which are particularly significant.  He notes that since instructors can track student performance in a class on a daily basis, expectations for students are raised, which is not popular with all students.  DeBourgh also speaks to the increased expectations for instructors:

“The greatest challenge is the new role for faculty to plan the curriculum and instruction around ‘deep comprehension’ rather than ‘covering content’ using a traditional lecture format.”

Comments: I read this article in advance of my presentation at the Health Professional Educational Research Symposium earlier in the month, and I was particularly impressed with Gregory DeBourgh’s eloquence in describing critical thinking in the context of nursing education and in describing ways that clicker pedagogies can foster those critical thinking skills.

As I’ve tried to capture above, DeBourgh describes a variety of ways of using clickers in nursing education, and he included one approach that was entirely new to me, one inspired by the 50-50 option in the television game show Who Wants to Be a Millionaire? DeBourgh suggests that before the correct answer to a clicker question is revealed to students, an instructor might ask the students which answer choices should be eliminated.  It’s a little unclear how DeBourgh implements this, but I can even imagine setting up a multiple-mark question with four answer choices, then asking students not to select the one correct answer but to select two incorrect answers.  This would offer a nice change of pace in question format and would help students focus on more than just the correct answer.  It’s often useful for students to consider why some answer choices are plausible on the surface but actually incorrect.

Hopefully it’s also above that DeBourgh puts an emphasis on teaching with case studies (multimedia case studies, at that) in his article.  I understand that case study methods are perhaps more common in nursing than they are in other disciplines, and I appreciated reading this article as a way to better understand why that was the case.  DeBourgh’s comments about using clickers for discussing ethics also helped me better understand the disciplinary context here.

If you’re a nursing educator, please share a thought or two about using clickers in your field in the comments section!

Update: Greg DeBourgh emailed me and clarified his 50-50 technique.  Here’s what he said:

I display the potential four-answers to a given question, then before the students “vote” with their clickers, I ask for a volunteer or select a student at random (my clicker system has this feature) and ask the student to eliminate 2 of the 4 potential answers and to explain why they are eliminating these two. This speaking out loud of their rationale for eliminating two of the potential answers that are not related to the question strengthens the students’ reasoning skills. They actually get quite good at it. If the student I called upon to answer hesitates or is reluctant to speak, I invite them to choose a “consultant” in the room to help them out. I hope this clarifies a bit for you.

I asked Greg what he does if the student eliminates the correct answer.  Here’s his response:

If the student eliminates one of the correct answers, it is still learning, and so I ask “does everyone agree with the 50/50 elimination?” If someone objects, I ask for their rationale. If no one objects, I just let the process go and during the “reveal and rationale” we talk about why each answer is incorrect or correct.

Thanks, Greg, for this clarification, and for this great use of clickers.

I’ve recently become a fan of the “mark all that apply” type of question my classroom response system facilitates.  I call these “multiple mark” questions in my book.  Here’s one I used in the linear algebra course I’m teaching this fall.

This question is adapted from one of the questions written by Project MathQuest out of Carroll College.  Their version of the question wasn’t a multiple mark question.  Instead, it included a fifth option, “More than one of the above are possible.”  While that option makes the question more interesting and more challenging for the student, it also yields inconclusive data about student learning since students submitting that response may have different ideas about which of the four options are possible.  That’s not all bad, of course.  Given my interview with Kelly Cline, one of the PIs for Project MathQuest, I can imagine Kelly leveraging that ambiguity into a productive classwide discussion of the question.

However, I decided to turn this question into a multiple mark question by adding the instruction, “Mark all that are possible.”  As you can see from the results, 20 of the 20 students present that day indicated that option 3 was possible, 19 of the students indicated options 1 and 2 were possible, and 14 of the students felt that option 4 was possible.  This was very useful feedback for me, since I could quickly tell that the class was in agreement on options 1 through 3, but option 4 deserved some further discussion.

I’ll admit, however, that I got a little tripped up on my own question logic here.  As it turns out, all four options are possible, which was not my intent when I included this question in my lesson plan.  Option 2 is only possible if the third plane intersects the two overlapping planes and option 4 is only possible if the three planes are parallel because they are in fact the same plane.  The way I’ve worded the question, these wrinkles aren’t addressed, making all four options possible.  As a result, the question doesn’t do a great job at uncovering student understanding of these wrinkles.

Here’s the question I should have asked instead:

Suppose you have a system of 3 linear equations in 3 variables.  Which of the following conditions would guarantee that the system has an infinite number of solutions?  Mark all that apply.

1. All three equations represent the same plane.
2. Two of the equations represent the same plane.
3. The three equations represent planes that intersect along a line.
4. The three equations present parallel planes.

With this question, only options 1 and 3 are correct.  With option 2, it could be that the third plane is parallel to but distinct from the two overlapping planes, yielding no solutions instead of infinitely many solutions.  With option 4, it could be that the three planes are not the same plane, again yielding no solutions instead of infinitely many solutions.  This wording of the question puts the special cases in their proper places.

Have you used multiple mark questions?  Do you find them more difficult to write?  How do your students respond to them?