Continuing my reports from the contributed paper session on teaching with clickers I helped coordinate at the Joint Mathematics Meetings back in January…

“Using Personal Response Systems (Clickers) in Liberal Arts Mathematics Courses to Support a Lecture Format,” Janet A. White, Millersville University of Pennsylvania [Slides]

Just like Jean McGivney-Burelle and Kimberly Burch, Janet White shared her experiences teaching with clickers in a “liberal arts” mathematics course taken by non-majors. Unlike Jean and Kimberly, who teach relatively small sections of this kind of course, Janet teaches in a large lecture hall with 75 students per section. Janet had used clickers in courses for pre-service math teachers in the past and found them useful, so when it was her turn to teach this larger course, she decided to use them again. A classroom response system was hardly the only technology Janet used in this course: She also had students complete online homework and quizzes and she annotated her PowerPoint lecture slides using an Interwrite Mobi.

Janet used clickers on a daily basis in her course, usually either to assess students’ prior knowledge or to assess their understanding of a topic taught during lecture.  Her questions came from a bank of multiple-choice questions provided by her textbook publisher.  She counted the clicker questions as part of her students’ participation grades, but in a low-stakes manner.  Given her use of the questions as well as the source of the questions, many were on the lower levels of Bloom’s taxonomy, aimed at recall and application of procedural knowledge.  She shared an example of a prior knowledge question that asked students to find the measure of an angle that complements a 36 degree angle.  A slightly harder question aimed at assessment of something taught during the course asked students to identify the cut edge in a given graph (or to assert that the graph had no cut edge).

Student survey results indicated that 85% of Janet’s students who used clickers regularly liked using them, and 71% said that using clickers helped them learn the material.  Students who used clickers regularly during the course ended up with higher grades in the course than students who didn’t, but, of course, that can’t necessarily be attributed to the use of the clickers.  (And since clicker questions were factored in the course grade, students who participated more frequently in clicker questions would almost certainly have higher grades in the course anyway.)

Student comments about the clickers were generally positive.  My favorite one was, “I liked getting the wrong answer anonymously.”  Other comments addressed the usual points that students like about clickers: They liked the interactivity, they liked discussing questions with classmates, they liked seeing where they stood relative to their peers, and they liked the feedback on their own learning the clicker questions provided.  The only significant negative aspect for the students was the cost, about $50 in Janet’s case.

Janet found that having students discuss clicker questions in small groups led to very engaged students, even in the large auditorium environment.  In the future, she plans to write more of her own questions, instead of relying on ones from the textbook’s question bank.  She hopes to write more difficult questions that will generate even more engaged discussion during class.  She’s also hoping to find ways to reduce the technology cost to the students, either by selecting a different vendor or facilitating the resale of clickers after each semester to students taking the course the next semester.

Also, Janet mentioned that the earth science faculty at Millersville are big users of clickers.  Earth science instructors looking for advice on using clickers might want to investigate!

Image: “Recursive Daisy” by Flickr user gadl / Creative Commons licensed

Classroom assessment techniques (CATs) are simple, non-graded, usually anonymous, in-class activities designed to give you and your students useful feedback on the teaching-learning process as it is happening.  The standard reference on CATs is Classroom Assessment Techniques: A Handbook for College Teachers by Thomas A. Angelo and K. Patricia Cross (Jossey-Bass, 1993). This book includes 50 CATs, each described in detail with examples from a variety of disciplines.  You’ve probably heard of a few of these, such as the minute paper, muddiest point exercise, and background knowledge probe.

CATs provide what is known as formative assessment, something I’ve frequently blogged about.  This is assessment of student learning intended to inform future teaching.  Formative assessment is often contrasted with summative assessment, which is performed in order to evaluate student performance.  Summative assessment comes at the end of a learning experience; formative assessment happens as the students are learning.  Feedback from formative assessment can provide instructors with useful insight into what students are understanding, what they are not understanding, and how they might target their teaching to their particular students.

At the recent EDUCAUSE Learning Initiative conference, Charlotte Briggs (University of Illinois-Chicago) and Deborah Keyek-Franssen (University of Colorado-Boulder) presented the results of a very useful study.  They combed through all 50 CATs in the Angelo and Cross book and determined that 23 of them could be used with clickers.  I’ve long thought of classroom response systems as a sort of “technoCAT,” a technology-enhanced classroom assessment technique, since they provide such useful formative assessment of student learning.  Charlotte and I connected via Twitter some time ago, and she had let me know that this analysis of the Angelo and Cross book was in the works.  I was very excited to see her work presented at the ELI meeting!

Charlotte and Deborah’s PowerPoint slides are available, as is their handout listing all 23 CATs that can be performed with clickers.  In their slides, they provide the following example of a CAT that can be used “as is” with clickers.

Background Knowledge Probe: Which of the following is NOT a characteristic of Romanticism?

1. Attention to “the natural”
2. Valued “folk” literature, such as fairy tales
3. Had a strong geographical center in Düsseldorf
4. Referred to “the blue flower” as a central symbol for longing
5. Valued medieval literature and art.

You can imagine asking this kind of clicker question at the start of a unit on Romanticism–or a unit that referenced Romanticism but didn’t focus on it.  If knowledge of Romanticism is important for participating in the discussion that followed, then this question will let instructors know how much time they need to spend reviewing Romanticism at the start of the unit.

The background knowledge probe CAT is one that I referenced in my book since it’s such a common use of clickers.  Where Charlotte and Deborah’s work gets more interesting is in their analysis of the other 49 CATs in the Angelo and Cross book!  For instance, they identify 12 other CATs that can be used “as is” with clickers, including such CATs as approximate analogies, problem recognition tasks, self-confidence surveys, and goal ranking and matching.  They also identify 10 CATs that can be modified to work with clickers.

For example, Angelo and Cross describe the “one sentence summary” CAT, in which students are asked to write a one-sentence summary of a reading or lecture using the WDWWWWHW format: Who Does What to Whom When Where How and Why.  Charlotte and Deborah note that students aren’t able to construct and submit these sentences using clickers.  However, they can be given a potential one-sentence summary and asked to identify its flaws.  The example they share in their slides is this one:

One-Sentence Summary: Find the errors in WDWWWWHW: A grand jury is a panel of judges (who) that decides if someone should be charged with a crime (does what to whom) when the offense might be a felony carrying prison time (when) if federal courts and most state courts (where) by listening to arguments by attorneys from both sides (how) so common sense and community perspectives are part of the criminal justice system (why).

1. Who and Why
2. When and Where
3. How and Why
4. Who and How
5. Does What to Whom and How

This clicker serves much the same purpose as a “traditional” one-sentence summary, in part because it’s a “multiple mark” style of question, asking students to identify not one, but two things wrong with the given summary.  If your clicker system allows actual multiple-mark questions, allowing students to select as many incorrect elements as they wish, the question becomes even more complex–and thus closer in usage to the one-sentence summary described by Angelo and Cross.

Charlotte and Deborah make a few very good points about modifying CATs to work with clickers.  They note that doing so “tends to down-grade the complexity” of the CAT itself.  With the one-sentence summary, for instance, you lose the ability to see what surprising things students might say in their constructed sentences.  However, Charlotte and Deborah point out that class discussion of the clicker question can restore that complexity.  As they write, “Instructors often get the most out of clickers when they are used to prompt discussion,” which is a point I always make when I talk about teaching with clickers.

Here’s one more great example along those lines.  Instead of asking students to write down the “muddiest point” of a lecture at the end of class, Charlotte and Deborah suggest in their handout the following:

List potential topics on slide and include an “other” option. Ask students to indicate the topic with which they had the most difficulty. If a significant proportion of the class selects “other”, probe the class to identify other “muddy” issues.

For other ideas on adapting CATs for use with clickers, take a look at their handout.  The Angelo and Cross CATs book is well-known in some educational circles (not so much in others, unfortunately), and Charlotte and Deborah’s work serves as a nice introduction to teaching with clickers for those familiar with the book.  Conversely, those already teaching with clickers are likely to find a few new ideas for using them as they explore the CATs framework.

Here are a few interesting ideas shared during the first set of talks about teaching with clickers at the Joint Mathematics Meetings earlier today.  (This post’s title inspired by the following books seen at the exhibits: Calculus Gems, Mathematical Diamonds, More Mathematical Morsels, and Biscuits of Number Theory.)

Kathryn Ernie (University of Wisconsin-River Falls) shared ways that she and her colleagues use clickers in their college algebra courses.  One use that she mentioned was to warm students up before “traditional” in-class quizzes.  By asking students a clicker question or two, then discussing those questions prior to a graded quiz, the students are able to approach the quiz with a little more confidence.

Ben Galluzzo (Shippensburg University) also talked about graded quizzes.  However, in his case, clicker questions aren’t warm-ups for the quiz; the clicker questions are the quiz.  Clickers allow Ben to turn his quizzes into learning experiences for his students.  After each quiz question, he discusses the question with the class before moving on to the next question.  This can work particularly well when he has more than one quiz question of the same type.  Students who miss the first one can learn from the discussion of that question and apply what they’ve learned to the subsequent question.  Students like this because they appreciate the chance to redeem themselves.

Aprillya Lanz (Virginia Military Institute) mentioned that teaching with clickers help students stay awake and engaged during class.  This is particularly important for her since many of her students are freshmen (“rats” as they’re called at VMI) who are required to participate in all kinds of strenuous physical activities, particularly on Sunday nights.  This can make for some very sleepy students in Monday morning classes.

Daniel Joseph (also VMI) described a problem that those who teach calculus often see: His calculus students often struggle because of pre-calculus misconceptions.  They can’t tackle the calculus because they get tripped up by algebra and other pre-calc topics.  He described several methods he’s tried to combat this, but he finds that the students’ over-confidence trips keeps these methods from working.  The students say they “know” all the pre-calculus material because they’ve studied it in the past.  Daniel appreciates how clickers provide his students with frequent evidence that they don’t know it as well as they think they do.

Daniel shared one approach to attacking this problem–using clicker questions in a pre-semester pre-calculus course for incoming freshmen.  He’s interested in hearing ideas for hitting this issue, with or without clickers, in the calculus course itself.  Any ideas?

(By the way, Daniel used the phrase “attack this problem” at least five times in his presentation.  Given that he teaches at a military institute, I figured that was language that comes naturally to him.  Thus my use of the verbs tackle, combat, attack, and hit above!)

I couple of weeks ago on this blog, I shared a tweet by Colin Morris, a student at Kent State University in Ohio.  His comment (via Twitter) was, “44% OF MY U.S. HISTORY CLASS THINKS WATERBOARDING IS A SURFING TERM. I take back everything I’ve said about these ‘clickers’ being useless.”  After I shared this tweet on my blog, a few interesting things happened.

1. Colin Morris, the student who tweeted this (on the left above), found out about my blog post via Twitter and commented on my blog post, indicating that he saw pedagogical value in clickers but objected to the cost of his clicker, particularly as a senior who won’t use it in future courses.
2. Then Twitter user @iclickercrs, apparently affiliated with the i>clicker company, tweeted about my blog post.  Another Twitter user, @JonathanRose, a professor at Queen’s University in Ontario, Canada (on the right above), saw this tweet and decided to see how many of his (Canadian) students knew what waterboarding is.  He contacted Colin Morris, who directed Jonathan to the Kent State University instructor who posed this clicker question during class, John Jameson.  Jonathan Rose then used the same question in his introductory political science course.
3. Jonathan then posted the results of both questions–the student responses from Kent State and the ones from his Queen’s University students.  Here are the results [PDF].  As you can see, only 28% of his students thought that waterboarding is a surfing term.  Also, more of his students than the Kent State students viewed waterboarding more as torture than an interrogation technique.
4. To bring things full circle, Jonathan Rose tweeted about these results and @iclickercrs re-tweeted Jonathan’s tweet.  I saw this re-tweet, then tracked down Jonathan.  He let me know about items 2 and 3 above, filling in the gaps in my knowledge of this whole social media process.

Watching this all unfold has been very interesting, not only for the interesting uses and reactions to clicker questions, but for the way that Twitter has facilitated connections that might not have happened otherwise.

One Last Update: Colin Morris blogged about this, too, noting the importance of keeping in mind potential audiences when using social media.

Reference: Campt, D., & Freeman, M. (2009). Talk through the hand: Using audience response keypads to augment the facilitation of small group dialogue. The International Journal of Public Participation, 3(1), 80-107.

Summary: This article by David Campt and Matthew Freeman describes ways to use clickers to facilitate dialogue among small-to-medium-sized groups of people (6 to 40 people) with common interests but diverse perspectives.  For example, the authors mention using clickers with residents of an urban neighborhood facing tough questions that involve race and class, as well as with employees from multiple levels of hierarchy within a business discussing the mission and functions of the business.  The authors describe themselves as “dialogue facilitators” and their work as collaborative actions, which uses “dialogue, inquiry, and deliberation to inspire participants, build working relationships, and make decisions about collaborative actions they will take to improve their communities.”  (Wilson, P. (2004). Deep democracy: The inner practice of civic engagement. Fieldnotes: A Journal of the Shambala Insitute. 3, 1-6.)

The authors describe a few different types of clicker questions they use to foster dialogue, including demographic questions exploring participants’ diverse backgrounds, experience questions asking participants “whether or how frequently they may have had specific experiences,” opinion questions about internal and external issues relevant to the community, and fact questions designed to explore differences between objective facts (such as statistics about demographics in the United States) and participant perceptions of those facts.

The authors also describe dialogue focused on collaborative action to have several phases, including introducing participants to each other and to the dialogue process, sharing participant experiences and perceptions, exploring diversity and commonalities with the goal of understanding “underlying social conditions” that produce diverse perspectives, and exploring possibilities for action.  The authors describe several ways that clickers can enhance dialogue in each phase, but they focus primarily on the earlier phases.

For instance, asking demographic clicker questions during the introduction phase can help participants learn about each other more quickly, particularly around demographic characteristics that aren’t immediately visible, such as political affiliation or sexual orientation.  These questions can provide “teachable moments” about group processes, such as reminding participants to be respectful of those with backgrounds different from their own, and help enhance “participants’ sense of empathy for others.”

During the introductory phase, clicker questions can also help to surface common intentions among participants.  The authors note that when there are two “sides” on a contentious issue, often both sides have similar goals but different opinions about reaching those goals.  Asking a clicker question that makes evident participants’ common intentions can help defuse some of the tension in the room that might otherwise arise.

Furthermore, “fact” questions can help bring important facts into the subsequent conversation, often “demonstrating that people in the group know less than they think they do about an issue of relevance,” leading to more open-minded attitudes.

The authors also discuss the use of participant experience questions (e.g. “How long has it been since the last time you can recall witnessing an act of racial discrimination?”) in the second phase of their dialogue facilitation-helping participants understand the variety of perspectives they have on the topic at hand.  Asking such a question, then hearing from a few participants, then commenting on any pattern that emerges (e.g. “It seems that more of the people of color have recent stories.”) is one approach.  However, having those patterns emerge through the results of a clicker question can demonstrate such patterns more quickly and prevent participants from thinking the facilitator is finding patterns that he or she wants to see in the responses.  The authors also note the use of demographic comparison questions, parsing the results of an experience question according to some demographic characteristic of the participants.

Other uses are discussed, as well, including showing matches between clicker question results and national polling data for some questions, helping participants come to decisions about collaborative action steps, and providing both facilitator and participants with information about participants’ feelings about a session at the end of the session.

Finally, the authors make the point several times that clicker questions and their results serve to generate productive dialogue.  They are not an end to themselves.

Comments: While I typically discuss the use of clickers in college and university settings on this blog, I wanted to share and comment on this article since the authors have a particularly nuanced and informed approach to fostering dialogue-with and without clickers-that college and university instructors reading this blog might find useful, particularly those who discuss controversial or sensitive issues in the classroom.  Their writing is also informed by a research literature on fostering dialogue that would likely be unfamiliar to most academics.  I’m also excited by growing use of clickers and other response systems in non-academic educational settings, such as community dialogues as described in this article, as well as church services, corporate presentations, and social science research.

I found the author’s description of types of clicker questions they use to align nicely with the types of clicker questions I group under the umbrella term “student perspective questions.”  I usually think of these questions as being about student demographics, student experiences, or student opinions.  I hadn’t thought about putting factual questions in this category, but it makes sense.  Seeing how students (or dialogue participants) perceive objective facts serves a similar purpose as these other types of questions-helping the community better understand each other and helping the teacher / facilitator better understand the community.  When used to demonstrate to students or participants that they know less than they think they do about a particular topic, these questions also serve to generate a “time for telling.”

When reading about the use of clicker questions to surface common intentions (as described above), I wondered if there’s a risk of having participants feel like such a question is rigged, that the facilitator is asking it mainly as a set-up to make the point that “we all have something in common.”  If there’s a risk of that, I wonder what Campt and Freeman might do to minimize that risk.  I also wonder what they might do if this kind of question backfires, showing that the participants have less in common than they think they do.

A few other questions occurred to me as I was reading the paper’s section on directions for future research on the use of response systems in dialogue facilitation.  The authors ask, “Are there people whose verbal participation in dialogues increases as keypad use increases?”  I would also ask, might a participant who finds out he or she is in the distinct minority on a particular issue be less likely to participate in discussion?  The authors also ask if the availability of providing anonymous feedback might have some distorting effect on reported opinions.  I wondered that, as well, thinking about how contentious or smart aleck participants might abuse the ability to respond anonymously.

I also wonder if there might be a role for pair or small-group discussion prior to voting in these settings.  Peer instruction is a common application of clickers in educational settings-might something similar play a role in dialogue facilitation?  Also, what about asking the same questions at the start and end of a dialogue session as a way to show participants how they’ve changed their perspectives over the course of the dialogue?  Might that be useful in some contexts?

I just had to share this tweet I saw a few weeks ago.

@colinmorris: 44% OF MY U.S. HISTORY CLASS THINKS WATERBOARDING IS A SURFING TERM. I take back everything I’ve said about these “clickers” being useless.

Last week teacher Lisa Short of Gaithersburg Middle School in Maryland shared her perspectives on educational technology with the education committee of the United States House of Representatives.  Education Week covered this presentation, and you can see an eight-minute video of Short’s presentation below.  The first half of her presentation is focused on interactive whiteboards.  The clickers are demonstrated just after 5 minutes in.

Short’s presentation caught my eye because she demonstrated classroom response system technology to the committee, arguing that the anonymity the system provides her students allows her to more accurate assess their misconceptions and prior knowledge since they can answer her clicker questions honestly without fear of embarrassment in front of their peers.

Short also mentioned that between class sessions, she can see which students miss her questions, providing her useful data with which to evaluate her lessons.  For instance, do some students have particular learning styles (visual, auditory, tactile, and so on) that she could address more effectively in future lessons?  And if most of her students miss a question, she knows she’ll need to return to that topic in the next class session.

I’m glad that this House committee had a chance to learn about educational technologies like clickers and interactive whiteboards.  I wish that Short had mentioned that the results of clicker questions can be used immediately to make teaching choices, not just between classes.  The capacity to facilitate such agile teaching is a strength of the technology.

Also, the fact that all of the members of the committee answered her clicker question (about the percent of schools in the US with interactive whiteboards) correctly was a little disappointing, since it meant their was no opportunity to talk about peer instruction.  However, that result did set up her punchline effectively–that only 16% of US schools have interactive whiteboards but 70% of UK schools do!

I attended a presentation by Daniel King, a chemistry faculty member at Drexel University, at a recent conference.  He’s been using clickers for several years in both large, introductory courses and small, upper-level courses, and I thought it might be interesting to share some of his approaches to doing so here on the blog.

Daniel shared several types of clicker questions he uses.  He uses clicker questions to assess students’ knowledge of course prerequisites at the beginning of lessons in which those prerequisites will be used.  He likes to stimulate students’ interest in topics before discussing those topics by asking clicker questions that have non-intuitive correct answers, creating “times for telling.”

Daniel also has students predict the outcome of classroom demonstrations as a way to engage them in those demonstrations.  He noted that many students don’t pay attention to demonstrations until something dramatic happens; his prediction questions engage them earlier in the process.  He also frequently uses the think-pair-share / peer instruction method, engaging students in small-group discussions about difficult questions.

As for grading clicker questions, Daniel prefers to grade on effort and not to penalize students for incorrect answers.  This is because (a) his questions are often designed to introduce students to topics and thus aren’t likely to be answered correctly by many students and (b) he doesn’t want his students to worry about their grade when responding; he wants them to be thinking about the chemistry.

The first semester he included clicker questions in his students’ grades, he counted them toward 5% of his students’ grades as a participation grade.  Students would earn these points by answering at least 75% of the clicker questions during the term.  He found, however, that a number of students who ordinarily wouldn’t attend class starting coming to class just to earn these participation points.  This was problematic because they were often disruptive (chatting among themselves instead of paying attention) and because they frequently responded to clicker questions without thinking about those questions, making it difficult for Daniel to interpret the results of his questions.

To alleviate these problems, the next semester, Daniel awarded 5 bonus points to the final exam scores of students who answered at least 75% of the clicker questions.  This reward wasn’t sufficient to motivate students to attend class if they really didn’t want to, but it did reward the effort of those students to came to class and participated regularly.

Daniel provided some insight into his decision-making process regarding when to move on after a clicker question.  He said it depends on the reasons he has for asking the question.  If the question is meant to assess students knowledge of a concept or technique they’ll need to understand in order to follow the rest of class, he’ll spend time discussing the question unless 85% or more of his students answer it correctly.  If only 50% of students answer a question correctly that he thinks they should have answered correctly had they spent some time studying, he’ll tell the 50% of students who missed it to hit the books and move on with his lesson.

Daniel shared several other aspects of his use of clickers, including his use of a couple of clicker questions early in the semester that most students answer incorrectly to teach students that the most popular answer is not necessarily the correct one.  He’s clearly thought a lot about his teaching choices when using clickers, and he did a great job of articulating his reasons for his choices during his presentation.

Reference: Bombaro, C. (2007). Using audience response technology to teach academic integrity: “The seven deadly sins of plagiarism” at Dickinson College. Reference Services Review, 35(2), 296-309.

Summary: In this article, Christine Bombaro of Dickinson College describes how clickers were used to enhance workshops on plagiarism attended by all first-year students at her school.  Clicker questions were used as “background knowledge probes” to assess students’ prior knowledge of plagiarism in the abstract.  These questions were followed by application questions asking students to apply their knowledge of plagiarism in particular situations (e.g. “Is the following student work plagiarism or not?”).  Students who answered the abstract questions about plagiarism correctly often missed the more applied questions, providing the instructors with useful opportunities to clarify student misconceptions about plagiarism.

A few personal experience questions were asked, as well, including one asking students to report whether they had committed plagiarism. When this question was asked at the start of the workshops, 52% of students said they had done so. When this question was asked again at the end of the workshops, 68% of the students said they had done so, indicating they had gained a broader understanding of plagiarism.

When surveyed about the workshops, students reported that the use of clickers was fun and helped them participate. Another advantage of using clickers was that they provided useful data for revising future workshops by identifying questions that challenged students the most.

The entire “Seven Deadly Sins of Plagiarism” PowerPoint presentation, including clicker questions, is available online.

Commentary: It sounds like Bombaro and her colleagues designed some very effective “time for telling” questions.  Schwartz and Bransford use the term “time for telling” in their 1998 Cognition and Instruction article “A Time for Telling” to describe moments when students are ready and willing to learn from an instructor’s explanation.  Students don’t always begin a class session ready to hear and understand a particular explanation, but they can be prepared to do so.  Bombaro’s question sequences helped to do so in this instance.

The increase of the percent of students who admitted committing plagiarism is a strong indicator of the effectiveness of the workshop. It also leverages the anonymity that clickers provide.

I think that these workshops would have been enhanced by peer instruction around some of the more challenging questions. The workshop facilitators also worked from scripts to make sure that all students received the same messages about plagiarism. I see why that would be important, but it eliminated one of the primary advantages of clickers–allowing instructors to practice “agile teaching.”