Over on the Vanderbilt University Center for Teaching blog, my Vanderbilt colleague Isabel Gauthier, professor of psychology, has shared her experiences asking her students to write their own clicker questions. I met with Isabel a few years ago and briefly discussed ways to use clickers in her courses, and she’s really taken the technology (and pedagogy) and run with it. She’s got a great handle on how to have students write their own clicker questions, and I’ve been wanting to share her experiences here on the blog for a while. Here’s her article, in her own words:

It is difficult to write meaningful and discriminative multiple-choice questions that students find clear and fair. Years ago, I met with CFT assistant director Derek Bruff, who gave me useful pointers to perfect this skill. But a side effect of this interaction transformed entirely the way I teach: I learned so much by working on writing better questions, surely my students could learn too! Derek said something like, “You know, some teachers ask their students to generate questions…” This idea took me on a path to use this strategy, cautiously at first, and then more boldly, as the central pedagogical and evaluative strategy in some of my courses, including Brain Damage and Cognition and Principles of Experimental Design.

I teach these courses three days a week. On two of these days each week, I lecture on course material. These lectures are informed by questions about the readings posted online by students and issues that emerge from a hands-on, semester long project I assign my students. On the third day, we use clickers to go through student-generated multiple choice questions.

Each week each student is responsible for turning in a single question on the weeks’ readings. Students use a PowerPoint template to submit their questions which facilitates use of the question in my clicker software, TurningPoint. In the notes area of the slide, each student includes their name, the correct answer, the page(s) that inspired the question, and, optionally, a justification for the correct answer. Before class, I concatenate all the questions in a single file and read them, grading each on a scale of 1 to 5. The grade goes in the notes area, and, in a textbox on the slide, I write comments about the question. This allows me to print the slides as a PDF with student names removed so that all questions and comments can be distributed to students. I then reorder the slides to choose the right mix of questions I want to use in class with the clickers.

This provides me in a single step with my preparation for the next class, an idea of what I need to focus on during my lecture days, an evaluation of each student, and a mechanism for providing students with feedback on their learning. This weekly feedback allows students to realize how difficult it is to write a good question, one that raises an important issue clearly and is appropriately challenging for their peers. Students eventually learn to key in on critical concepts and relationships in the readings and sometimes even go beyond the readings in interesting ways. They take a more active part in their own and their peers’ learning, and their questions keep me focused on what is most challenging for these students at each point in the course.

Each week students answer the best of these questions in class using clickers, accumulating points for their answers using a generous but motivating grading scheme. If there’s controversy over the correct answer to a question, the class can decide to eliminate a question or to accept multiple answers as correct, provoking interesting discussions. As needed, I can lecture for a few minutes, but issues are generally clarified in class discussion. Questions are used anonymously in class, but students want their question to be picked and use wit and humor to this effect, making the experience more enjoyable for everyone.

This method completely replaces any exams I used to give: They are no longer needed since my students now share the responsibility to evaluate their own learning throughout the semester.

Isabel and her use of clickers were featured on Nashville’s NewsChannel 5 last year. Here’s the video clip:

Reference: Webking, R., & Valenzuela, F. (2006). Using audience response systems to develop critical thinking. In Banks, David A. (Ed.), Audience Response Systems in Higher Education: Applications and Cases. Hershey, PA: Information Science Publishing.

Summary: Webking and Valenzuela describe ways they use classroom response systems in their political sciences courses at the University of Texas-El Paso to foster critical thinking through active participation and class discussions. After noting some commonly cited advantages of teaching with clickers—easier attendance and participation record-keeping, greater participation through anonymity and accountability, and the collection of data to inform agile teaching decisions—the authors provide several concrete examples of clicker questions they have found valuable for developing their students’ critical thinking skills.

The authors’ first example is a sequence of clicker questions that serve to guide students through a close reading of a few passages in the play Antigone. At one point in the play, Antigone makes a statement that seems to very clearly express her belief that obedience to the gods trumps obedience to the king. At another point, however, she makes a somewhat cryptic statement that calls this previous assertion into question. Webking and Valenzuela start with an understand-level question that asks students to clarify this second statement. They follow this with an application-level question asking students to identify a logical consequence of her cryptic statement, one which seems to run counter to her earlier statement about serving the gods. Their third question is an analysis-level one, and it asks students to reconcile the two seemingly contradictory statements by Antigone by identifying a hidden motivation of hers that makes her statements consistent.

Webking and Valenzuela also describe how they use a particularly challenging, analysis-level question about Plato’s Euthyphro. The question asks students to identify the central argument of a particular passage, one that deals with the relationship between justice and piousness. The question is one that Jean McGivney-Burelle would call a “horizontal question” since students answering the question are typically split evenly among three answer choices. Webking and Valenzuela note that one of the three popular responses can’t be supported by the text. Students who argue for this answer choice quickly realize that they were projecting their own perspectives on the text, not arguing from the text. This is a useful metacognitive moment for these students. The class discussion then focuses on the remaining two popular answer choices. Making sense of these two choices requires the students to grapple with categorical logic, the kind that is well-represented by Venn diagrams. Once the students have discussed their way to the correct answer, they realize the value of categorical logic in making sense of arguments like the ones Plato makes—another metacognitive moment.

The Plato example comes from one of the authors’ smaller, upper-level courses, and they assert that “it is in a smaller class that the [classroom response] system is at its best in encouraging discussion and precise argument.” They reach this conclusion, in part, because of the ability of their classroom response system to report to the instructor individual student responses to clicker questions as those responses are submitted. The authors use these individual, real-time results to guide their post-vote discussions, focusing on “groups which had difficulties in reaching consensus, students or groups which answered particularly quickly or particularly slowly, students who disagreed with their groups, students who changed their minds, and so on.” They argue that the ability to see individual, real-time results is important in leading effective post-vote discussions since it allows instructors to analyze “each student’s rational odyssey with each question.”

Also in the article are two examples of student perspective questions the authors use to motivate particular topics in their courses. In one example, they ask students to identify questions they aren’t likely to ask someone they’ve just met. Invariably, students identify the questions about religion and politics. The authors point out to students that one reasonable conclusion from this is that religion and politics are the least important things to know about when getting to know someone. This motivates students to want to learn why this social phenomenon exists.

Comments: This would be a great article to give a faculty member in political science or philosophy who’s interested in getting started teaching with clickers. Webking and Valenzuela provide a concrete, interesting example of a guided close reading of a text (Antigone) using clicker questions of increasing difficulty. This is a great model for instructors in the humanities and social sciences interested in helping their students develop critical thinking and close reading skills. I wish, however, that they had included some voting data in this example and had discussed how they use the results of these questions to guide discussions, as they did with their Plato example.

The Plato example is a great model of clicker use in text-based courses, too. One reason is that the approach Webking and Valenzuela use leads students to appreciate the nature of argument in their discipline. They write, “In time, and actually not very much time, students learn to care more about the strength of the argument than about having their initial position defended as right.” The authors present a useful list of options for leading these kinds of class discussions—focusing on groups that were conflicted, students who answered quickly or slowly, students who changed their minds, etc.

The authors assert that the quality of discussions they can foster depends on the availability to the instructor of real-time, individual voting data. Not all classroom response systems have this feature and, in my experience, instructors who have the option of looking at individual results as they come in don’t frequently take advantage of this option. I think that perhaps the availability of real-time, individual results isn’t as critical as Webking and Valenzuela assert. I’ll often have my students vote on a question individually, then discuss it in groups, then vote again. I’ll sometimes ask for a student who changed his or her mind from the first vote to the second vote to explain his or her reasoning. I can also see asking for a student who disagreed with his or her group to contribute to the post-vote discussion.  (That’s a nice idea, one that I’ll have to try soon!)

My approach, using the aggregate and not individual voting data, relies on students who fit certain profiles volunteering to share their perspectives with the class. Webking and Valenzuela’s approach doesn’t rely on volunteers, but it isn’t quite cold-calling, either, since they select students only after the students have had a chance to consider and respond to the clicker question. I’d like to call this “warm-calling” since the students have had a chance to warm up to the question and since the instructors aren’t calling on students without any knowledge of what those students might contribute to the discussion. I’m not familiar with many instructors who practice warm-calling.  If you do, I’d love to hear from you in the comments about your experiences doing so.

Image: “Coffin Sculpture of Antigone” by Flickr user Xuan Rosamanios / Creative Commons licensed

I’ve often said that those teaching in the social sciences have the most options for using clickers.  Both content and opinion questions are typically on-topic in a social science course, giving these instructors the ability to use clickers in just about any way imaginable.  Case in point: The video below by Russell James, who teaches in the housing and consumer economics program at the University of Georgia.

James covers a lot of ground in this video.  He shares examples of several types of clicker questions he uses, including student perspective questions (sometimes used to connect student opinions with results from national opinion polls), experiment questions (in which students participate in experiments designed to illustrate certain economic behaviors), and prediction questions (in which students predict the outcomes of research experiments from the literature).  James moves very quickly in this video, so be ready to pause it in order to read his sample questions.

James mentions other uses of clickers, too, such as taking a minute at the end of each class to ask students the kinds of rating questions that typically appear on end-of-semester course evaluations.  He says this is the “number one” use of clickers that has transformed his teaching, since it generates regular data on his teaching effectiveness.  James mentions a use I would call a monitoring question–asking students to click in when they’ve finished a particular task.  He notes that this lets him know when it’s time to move on after an activity and that the count of students who have finished displayed on-screen sends a message to students who aren’t keeping up with their peers.

James also describes a game he calls “clicker wars.”  In this game, often used to review for exams, he divides his students into groups, perhaps based on gender or class year.  Each group is then divided into teams of two or three students each, and each team is given a single clicker.  James then poses questions to his students, and each team must come to consensus on its answer.  If a team misses a question, they’re out of the game as a team, but can still help other teams in their group.  The winning team gets some kind of prize at the end of the game, and the winning group gets a prize, too, although a lesser one.  James says this gives students a lot of incentive to stay engaged in the game throughout.

James also suggests a few ways to handle students who cheat with clickers by bringing their absent friends’ clickers to class, making it appear that those friends are present.  Most of James’ suggestions I’ve mentioned here on the blog before, but he had a novel one, too.  He suggests taking a digital photo of the class as a deterrent.  If a student’s clicker says that student was present but the student isn’t in the photo, that becomes an honor code violation.  James says that telling students you’re doing this will prevent some cheating.

Thanks to Russell James for sharing his creative ideas for teaching with clickers!

A few weeks ago, Jason B. Jones, one of the editors at my favorite group blog, ProfHacker, invited readers who teach in math and science disciplines to contribute articles to the blog.  I took Jason up on his offer, and I’ve now contributed two posts to ProfHacker.

In the first post, “Multiple-Choice Questions on Exams,” I describe some of the reasons I use multiple-choice questions on my exams.  As regular readers of my blog know, I find multiple-choice questions very useful as clicker questions.  I think many instructors underestimate their use in the classroom.  However, as I write in my ProfHacker post, I’ve found multiple-choice questions very effective for assessing students’ conceptual understanding in my courses.  Moreover, since I have my students spend a good chunk of each class session grappling with multiple-choice questions, it seems appropriate that these kinds of questions would show up on their exams!

In my second ProfHacker post, “Getting Students to Do the Reading: Pre-Class Quizzes on WordPress,” I describe the rationale, use, and implementation of pre-class reading quizzes in my math courses.  As I’ve mentioned on this blog, I ask my students to read their textbook before coming to class as a first encounter with the course material.  This frees up some class time for more active learning (such as clicker-facilitated peer instruction), and it also allows me to practice “just-in-time teaching” by letting student responses to pre-class quiz questions inform my lesson plans so that our use of class time is more responsive to student learning needs.  See the ProfHacker post for lots more details and a great set of comments by other ProfHacker readers.

On his “Old is the New New” blog, Rob MacDougall recently argued that the question “If a viking and a samurai fought, who would win?” has great potential for helping students learn to think critically.  Here’s why:

Take a counterfactual question–as far as I know, vikings never fought samurai–and have at it. You can enter this debate with any level of starting knowledge, arguing solely from the evidence in the pictures (that samurai looks pretty fierce, but the viking has his buddies with him). Yet there is no bottom to the amount of evidence you could gather or the complexity of the arguments you could marshal on either side. You could talk about military tactics or metalworking technology. You could research the agricultural potential of Scandinavia or the codification of Bushido. You could spin out a whole saga in which a Nihonese armada devastates the Vinlander entrepots at “Perleshavn” and vengeful Norsemen go a-viking into the Inland Sea.

This is a great example of how a simple, multiple-choice question (even one with merely two answers!) can potentially be used to generate rich in-class discussions.  I don’t know if Rob MacDougall has used his vikings vs. samurai question as a clicker question, but just imagine how well it would work as one!  I can see students getting pretty fired up over the results of such a clicker question.  Poll the students, then have them discuss the question as a class, then poll them again to see if they’ve shifted opinions, then discuss further and poll perhaps a third time.

Here’s Rob again, in one of the comments on the post:

The conversations I’ve had in the last few days about samurai and vikings, with everyone from a 3 year old to a professor of Japanese history, suggest to me that even banal questions can scale to accommodate multiple levels of historical knowledge.

I should emphasize that a vikings vs. samurai clicker question isn’t of much value by itself.  It’s the discussion that it frames and motivates that’s of real educational value.  And that discussion could happen without a clicker question, but I suspect that in many instances, a clicker question would enhance that discussion.

Plus, vikings versus samurai?  How cool is that?

Update: Rob MacDougall’s follow-up post, “Would You Rather,” features more counterfactual questions, ones that point students toward social history and material culture rather than military history and the history of technology.

Image: “Viking” by Flickr user hans s / Creative Commons licensed

A math colleague of mine, who blogs under the name Doc Turtle, recently blogged about his use of a calculus worksheet that helps his students “guide themselves through the algebraically intense process of partial fractions.”  Doc Turtle reports that his students look forward to this kind of work, and he’s planning to develop more activities along these lines.

I’ve heard from several instructors who have students engage in this kind of active, self-directed learning in class (through worksheets, clicker questions, and so on) that some students complain that the professor isn’t doing any work.  I suspect that these are the students who expect to come to class, take a lot of notes, and figure the material out while working through their homework.  They can sometimes push back when their instructor isn’t presenting course content in the way they expect.

Of course, instructors who design and implement activities like Doc Turtle’s worksheet activity aren’t avoiding the hard work of teaching.  Instead, they’re being intentional about what they want their students to learn and they’re planning and facilitating experiences designed to help their students learn those things.

As Ian Beatty wrote over on his blog, “It’s not really creating [clicker] questions that’s tough.  The hard part is figuring out what I want my students to learn from the class, and casting that in terms of what I want my students to be able to do.”  Once he’s done that, he says it’s relatively easy for him to write effective clicker questions.  “Just formulate a question asking them to do that (in a particular context), and then much of the class activity is me helping them struggle through the process as they learn how.”

What struck me about Doc Turtle’s post was how excited his students are to engage in this kind of active learning.  As I mentioned above, not all students see this kind of learning as valuable.  Did Doc Turtle just get lucky with a batch of exceptional students?  I suspect not.  I’m guessing that he’s been teaching his students to learn this way since the first day of classes so that by this point in the semester, his students are perfectly willing to see this kind of activity as valuable.  I think that’s an important takeaway: If we’re asking our students to learn in a “new” way, then we need to help them learn how to learn in that way.

Do you find that your students push back when you ask them to engage in active learning in class?  How do you help them see the value in this kind of learning over time?

Eric Tremblay recently blogged about his plans to have his students write clicker questions for him to use during class.  He’s not lazy; he wants his students to think about the material in his course and possible misunderstandings of that material.  Student questions will be posted in a class forum, and he’ll select one or more each week to use in class.  Students earn participation credit for posting questions and triple credit when their questions are the ones selected.

Having students write exam questions as a way of preparing them to take exams is a time-honored teaching strategy, but I have only lately heard of instructors like Eric having students craft clicker questions as a way to have them engage with course material.  Writing clicker questions is difficult, but that’s due in part to the difficulty of predicting student misunderstandings, which is required for constructing good wrong answer choices.  I wonder if students might be better able to identify potential misunderstandings since they are not experts in their fields and are thus closer, in a sense, to those misunderstandings.

This idea of having students write clicker questions came up a few times at the recent Joint Mathematics Meetings I attended.  Have you tried this?  I would be interested in hearing how this plays out.

Here’s a nice follow-up to my previous post about backchannel use during live performances.  Over at Abilene Christian University, where all the students (more or less) have iPhones, a group of students were given extra credit for watching the recent State of the Union address by US President Barack Obama.  However, they didn’t just watch it; they responded to clicker questions asked by their instructor on-the-fly during the speech.  The students used their iPhones to respond to these questions, but any kind of classroom response system would do the trick for something like this.  This seems like a great way to use some student perspective questions to help students engage more meaningfully with a live broadcast of this nature. (You may recall that I mentioned some universities that did something similar during the 2008 presidential debates.)

ACU also thought ahead to video the evening!

A little while back, Jeff McClurken directed me (via Twitter, naturally) to a 2006 blog post by Gardner Campbell describing an APGAR test for class meetings.  In spite of having read Gardner’s work for a couple of years now, I hadn’t seen this idea until Jeff mentioned it as a way to use clickers to help students assess their readiness for a class session.  I think it’s a great idea, so I wanted to mention it here.  (By the way, this is yet another example of how Twitter has led to good things!)

In 1953, physician Virginia Apgar published a test designed to capture a newborn baby’s health in a score between 0 and 10.  The test measures five aspects of a newborn’s health: Appearance, Pulse, Grimace, Activity, Respiration–a convenient mnemonic device based on the inventor’s name.  Each aspect is given a score of 0, 1, or 2, yielding a total score between 0 and 10.  A baby with an APGAR of 10 appears to be in perfect health; a baby with an APGAR of 4 or lower is not doing well.

Gardner’s idea was to devise an APGAR test that students would take before a class session as a way to self-assess their readiness for that class session.  He’s put his five APGAR questions in a slide available on Slideshare with a photo of Dr. Apgar herself:

As Gardner suggested in his post, you could easily ask these questions using clickers, either as five separate clicker questions or as one question asking students to report their total score.  Most clicker systems will let you see the average of the responses for a rating question, so you could quickly see the average APGAR score for your entire class.

Why might you use this APGAR test for students?

• As Gardner points out, just sharing these questions helps to communicate to students your expectations for them.  I frequently hear from students and instructors that mismatched expectations lead to all kinds of frustrations, so clarifying your expectations for your students out-of-class studying can be very helpful.
• Furthermore, having students assess their own readiness for class creates the opportunity for a “metacognitive moment” in which students learn something about their own learning.  As a student considers these questions, she is motivated to consider how the five actions described in the questions might (or might not) help her learn.  As Steven Greenlaw points out, this can help students, particularly first-year students, become more intentional learners, which is a good thing.
• On the instructor side of things, the results of a class’ APGAR test can provide useful information on students’ readiness for class, information that can be acted upon immediately.  If the class average is low, you’ll have to make some on-the-fly decisions about how to respond to the fact that your students aren’t that ready for class.  If the class average is high, you can look forward to some engaging discussion that day!

One potential problem with giving your students an APGAR test is that students might over- or under-report their readiness for class.  Perhaps they don’t want you to think they’re slacking off, so they rate themselves high.  Or perhaps they think you’ll go easy on them during class if they rate themselves low.  So you would have to take their responses with a grain of salt, considering what social dynamics might be at play that would lead them to fib a little on their scores.  The fact that clickers allow students to respond anonymously to questions like these helps, certainly, since they won’t be as subject to peer pressure when responding.

Have you tried something like this with clickers?  If so, how did it go for you?  If not, does this sound promising?

I couple of weeks ago on this blog, I shared a tweet by Colin Morris, a student at Kent State University in Ohio.  His comment (via Twitter) was, “44% OF MY U.S. HISTORY CLASS THINKS WATERBOARDING IS A SURFING TERM. I take back everything I’ve said about these ‘clickers’ being useless.”  After I shared this tweet on my blog, a few interesting things happened.

1. Colin Morris, the student who tweeted this (on the left above), found out about my blog post via Twitter and commented on my blog post, indicating that he saw pedagogical value in clickers but objected to the cost of his clicker, particularly as a senior who won’t use it in future courses.
2. Then Twitter user @iclickercrs, apparently affiliated with the i>clicker company, tweeted about my blog post.  Another Twitter user, @JonathanRose, a professor at Queen’s University in Ontario, Canada (on the right above), saw this tweet and decided to see how many of his (Canadian) students knew what waterboarding is.  He contacted Colin Morris, who directed Jonathan to the Kent State University instructor who posed this clicker question during class, John Jameson.  Jonathan Rose then used the same question in his introductory political science course.
3. Jonathan then posted the results of both questions–the student responses from Kent State and the ones from his Queen’s University students.  Here are the results [PDF].  As you can see, only 28% of his students thought that waterboarding is a surfing term.  Also, more of his students than the Kent State students viewed waterboarding more as torture than an interrogation technique.
4. To bring things full circle, Jonathan Rose tweeted about these results and @iclickercrs re-tweeted Jonathan’s tweet.  I saw this re-tweet, then tracked down Jonathan.  He let me know about items 2 and 3 above, filling in the gaps in my knowledge of this whole social media process.

Watching this all unfold has been very interesting, not only for the interesting uses and reactions to clicker questions, but for the way that Twitter has facilitated connections that might not have happened otherwise.

One Last Update: Colin Morris blogged about this, too, noting the importance of keeping in mind potential audiences when using social media.