I’ve often said that those teaching in the social sciences have the most options for using clickers.  Both content and opinion questions are typically on-topic in a social science course, giving these instructors the ability to use clickers in just about any way imaginable.  Case in point: The video below by Russell James, who teaches in the housing and consumer economics program at the University of Georgia.

James covers a lot of ground in this video.  He shares examples of several types of clicker questions he uses, including student perspective questions (sometimes used to connect student opinions with results from national opinion polls), experiment questions (in which students participate in experiments designed to illustrate certain economic behaviors), and prediction questions (in which students predict the outcomes of research experiments from the literature).  James moves very quickly in this video, so be ready to pause it in order to read his sample questions.

James mentions other uses of clickers, too, such as taking a minute at the end of each class to ask students the kinds of rating questions that typically appear on end-of-semester course evaluations.  He says this is the “number one” use of clickers that has transformed his teaching, since it generates regular data on his teaching effectiveness.  James mentions a use I would call a monitoring question–asking students to click in when they’ve finished a particular task.  He notes that this lets him know when it’s time to move on after an activity and that the count of students who have finished displayed on-screen sends a message to students who aren’t keeping up with their peers.

James also describes a game he calls “clicker wars.”  In this game, often used to review for exams, he divides his students into groups, perhaps based on gender or class year.  Each group is then divided into teams of two or three students each, and each team is given a single clicker.  James then poses questions to his students, and each team must come to consensus on its answer.  If a team misses a question, they’re out of the game as a team, but can still help other teams in their group.  The winning team gets some kind of prize at the end of the game, and the winning group gets a prize, too, although a lesser one.  James says this gives students a lot of incentive to stay engaged in the game throughout.

James also suggests a few ways to handle students who cheat with clickers by bringing their absent friends’ clickers to class, making it appear that those friends are present.  Most of James’ suggestions I’ve mentioned here on the blog before, but he had a novel one, too.  He suggests taking a digital photo of the class as a deterrent.  If a student’s clicker says that student was present but the student isn’t in the photo, that becomes an honor code violation.  James says that telling students you’re doing this will prevent some cheating.

Thanks to Russell James for sharing his creative ideas for teaching with clickers!

Earlier this month, David Clemens, who teaches literature at Monterey Peninsula College, authored a post titled “The Data-Driven Classroom” on the National Association of Scholars (NAS) blog.  In his satirical post, Clemens he refers to clickers as “the future of edubiz:”

Clickers generate tsunamis of admin’s holy grail—data! With data I can prepare reports and quantify learning to show that I’m accountable.

His post goes on in this tone, satirizing those who use technology for technology’s sake.  (He includes a few shots at the use of PowerPoint and hyperlinked texts.)  I objected to his characterization of clickers as primarily used to generate assessment data for administrators, so I left a couple of comments on the post, arguing that clickers are primarily used to engage students during class, not to generate data for administrators.

David Clemens then wrote a follow-up post, “Reply to Bruff,” on the NAS blog.  In this post, he asserts the following:

Students urgently need unmediated classrooms and Exemplary-level teachers [those who model critical reasoning for students] who through Socratic dialogue and shared inquiry develop independent, informed, and original minds.

He argues that clickers (and PowerPoint and student learning objectives) don’t help teachers meet this goal.  Since I believe they do and that the useful effects of clickers are frequently underestimated in the humanities, I posted a lengthy comment elaborating on my earlier comments on the NAS blog.  You can read Dr. Clemens’ posts by following the links above.  Below you’ll find my response.

I wanted to share some additional thoughts on Cliff Atkinson’s new book, The Backchannel, and its implications for higher education.  As I mentioned in my earlier post, the first chapter of the book is available online and provides a very clear introduction to the logistics and possibilities of the backchannel.  What might the backchannel look like in educational settings?  Here are a couple of examples.

“The Twitter Experiment,” a five-minute YouTube video, shows how UT-Dallas history professor Monica Rankin used Twitter to facilitate a backchannel discussion.  In her case, she had a somewhat large class that she broke into smaller discussion groups.  The students were encouraged to post their thoughts on Twitter during the small-group discussion time.  The Twitterstream was displayed on the big screen for the whole class to see.  This led to some “cross-fertilization” of small-group discussions as ideas generated by one group were read and discussed by other groups.  Dr. Rankin also had a TA monitor the backchannel, responding to student questions and surfacing important points for Dr. Rankin to discuss with the entire class from time to time during the class session.  For more details on Dr. Rankin’s use of Twitter, see my earlier post on this topic.

Purdue University has developed a system called Hotseat that facilitates backchannel discussion.  This system allows students to contribute to the backchannel in a variety of ways, including Twitter and Facebook.  The student contributes are typically displayed on a big screen for the entire class to see, and the instructor typically takes a “Hotseat break” of sorts every now and then to respond to the questions raised in the backchannel.  Students can comment on other students’ posts and they can “vote up” comments or questions their peers post so that instructors have an easier time identifying the most pressing topics.  The Purdue team shared their work on Hotseat at the EDUCAUSE Learning Initiative earlier today, and, according to Twitter user @eyb, who “live-tweeted” the presentation, students really liked the system.  They didn’t necessarily think it helped them learn better, but they liked it and they wanted their instructors to spend more time responding to the questions raised in the backchannel.

(I’ve been meaning to talk about Hotseat here on the blog for a while now.  Thanks to @eyb for some great reporting at ELI!  I feel I have a much better sense of the system now, technologically and pedagogically.)

What are some other ways that backchannel might function in educational settings?  Cliff Atkinson describes some common and uncommon uses of the backchannel in Chapter 3 of his book.  Here are my thoughts on how Atkinson’s uses might map over to educational settings:

• Notetaking: Students can take their notes during a class in the backchannel.  This provides an electronic (and thus searchable) set of notes for the student.  Moreover, students can read and use each other’s notes more easily.  You might even select two or three students each day to be official class note-takers, freeing other students up for more engagement in class.
• Sharing Resources: Students can also look online (or, call me crazy, in their textbooks) for information that supplements the lecture or class discussion.  It’s easy to share links in the backchannel thanks to all the URL shortening services, and students can be very good at finding useful and relevant information online.  And if a resource shared by a student isn’t useful or relevant, it creates an opportunity to discuss with students how to find and evaluate online information resources.
• Commenting: Students can also comment on the ideas being share or discussed in class.  Just providing a visible venue for student comments is likely to encourage more students to reflect actively during class.  Plus, students can read and respond to each others’ reflections.  Sure, students can contribute to online discussions after class, but there’s something exciting about having more students engage in discussions during class–more than just those who are bold enough and quick enough to contribute verbally.
• Amplifying: The Hotseat feature mentioned above that allows students to “vote up” peer comments they find important is an example of what Atkinson calls “amplifying what others are saying.”  On Twitter, this happens via retweeting: If a comment is retweeted frequently, then many people find it interesting enough to share.  Google Moderator is a free service that works similarly–students can post questions and others can vote them up or down.  This kind of feature is a great way to handle the problem I’ve identified here on the blog several times: It’s really hard for an instructor to follow and make sense of the backchannel during class given the open-ended nature of the comments.  Giving the students the ability to identify more or less relevant comments is one way to help with this.  (Monica Rankin’s use of a moderator–her TA–is another.)
• Asking Questions: I’ve put this a few spots down the list since I think it’s a more obvious use of the backchannel than some of the ones listed above.  Backchannel provides students an additional way to ask questions.  Students are frequently hesitant to ask questions in class for a variety of mostly social reasons–they don’t want to look “dumb” in front of their instructor or their peers.  Anonymous backchannel discussions make it extremely easy for these students to surface their questions.  Even when students are identified on the backchannel, having a venue where questions are encouraged is likely to make it easier for students to share questions.  And if the backchannel includes an amplification tool, then students can support each others’ question-asking very directly.
• Helping One Another: Keep in mind that there are several kinds of backchannel conversations, including student-to-student conversations.  When one student poses a question on the backchannel, another student might very well answer that question before the instructor can get to it.  This kind of peer instruction is a common use of clickers, and it can work well in the backchannel, too.
• Offering Suggestions: The backchannel can give students a voice in where a class discussion goes.  Students can suggest discussion topics or questions.  They can also suggest useful readings, activities, or topics for subsequent classes.  They can provide instructors with feedback on what’s working and what’s not from their perspective.  Many instructors have students complete a “minute paper” at the end of each class in which students identify the most important point of the day or ask a question.  The backchannel allows instructors to gather this kind of feedback whenever students are ready to share it during class.
• Building Community: Particularly in large classes, it can be hard for students to get to know more than just the few students they sit near.  Backchannel discussions can help students get to know each other in a variety of ways.  I would argue that it’s important for students to have avatars or icons attached to their backchannel posts, preferably photos of themselves.  Seeing someone’s face along with their comments and their name helps build actual, not just virtual community.
• Opening the Classroom: Some backchannels are private; that is, only the instructor and students can see or participate in the backchannel conversation.  Others, like Twitter, are public, allowing those outside the classroom to participate in the discussion.  This provides an opportunity to open the class discussion to those not currently enrolled in the course–students taking other courses, students who took the course in the past, academic experts at other institutions, and more.  These external people have the potential to learn from and contribute to the backchannel discussion.

That was fun thinking through these options!  You can have fun, too: What did I miss?  Comments or suggestions for the uses I’ve listed above?

I’ve recently become a fan of the “mark all that apply” type of question my classroom response system facilitates.  I call these “multiple mark” questions in my book.  Here’s one I used in the linear algebra course I’m teaching this fall.

This question is adapted from one of the questions written by Project MathQuest out of Carroll College.  Their version of the question wasn’t a multiple mark question.  Instead, it included a fifth option, “More than one of the above are possible.”  While that option makes the question more interesting and more challenging for the student, it also yields inconclusive data about student learning since students submitting that response may have different ideas about which of the four options are possible.  That’s not all bad, of course.  Given my interview with Kelly Cline, one of the PIs for Project MathQuest, I can imagine Kelly leveraging that ambiguity into a productive classwide discussion of the question.

However, I decided to turn this question into a multiple mark question by adding the instruction, “Mark all that are possible.”  As you can see from the results, 20 of the 20 students present that day indicated that option 3 was possible, 19 of the students indicated options 1 and 2 were possible, and 14 of the students felt that option 4 was possible.  This was very useful feedback for me, since I could quickly tell that the class was in agreement on options 1 through 3, but option 4 deserved some further discussion.

I’ll admit, however, that I got a little tripped up on my own question logic here.  As it turns out, all four options are possible, which was not my intent when I included this question in my lesson plan.  Option 2 is only possible if the third plane intersects the two overlapping planes and option 4 is only possible if the three planes are parallel because they are in fact the same plane.  The way I’ve worded the question, these wrinkles aren’t addressed, making all four options possible.  As a result, the question doesn’t do a great job at uncovering student understanding of these wrinkles.

Here’s the question I should have asked instead:

Suppose you have a system of 3 linear equations in 3 variables.  Which of the following conditions would guarantee that the system has an infinite number of solutions?  Mark all that apply.

1. All three equations represent the same plane.
2. Two of the equations represent the same plane.
3. The three equations represent planes that intersect along a line.
4. The three equations present parallel planes.

With this question, only options 1 and 3 are correct.  With option 2, it could be that the third plane is parallel to but distinct from the two overlapping planes, yielding no solutions instead of infinitely many solutions.  With option 4, it could be that the three planes are not the same plane, again yielding no solutions instead of infinitely many solutions.  This wording of the question puts the special cases in their proper places.

Have you used multiple mark questions?  Do you find them more difficult to write?  How do your students respond to them?

Teaching with clickers provides instructors with a wealth of information they can use to learn about the effectiveness of their own teaching.  Finding out that half your students don’t understand a topic (as evidenced by wrong answers to a clicker question) minutes after you’ve explained it can be disappointing, but it’s better to know your students are confused than to assume they’re following along.  More direct clicker questions (like, “How well are you following my lecture right now? Very, somewhat, a little, or not at all?”) can also provide formative feedback on one’s teaching.

But what about using clickers for more summative evaluation of one’s teaching?  Might clickers be used in place of the handwritten or online end-of-semester course evaluations?  I haven’t heard of many schools doing so, but I did speak with someone involved in such an effort back at EDUCAUSE in the fall of 2008:

I spoke with Danny Sohier of Université Laval in Québec after the session.  His school is using clickers to conduct end-of-semester course evaluations during class.  They found that online course evaluations resulted in low response rates, a problem I’ve heard about from many institutions.  They now use clickers to collect student responses to multiple-choice evaluation questions during class in some courses, inviting students to respond to open-ended questions online outside of class.  Danny indicated that this arrangement is working pretty well.

These seems sensible to me.  The advantage of handwritten course evaluations completed during class is that response rates are fairly high, since often most of the students enrolled in a course show up on the day evaluations are completed.  The disadvantage is that handwritten evaluations take more work to analyze.  Using clickers during class to ask these kinds of questions keeps response rates high and yields data that are easy to use.

The advantage of online course evaluations is that students can take their time and compose thoughtful and lengthy replies to open-ended questions about one’s course or teaching.  The disadvantage is that relatively few students do so!  The system described above allows motivated students to submit thoughtful responses to open-ended questions after class, while hearing from all (or almost all) the students in a course on some useful multiple-choice questions during class.

This topic has been on my mind since Nira Hativa of Tel Aviv University posted an inquiry to the POD Network listserv about using clickers for formal teaching evaluation.  Kevin Owens of Turning Technologies outlined one way to do so:

1. Evaluation questions are entered ahead of time into a PowerPoint presentation (via our TurningPoint software which is free) and saved onto the memory stick or network drive.
2. Instructor leaves room
3. Selected facilitator (staff member, student worker, etc.) loads interactive PowerPoint presentation and distribute clickers to students
4. Selected facilitator administrates the questions giving 5 to 10 seconds on each question to allow students time to submit responses
5. When finished, facilitator saves data onto memory stick or network drive for future reporting.
6. Saved data can produce up to 30 automated reports within our TurningPoint software or can produce raw data available for export into existing reporting tools on your campus.

Mark Scarbecz of the University of Tennessee College of Dentistry pointed to a poster he presented about UT Dentistry’s experiences using clickers for evaluating teaching.  His conclusions?

Response rate and acceptance of the ARS for course evaluation were greater than for a web-based system. The ARS was effective and efficient for data collection. Random selection of keypads provided anonymity. ARS software had multiple formats for data reporting. Limitations of the ARS are the following: 1) a small question set reduces the length of evaluation sessions and student boredom, but also information collection; 2) student conversations during sessions may bias responses; 3) the ARS provided no mechanism for open-ended feedback; and 4) development/ presentation of question sessions and dissemination of data are timeconsuming and labor-intensive.

Nira Hativa replied to the listserv to note that her situation has a particular challenge.  The courses she’s evaluating rotate instructors every two or three class sessions, so waiting until the end of the semester to collection feedback on those instructors isn’t practical.  She discussed this challenge with Mike Theall of Youngstown State University, and it appears that she doesn’t have the staff power to send a facilitator in these classes every two or three class sessions to conduct this evaluation, the students might not provide honest feedback if the instructor is the one administering the evaluations, and some of her instructors might object to one of the students in the class proctoring the evaluations (as has often been done in the past with paper-based evaluations).

If the student proctor option is off the table, I’m not sure Nira’s problem has a solution.  However, I throw the question to you now: Any ideas for helping Nira?  And do you have any experience with or ideas about using clickers for course evaluations?

Last week teacher Lisa Short of Gaithersburg Middle School in Maryland shared her perspectives on educational technology with the education committee of the United States House of Representatives.  Education Week covered this presentation, and you can see an eight-minute video of Short’s presentation below.  The first half of her presentation is focused on interactive whiteboards.  The clickers are demonstrated just after 5 minutes in.

Short’s presentation caught my eye because she demonstrated classroom response system technology to the committee, arguing that the anonymity the system provides her students allows her to more accurate assess their misconceptions and prior knowledge since they can answer her clicker questions honestly without fear of embarrassment in front of their peers.

Short also mentioned that between class sessions, she can see which students miss her questions, providing her useful data with which to evaluate her lessons.  For instance, do some students have particular learning styles (visual, auditory, tactile, and so on) that she could address more effectively in future lessons?  And if most of her students miss a question, she knows she’ll need to return to that topic in the next class session.

I’m glad that this House committee had a chance to learn about educational technologies like clickers and interactive whiteboards.  I wish that Short had mentioned that the results of clicker questions can be used immediately to make teaching choices, not just between classes.  The capacity to facilitate such agile teaching is a strength of the technology.

Also, the fact that all of the members of the committee answered her clicker question (about the percent of schools in the US with interactive whiteboards) correctly was a little disappointing, since it meant their was no opportunity to talk about peer instruction.  However, that result did set up her punchline effectively–that only 16% of US schools have interactive whiteboards but 70% of UK schools do!

I had hoped that my second podcast episode would follow the first one a little more closely in time, but the spring semester got away from me, particularly with the book coming out in February.  However, I’ve got a great interview for you this time around.

This episode features an interview with Kelly Cline, associate professor of mathematics at Carroll College in Montana.  Kelly and two of his colleagues at Carroll, Holly Zullo and Mark Parker, are principal investigators for a National Science Foundation project called Project Math Quest.  Kelly and his collaborators have written and tested clicker questions for linear algebra and differential equations courses, and they’ve made their question bank–consisting of hundreds of questions–available online.

In this interview, Kelly talks about the ways that he uses clickers and classroom voting, as well as his energy and enthusiasm, to motivate his students to engage meaningfully with mathematical concepts and logical thinking in the classroom.  He also talks about elements of effective clicker questions, as well as how he has modified his approach to teaching to include more active learning elements while still covering all the content he needs to in his courses.

Click the following link to download and listen to the interview:

Podcast #2 – An Interview with Kelly Cline [41:14]

Links:

• Kelly Cline’s home page
• Project Math QUEST

Podcast: Play in new window
| Download

I finally got around to watching Bill Gates’ recent TED talk.  This is the talk you may have heard about where he released mosquitoes into the audience.  Well, after spending the first half of the talk speaking about the challenge of defeating malaria, Gates spent the last half addressing the question, How do you make great teachers?  He made a couple of points that made me think of classroom response systems.

One was that he praised the teacher at a KIPP school he recently visited for engaging all of the students in her classroom throughout the class period.  He argued that more classrooms should be like this, where students can’t goof off or not pay attention.  I would argue that classroom response systems are one great tool for helping teachers (at the K12 level or at the college level) keep all students in a class engaged with the material.  Since all students can be expected to respond to a clicker question, not just the two or three students who put their hands up first, clickers can be used to help keep all students on task.

Another was that he indicated that in some school districts, test score data that could be used to help teachers identify strengths and areas for improvement isn’t shared with teachers!  This surprised me.  If there’s existing data on student learning that teachers could use to improve their effectiveness, why not share that data with the teachers?

This reminded me of a conversation I had with an elementary school principle at the recent ConnectEd Summit.  (Yes, I’m still talking about that…)  She was looking at my book and discussing with me how clickers might be useful in a K3 setting.  One point she made was that classroom response systems have the potential to provide her teachers with useful data on student learning, particularly if the questions used were tied to state or national standards.  A teacher could use the data generated from one day to help determine the plan for the next day.

So if anyone reading this happens to know Bill Gates, please let him know that I would be glad to talk with him about how classroom response systems might help meet the challenge he outlined in his TED talk!

More ideas today for how using mobile devices (particularly smart phones like the iPhone) as part of classroom response systems might enhance or detract from the effectiveness of such systems.  I brainstormed these ideas while preparing for my presentation at the recent ConnectEd Summit.  See part one and part two of this series, too.

Display of Results – Not only does the bar chart showing distribution of student answers to a multiple-choice clicker question provide useful information to an instructor, but it also provides useful information to students.  Often students appreciate knowing where they stand relative to their peers, whether that’s in the context of a content question with correct and incorrect answers or a student perspective question asking students to share their opinions or experiences.  Knowing the distribution of answers can often motivate students to engage more seriously in classwide discussion, particularly if there are multiple popular answer choices.

How to replicate this advantage when having students respond to free-response questions with mobile devices?  I noted the challenge of coming up with useful reporting tools for these kinds of questions in my last post.  One idea that occurs to me, however, (because I think one of the ACU people mentioned it at EDUCAUSE) is that the entire set of responses could be sent to each student’s mobile device.  Then students could spend a little time individually or in small groups reading and analyzing these responses.  This would give the students a sense of their peers perspectives on the question and quite possibly enhance subsequent class discussion.  Sending response data back to students for analysis seems like an application of mobile devices with a lot of potential.

Data for Instructional Improvement – Clickers also make it fairly easy to archive student response data for later analysis.  Instructors can use these data over time to improve their clicker questions, perhaps by eliminating or revising answers choices not chosen by many students, or to assess the effectiveness of their teaching methods.

Including mobile devices in classroom response systems has the potential to generate much richer data sets since instructors would be less dependent on multiple-choice questions.  However, richer data sets are often more difficult and more time-consuming to analyze.  I see potential for enlisting students to assist in such efforts, just as students might help analyze these kinds of responses during class, as described above.

Epistemology through Technology – I claim that the ways we use technology in the classroom communicate to students a message about the nature of knowledge and knowledge-generation in our disciplines.  Having students respond to and discuss clicker questions, for instance, tells students that critical reasoning is part of the knowledge-construction process.

If we let our students use mobile devices to connect with the “cloud” during class, searching for answers to their questions and additional resources online, what does that communicate to them about the nature of knowledge?  It was said many times at the conference that with the information available to students on their smart phones (through access to the Internet), there’s little need to have students memorize great amounts of information.  There is, however, a greater need for teaching students to find, understand, apply, and evaluate the information they find online.  Using technologies like mobile devices, we can point students toward those higher-order learning goals.

I think I’ve got one or two more posts in me from the conference.  Stay tuned…

Reference: Lasry, N. (2008). Clickers or flashcards: Is there really a difference? The Physics Teacher 46(4), 242-244.

Summary: Lasry reports the results of a study contrasting the use of clickers and flashcards in facilitating peer instruction in an introductory physics course.  Two sections of the course were taught in the same semester by Lasry.  In one section, students responded to multiple-choice, conceptual understanding questions using clickers; in the other they responded using flashcards.  In both sections, student responses to the questions were used to determine what followed the question–further explanations of the topic at hand by the instructor if most students missed the question, moving on to the next topic if most students answered correctly, or peer instruction otherwise.

Lasry administered the Force Concept Inventory to both sets of students at the start and end of the semester as an assessment of the students’ conceptual understanding.  The normalized gain, (post%-pre%/100-pre%), for the clickers section was 0.486, and for the flashcard section it was 0.520, not a statistically significant difference in this case.

Lasry’s conclusion is that “using peer instruction with clickers does not provide any significant learning advantage over low-tech flashcards.”  He notes that clickers might provide other advantages, such as enabling instructors to analyze student response data for the purpose of improving in-class questions over time and interesting other instructors in experimenting with peer instruction.

Comments: Lasry’s data are certainly interesting and provide some evidence that peer instruction works as well with flashcards as with clickers.  However, he describes the “contributions of clickers” as being “more on the teaching side than on the learning side of the educational equation.”  I find this separation of teaching and learning a little artificial.  The effects on student learning that any instructional technology has depend on how the technology is implemented.  There are a couple of ways of implementing clickers that have the potential to positively impact student learning that don’t appear to be addressed in this study.  These factors might explain the lack of difference in learning gains between the two sections.

For example, since clickers allow an instructor to track individual student responses, they can be used to hold students more accountable for their responses than they would be using flashcards, which has the potential to increase student motivation to participate and engage with questions asked during class.  It’s unlikely that student responses in the clicker section in this study were factored into student grades since tracking individual student responses in the flashcard section would have been impractical and Lasry apparently tried to keep as many aspects of each section constant as he could.  If that’s the case, then students in each section would have been similarly motivated to participate, which might explain the lack of difference in learning gains.  Had student responses to clicker questions been included in student grades in the clicker section, students might have performed better on end-of-semester assessments.

One of the points that Tim Stelzer made in his clicker conference keynote last November was that student participation tended to decrease over time when flashcards were used at the University of Illinois.  I would be interested in finding out if there was any difference in participation in the two class sections in Lasry’s study.  If there was not, then there might have been other factors, such as instructor experience or instructor-student rapport, that kept participation high in the flashcard section and offering another explanation why the clicker section didn’t exhibit greater learning gains.

Another implementation choice that has a potential effect on student learning is “agile teaching,” that is, using response data from clicker or flashcard questions during class to make teaching decisions.  In Lasry’s study, response data were used to determine when to engage students in peer instruction.  Such choices are likely most effective when based on accurate assessments of student learning.  As Stowell and Nelson (2007) showed, the flashcard method can lead to instructors overestimating their students’ comprehension since the method makes it possible for students to see other students’ responses as they select their own responses.  Clickers tend to provide more accurate feedback on student learning since they promote independent answering by students.  It’s possible that in the Lasry study, the flashcard method provided accurate enough assessments for the teaching choices that were made.  Other kinds of agile teaching choices might have benefited from the more accurate data provided by clickers.  The impact of clickers on agile teaching is an issue that hasn’t been studied well to date to my knowledge.

Finally, another way in which clickers might provide benefits over flashcard methods is that clickers make it easy for students to see the distribution of responses to a question.  Flashcards provide this distribution (in rough form) only to the instructor.  Seeing the distribution of responses has a potentially motivating effect on students, particularly when students find out that most of their peers answered a question incorrectly.  It’s unclear from the article the extent to which clicker or flashcard questions were used to generate “times for telling” in this fashion.  It’s possible that in classes where these kinds of questions are asked more regularly, clickers have a bigger impact on student learning because of the easy display of results to the class.