Jeffrey R. Young's Chronicle article, "Reaching the Last Technology Holdouts at the Front of the Classroom," has apparently struck a nerve among professors, particularly those who are critical of educational technology. As I write this, the article has 59 comments on the Chronicle site, which is far more than most articles receive. Even the graph accompanying the article has received 13 comments!

Since clickers are mentioned in the article and in many of the comments, I thought I would weigh in here on the blog...

First, it's worth noting that Chris Dede, the Harvard University learning technologies professor interviewed for the article, doesn't make the argument that professors who don't use technology are shirking their duties. Several of those who left comments seem to think so, however. For example, here's comment #33 from Emily in NY:

"Dede does nothing in this article but set up a false dichotomy between professors committed to outdated, boring and irrelevant teaching methods and those eagerly embracing the modern technologies that contemporary students crave."

Here's the closest Dede comes to that argument, in the National Educational Technology Plan he helped draft for the US Department of Education in March:

"The challenge for our education system is to leverage the learning sciences and modern technology to create engaging, relevant, and personalized learning experiences for all learners that mirror students' daily lives and the reality of their futures."

Dede's arguments in the Chronicle argument are focused on motivating professors to tap into the latest research on learning and continue to improve their teaching practices over time. From the report he drafted, it's clear he thinks that technology can help with that, but he doesn't seem to be making the argument that professors who don't use technology are irresponsible, just those who stick with the same teaching methods you'd find in a classroom circa 1900. Sure, technology can be a big part of change, but many of the teaching innovations mentioned in the article (such as David Pace's work on enhancing history teaching) don't involve any technology.

Speaking of false dichotomies, however, here's one from comment #18 by user "tee_bee":

"What matters is that students learn--and a skilled teacher with a blackboard is still going to do a far better job than a bozo with some clickers and powerpoint slides."

True, a skilled teacher is going to do a better job than a bozo any day, regardless of technology. But comparing a skilled teacher to a bozo isn't really important here. Might technology (including clickers) help a skilled teacher be even more effective? Yes, that happens. And might technology help a relatively novice teacher become more effective? Yes, that happens, too. Those are the kinds of changes in teaching that are worth thinking about and encouraging, and I think that's a point that Chris Dede would agree with.

How might teaching with clickers help a good teacher be even more effective? Several comments on the Chronicle article were skeptical of clickers' potential for doing this. For example, here's what user "ikant" said in comment #21:

"I'm young, tech-savvy, and pretty unconvinced by this article. I can't speak for all fields, of course, but I'm pretty skeptical that good class discussions and quality writing in the humanities are particularly improved by clickers etc... the heart of what I do is in trying to educe questions, critical thought and excitement about books which students might previously have thought were utterly irrelevant to them, and (my evaluations indicate that) I do this very well with no particular technological bells and whistles in the classroom. Am I missing something?"

I'm glad that this instructor is capable of leading effective class discussions, foster critical thinking, and increase student motivation in the classroom. Let me clear: Doing so is entirely possible without clickers! However, not all instructors are as skilled as "ikant" appears to be and even for instructors like "ikant," it's possible that clickers would enhance an already productive classroom environment. Some examples from past blog posts:

• Case Study: Using clickers to promote critical thinking in text-based, political science courses
• Case Study: Using clickers to teach ethics in a history class
• My Thoughts: Using clickers to foster creativity and collaboration
• My Thoughts: Using clickers to teach critical thinking in the humanities

Here's a similar comment (#26 on the Chronicle site) from user "csgirl":

"The reason I don't use blogs and clickers is that they simply are not appropriate to the material I teach. Clickers in particular are useless to me - I care about the strategies my students are using to solve problems, not whether they can click the right answer in a quiz."

This is a common misconception about clickers, that they're just good for quizzing students basic conceptual understanding and recall. Here's another formulation of it, from user "chewy18" in comment #53:

"They might work well for understanding basic concepts or in preparation for recognition/recall examinations where the test question is a line long and the answer a word or two in length. What about those of us who teach upper division courses where we struggle with students who have not, until they reach senior status, even been exposed to the analytical reasoning process. Suddenly they discover that life is, after all, not a multiple choice test and developing an argument that could go either way, is a requirement. How does that appeal to the clicker technology?"

Sure, clickers work well for assessing basic conceptual understanding and factual recall, but they're useful for teaching at the higher levels of Bloom's Taxonomy, too. Here are some more examples from past blog posts that demonstrate this:

• Case Study: Using clickers to teach difficult material in upper-division physics
• Case Study: Using clickers to teach students about plagiarism in library instructional settings
• Case Study: Using clicker questions without single correct answers in mathematics
• Case Study: Using clickers to teach nursing students to make evidence-based decisions

And for "csgirl," here's a great collection of resources on using clickers and peer instruction in computer science from Daniel Zingaro.

Finally, you can imagine how this comment (#37) from user "fizmath" made me feel:

"The teacher/physician analogy is lousy. We have real data to show that new medical tech benefits patients. You can't say the same about blogs, videoconferencing and those stupid clickers."

(This is a response to Chris Dede's analogy that teachers who don't update their teaching methods over time are akin to physicians who don't update their medical practices over time.)

Want some research? Try these studies, all of which are well designed and support the claim that clickers used in appropriate ways enhance student learning:

• Stowell & Nelson (2007) - Clickers provided instructors with more accurate assessment of student learning during class than other response methods, including a show of hands.
• Yourstone, Kraye, & Albaum (2008) - The use of clickers for end-of-class quizzes improved student exam scores by four points over the use of pencil-and-paper quizzes discussed the next day in class, likely because of the immediate feedback clickers provided to students on their learning.
• Hoesktra (2008) - Clickers helped students be more attentive during class (since they know clicker questions could be asked at any time) and participate in more meaningful ways (both before votes are submitted and after results are displayed).
• Smith et al. (2009) - Students actually learned from each other when discussing clicker questions in pairs prior to voting. They don't "simply choose the answer most strongly supported by neighbors they perceive to be knowledgeable."
• Mayer et al. (2009) - Clickers made it easier for instructors to ask their students questions during class and for students to respond to those questions, leading to improved student learning through better class discussions.

My summary for those skeptical of using clickers in the classroom: Read the literature, find out how those in your discipline are using clickers effectively, and see (preferably by experimentation) if those methods might help you to enhance your teaching, regardless of how effective you are currently as a teacher. If a classroom response system doesn't help you do your job better, then don't use one. They're not for everyone. However, don't write clickers off without first investigating their potential. They're far more useful and versatile that you might think at first.

Image: "Innovation" by Flickr user thinkpublic, Creative Commons licensed

Continuing my reports from the contributed paper session on teaching with clickers I helped coordinate at the Joint Mathematics Meetings back in January…

“Using Prediction and Classroom Voting via Clickers to Address Students’ Overreliance on the Representativeness Heuristic,” Tami Dashley, University of Texas-El Paso [Slides]

Tami Dashley is a graduate student in math education and a student of Kien Lim, one of the organizers of the contributed paper session. She shared some of her thesis research, an investigation into the connection between classroom voting with clickers and certain misconceptions students have about probability. Her work focuses on the representativeness heuristic, which she defines as “determining the likelihood for events based on how well an outcome represents some aspect of its parent population.”

Tami gave the following example: Suppose you toss a coin six times, getting a sequence of heads (H) and tails (T). Which of the following is more likely to occur: TTHHTH or HTTHHH? Someone using the representativeness heuristic would say that TTHHTH is more likely to occur since it includes an equal amount of heads and tails, just like the coin does. The other option includes more heads than tails, so it would not seem as likely to someone using the representativeness heuristic. Actually, both of those outcomes are equally likely (each occurring with probability 1/64), so the representative heuristic is a misleading one in this example.

The issue is that the representativeness heuristic is useful in some cases, but not useful in all cases. The misconception that many students have is that it’s always useful.

How to help students stop over-relying on the representativeness heuristic? Tami has been investigating the use of prediction questions, ones that ask students to predict an outcome or probability without actually computing anything. For example, students might be asked to determine which of several outcomes is most likely to occur. Since students need not be as precise when responding to prediction questions, they have some cognitive processing power freed up to focus on concepts. Clicker questions are a natural match here, since they allow students to commit to their predictions and compare their predictions to those of their peers. Then discussion of the incorrect answer choices provides an opportunity to deal with misconceptions.

Tami conducted her research in a high school setting, using three groups of students. Her “control” group received a lesson exploring the representativeness heuristic that didn’t ask the students to predict any probabilities. A second group was asked several prediction questions but didn’t use clickers to respond to the questions. The third group used clickers to respond to prediction questions during the lesson. Tami used pre- and post-tests to determined the efficacy of these three different lessons.

Tami found that her “control” group did pretty well on the post-test compared to the two experimental groups. However, most of their success came from what she called a “learned response.” In this case, many of the students picked up on the fact that “all of the above outcomes are equally likely” is often the correct answer to questions exploring the representativeness heuristic. (These are what students might call trick questions!) When Tami looked at performance on questions where “all of the above outcomes are equally likely” was, in fact, not the correct answer, the prediction-with-voting group performed better than the control and prediction-only groups.

I was very impressed with Tami’s research design and the subtlety with which she explored student misconceptions in this teaching context. I don’t believe that Tami has published this work yet, but I look forward to reading it when she does.

Image: “Heads and Tails” by Flickr user canonsnapper, Creative Commons licensed

More from my round-up of articles on clickers in the health professions.  This time, another article that doesn’t add much to the literature, but raises an interesting idea.  Again, your comments are invited…

Reference: Williams, B., & Boyle M. (2008). The use of interactive wireless keypads for interprofessional learning experiences by undergraduate emergency health students. International Journal of Education and Development Using ICT, 4(1).

Notes: This article has features results from a survey of students using clickers in a “foundations of health” course taken by emergency health students as well as students majoring in “nursing, physiotherapy, occupational therapy, midwifery, health science, and social work.”  The authors refer to this as interprofessional education, “learning that represents a way of fostering collaborative and seamless, integrated patient-care education.”  I first heard about this approach from my POD Network colleague Marilla Svinicki, who is involved in interprofessional education at the Clinical Education Center affiliated with the University of Texas-Austin.  The CEC is an impressive initiative.

I was interested to hear how clickers would play out in this setting, one featuring students with a diverse set of backgrounds and career goals.  However, the course is a first-year course, so the students weren’t likely to have differentiated themselves yet.  Moreover, there’s no attention paid to these different majors in the survey results that are reported here.  (The survey results are very positive, however, and are in keeping with other surveys I’ve mentioned here on the blog before.)

Have you used clickers in a course that included groups of students from different majors?  I can imagine forming heterogeneous student groups, then giving each group a single clicker as part of small-group activities during class.  How would you teach a course like this?

Update: Just a couple of days after posting this, I learned that Vanderbilt University has an interprofessional program something like the one at UT-Austin.  The Vanderbilt program involves medical and nursing students from Vanderbilt and social work students from Tennessee State University.  The students work together (with mentors) in clinical settings half a day per week and participate in classroom-based learning (using reflective exercises and case study activities) half a day a week, as well.  The rest of the week they participate in their respective programs as normal.  Given the complex nature of health care today, this seems like an incredibly sensible approach to health professions education.

Image: “Colourful Army” by Flickr user maistora / Creative Commons licensed

Back in January I gave a keynote talk at the Health Professionals Education Research Symposium hosted by Nova Southeastern University.  Part of my preparation for that talk included reading some of the articles from related disciplines in my clickers bibliography.  Shortly after the conference, I blogged about one great article about using clickers to promote critical thinking in nursing (Debourgh, 2008), and I’ve been meaning to post some notes about the other articles I read.  Let’s get started…

Reference: Cain, J., & Robinson, E. (2008). A primer on audience response systems: Current applications and future considerations. American Journal of Pharmaceutical Education, 72(4), 77.

Notes: The literature review is the highlight of this article.  It’s not as comprehensive as other lit reviews, but it does a great job of describing a few studies of the use of clickers in the health professions with particularly positive results.  For example, Slain et al. (2004) report that students in clicker sections of two pharmacy courses scored significantly higher on exams than students in non-clickers sections.  Similar results were found by Schackow et al. (2004) in classes for family medicine residents and by Pradhan, Sparano, and Ananth (2005) in classes for obstetrics and gynecology residents.  These references are listed in my bibliography.  Hopefully, I’ll find some time to read and blog about them soon.

Cain and Robinson also include a useful exploration of some of the logistical aspects of teaching with clickers.  Instead of making recommendations, they describe the various choices a department might make and their pros and cons.  They note that any clickers initiative should make sense given an institutions teaching philosophy and technology plan.

For example, a pharmacy school with a mandatory laptop program may highly value an ARS that can utilize laptops as response devices, rather than basing the decision on other features.

They also recommend purchasing a set of clickers available to faculty and staff to check out for one-shot events, like continuing education programs and faculty meetings.

The section on recommendations for future research is a strong one.  Cain and Robinson write, “Any effects from using an instructional medium do not come from the use of the media itself, but from the instructional methods employed.”  That’s something I’ve argued here before.  Cain and Robinson call for research that explores the effects of very particular instructional strategies involving clickers, including strategies useful for facilitating discussion about matters of ethics and morality.  While ethical issues are present in every discipline, they are often particularly important in professional education.

Cain and Robinson make an interesting statement in their section on student considerations: “Finally, appropriate application of the ARS in the curriculum should be defined and encouraged.”  I understand the interest in encouraging instructors to use clickers in appropriate ways.  It’s the “defining” piece that makes me wonder if pharmacy education is a bit more top-down than the kinds of programs you find in, say, colleges of arts and science.  I find that faculty members in undergraduate liberal arts departments tend to have a high degree of autonomy when it comes to their teaching decisions.  They might not be comfortable having appropriate uses of clickers “defined” for them.  Am I reading too much into this word choice?  Does your department (whatever your discipline) set policy on educational technology use?

Image: “Rx, San Antonio, TX” by Flickr user Tadson / Creative Commons licensed

Today, as you may have heard, is Ada Lovelace Day, a day in which bloggers are encouraged to write about women in technology they admire.  The day is named in honor of Ada Lovelace, widely regarded as the world’s first computer programmer.  Lovelace worked with Charles Babbage, mathematician, cryptographer, and organ grinder hater, who designed–but did not build–the world’s first computer, the analytical engine.  (As a result of Babbage’s lack of follow-through, Lovelace’s computer programs were, sadly, not actually implemented in her lifetime.)

I would like to take this opportunity to recognize someone I admire in the world of educational technology: Angel Hoekstra, who recently completed a PhD in sociology at the University of Colorado-Boulder.  For her dissertation research, Angel studied student perspectives on learning with classroom response systems in large chemistry courses.  Her first paper on this work came out in 2008:

• Hoekstra, A. (2008). Vibrant student voices: Exploring effects of the use of clickers in large college courses. Learning, Media, & Technology, 33(4), 329-341.

From the abstract:

This study investigates social, educational, and emotional effects of the use of SRSs–clickers–at the University of Colorado at Boulder.  Methods include participant observation, survey data from over 2000 students enrolled in three semesters of General Chemistry, and in-depth interviews exploring the nature of student experiences with clickers.

Angel’s use of not one, but three interesting methods of surfacing student uses of and perspectives on clickers–surveys, classroom observations, and in-depth interviews–give her findings a great deal of weight.  I’m a great fan of this kind of qualitative research.  I find it’s often more useful than the kind of quasi-control group, quantitative research often conducted around teaching with clickers–more useful in the sense that it provides a richer understanding of how students learn both with and without clickers.  Dr. Hoekstra’s work is a significant contribution to the literature on classroom responses in particular and educational technology in particular.  (More of my thoughts on this paper are available in an earlier blog post.)

Beyond her dissertation work, Angel also uses clickers in the sociology courses she teaches.  Her recent paper (with Stefanie Mollborn) on the use of clickers in sociology education is the first and, so far, only item in the “sociology” section of my clickers bibliography.

• Mollborn, S., & Hoekstra, A. (2010). “A meeting of minds”: Using clickers for critical thinking and discussion in large sociology classes. Teaching Sociology, 38(1), 18-27.

Angel will be presenting her dissertation research at the upcoming clickers conference at the University of Louisville, June 4, 2010.  Here’s her abstract:

Drawing from five years of research into the effects of clicker use in higher education, this presentation will make use of quantitative and qualitative data to illustrate the practical experiences of students and teachers who use clickers.  The presentation will compare best practices for clicker use across disciplines, with a focus on effective pedagogy in natural and social science courses.  Data from four disciplines (chemistry, astronomy, sociology, journalism) will be used to illustrate the potential of clicker technology for fostering increased engagement, conceptual understanding, cooperation, and solidarity, while engaging intermediate and experienced clicker users in general and discipline-specific strategies for clicker use.

If you’re heading to the Louisville conference, I encourage you to hear from Angel about her work!  And if you’re not going to Louisville, her talk is another great reason to go.

I’m glad that Angel is bringing a sociologist’s perspective to the growing body of research exploring the effects of teaching with classroom response systems, and I look forward to reading and hearing about her future work in this area.

For more Ada Lovelace Day posts, see the list of posts (over 2000 and still going–March 24th has a few more hours in it depending on your time zone!) at FindingAda.com.  Here are a few posts from blogs I follow:

• ProfHacker – “Today Is Ada Lovelace Day“
• Partially Ordered Thoughts (Mitch Keller) – “Women in Science & Technology: Helen E. Grenga“
• Old is the New New (Rob MacDougall) – “Lovelace and Somerville“
• Great Sufficiency (Michelle M. Hudson) – “Ada Lovelace Day 2010 – Mary Lou Jepsen“
• Future Tense (podcast by Jon Gordon) – “Bloggers Pause to Honor Ada Lovelace“
  
• 2D Goggles (Sydney Padua) – “Ada Lovelace Day!“
  

Reference: DeBourgh, G. A. (2008). Use of classroom “clickers” to promote acquisition of advanced reasoning skills. Nurse Education in Practice, 8(2), 76-87.

Summary: Gregory DeBourgh provides a useful introduction to using clickers in nursing education, focusing on pedagogical strategies that use clickers to promote critical thinking.  His exploration of critical thinking in the context of nursing education is particularly interesting.  Here’s a sample:

“Reasoning is about using intellectual power to draw conclusions, form judgments, and make inferences based on evidence, education, and experience… The practical significance of acquiring skill in advanced reasoning is to move to the level of predictive clinical reasoning which enables one to anticipate both ideal and likely outcomes given a set of data.”

DeBourgh argues that using classroom response systems to engage students in high-level questions is an effective strategy for developing their critical thinking skills.  He supports this assertion by drawing on the literature on the roles of feedback and questioning in learning and by sharing concrete examples of clicker uses in nursing education.

Included are three sample questions, including a “one-best-answer” question that asks students to identify the likely cause of a particular symptom shown by a patient in a case study.  DeBourgh endorses the use of such questions since they better represent situations students are likely to encounter in clinical settings where they must deal with ambiguity.  He also suggests asking question sequences based around patient cases that “change the focus to add new variables,” noting that doing so also reduces the cognitive load students experience when familiarizing themselves with a new case.

DeBourgh makes a good argument for using clicker questions to model critical thinking skills for students:

“Anticipate likely incorrect responses and prepare ‘talking points’ for discussion as this facilitates ‘thinking on your feet’ and makes more visible to students how an expert uses heuristics, reasoning, and refined problem-solving skills to gain command of a clinical situation.”

Asking questions designed to provide an opportunity for the instructor to model critical thinking is one instance of many DeBourgh describes of crafting questions to meet particular teaching and learning objectives.  In doing so, DeBourgh draws on articles by Ian Beatty on good question design, transferring Ian’s advice to the context of nursing education.

DeBourgh also points out that clicker questions embedded in PowerPoint can be particularly useful in nursing, a field which frequently uses pictures, diagrams, sound clips, and video–media that can also be embedded in PowerPoint.  He also notes that nursing courses often involve discussion of nursing ethics and student opinions about ethical decisions, topics that lend themselves well to clicker questions.

The article also includes results from a study survey about clicker use.  Student responses to rating questions are summarized, and student responses to open-ended questions are presented, as well.

DeBourgh ends with a few challenges involved in teaching with clickers, two of which are particularly significant.  He notes that since instructors can track student performance in a class on a daily basis, expectations for students are raised, which is not popular with all students.  DeBourgh also speaks to the increased expectations for instructors:

“The greatest challenge is the new role for faculty to plan the curriculum and instruction around ‘deep comprehension’ rather than ‘covering content’ using a traditional lecture format.”

Comments: I read this article in advance of my presentation at the Health Professional Educational Research Symposium earlier in the month, and I was particularly impressed with Gregory DeBourgh’s eloquence in describing critical thinking in the context of nursing education and in describing ways that clicker pedagogies can foster those critical thinking skills.

As I’ve tried to capture above, DeBourgh describes a variety of ways of using clickers in nursing education, and he included one approach that was entirely new to me, one inspired by the 50-50 option in the television game show Who Wants to Be a Millionaire? DeBourgh suggests that before the correct answer to a clicker question is revealed to students, an instructor might ask the students which answer choices should be eliminated.  It’s a little unclear how DeBourgh implements this, but I can even imagine setting up a multiple-mark question with four answer choices, then asking students not to select the one correct answer but to select two incorrect answers.  This would offer a nice change of pace in question format and would help students focus on more than just the correct answer.  It’s often useful for students to consider why some answer choices are plausible on the surface but actually incorrect.

Hopefully it’s also above that DeBourgh puts an emphasis on teaching with case studies (multimedia case studies, at that) in his article.  I understand that case study methods are perhaps more common in nursing than they are in other disciplines, and I appreciated reading this article as a way to better understand why that was the case.  DeBourgh’s comments about using clickers for discussing ethics also helped me better understand the disciplinary context here.

If you’re a nursing educator, please share a thought or two about using clickers in your field in the comments section!

Update: Greg DeBourgh emailed me and clarified his 50-50 technique.  Here’s what he said:

I display the potential four-answers to a given question, then before the students “vote” with their clickers, I ask for a volunteer or select a student at random (my clicker system has this feature) and ask the student to eliminate 2 of the 4 potential answers and to explain why they are eliminating these two. This speaking out loud of their rationale for eliminating two of the potential answers that are not related to the question strengthens the students’ reasoning skills. They actually get quite good at it. If the student I called upon to answer hesitates or is reluctant to speak, I invite them to choose a “consultant” in the room to help them out. I hope this clarifies a bit for you.

I asked Greg what he does if the student eliminates the correct answer.  Here’s his response:

If the student eliminates one of the correct answers, it is still learning, and so I ask “does everyone agree with the 50/50 elimination?” If someone objects, I ask for their rationale. If no one objects, I just let the process go and during the “reveal and rationale” we talk about why each answer is incorrect or correct.

Thanks, Greg, for this clarification, and for this great use of clickers.

Reference: James, Mark C., Barbieri, F., & Garcia, P. (2008). What are they talking about? Lessons learned from a study of peer instruction.  Astronomy Education Review, 7(1).

Summary: This study is a follow-up to James (2006), a study comparing the effect on student discourse during clicker questions of grading incentive.  In the earlier study, student conversations were audio-recorded in two different astronomy course, each taught by different instructors and each involving different grading schemes for clicker questions.  The main finding of the earlier study was that a low-stakes grading scheme (one in which incorrect answers counted as much as correct answers) encouraged richer student-to-student discussions prior to voting on clicker questions than a high-stakes grading scheme (one in which incorrect answers counted only a third as much as correct answers).

A key drawback to James’ earlier study was that the two courses being compared were different in significant ways other than the grading scheme used for clicker questions.  In particular, they had different topics and different instructors.  That drawback has been mostly eliminated in the current study by James, Barbieri, and Garcia.  In this study, the same instructor taught the same course, an introduction to astronomy course with about 180 students, in two consecutive semesters.  In both semesters, clicker questions contributed 12.5% of the students’ overall course grades.  In the first semester, incorrect answers counted one-third as much as correct answers, but in the second semester, incorrect answers counted 90% as much as correct answers.

The instructor used a version of the standard peer instruction technique.  Students were not asked to vote on clicker questions independently, but were instead asked to discuss the questions in pairs prior to voting.  Random samples of students in each semester were audio-recorded during these pair discussions throughout the two semesters.  The audio-recordings were analyzed in two different ways to measure “discourse bias,” “the difference between the fractional contributions to a conversation between partners.”  For instance, if one partner contributed 70% of the time and the other contributed 30% of the time, then the pair’s discourse bias would be 40%.

First, each idea shared by the students during the discussions was coded according to ten categories, including categories such as restating question elements, stating answer preferences, and providing justifications.  (One side finding was that there was no correlation between “type” of clicker question and the nature of the ideas shared by students during the discussion.)  Second, the total number of words produced by each student during the discussions was counted.  Both techniques provided measured of each student’s contribution to the discussions.

The results strongly indicated that the low-stakes grading scheme encouraged more balanced participation by students during pair discussions.  For example, when using the first measure of discourse bias (counting ideas), the average bias for the high-stakes class was 33.2%, whereas the average bias for the low-stakes class was 19.5%.  That is to say that each pair of students engaged in a conversation more dominated by one of the students in the high-stakes class.  The second measure of discourse bias (counting words) provided similar results-an average bias of 39.8% in the high-stakes class and 26.6% in the low-stakes class.

The authors also note that the low-stakes grading scheme promoted more independent student responses to clicker questions following pair discussions.  In the high-stakes class, only 7.6% of the time did two partners submit different answers to clicker questions, whereas in the low-stakes class, this occurred 17.1% of the time.  The authors conclude from this that in the high-stakes class, students’ concern for earning points motivated them to submit their partners’ answers to clicker questions even when they didn’t really believe those answers.

Comments: This study improves on James’ earlier study and provides persuasive evidence that low-stakes grading schemes for clicker questions promotes more meaningful student participation in small group discussions prior to voting.  True, this wasn’t a double-blind, randomized control group experiment (in which students were randomly assigned to the two grading schemes and the instructor didn’t know which grading scheme would be used with each group of students), but such experiments are practically impossible to implement in educational settings.  Short of that “gold standard,” this is a very well-designed and persuasive study, in part because many of the possibly confounding variables in the earlier study were eliminated and in part because of the use of direct, qualitative measures of student participation.

Willoughby and Gustafson (2009) conducted a similar study in physics courses, audio-recording student discussions in some sections and not in others.  They found that students in the sections that were not audio-recorded “block-voted” more when high-stakes grading schemes were used for clicker questions and less when low-stakes grading schemes were used.  In the sections where audio-recorders were used, there was no statistically significant difference in block-voting rates.  They concluded that the presence of the audio-recorders might have influenced student voting behaviors (an example of the Hawthorne effect).  If true, it’s possible that the difference James, Barbieri, and Garcia found in block-voting behavior might have been even greater had audio-recorders not been used-all the more reason to use low-stakes grading schemes when using clickers for formative assessment.

I would have liked to have seen a little additional information provided in the article about the use of clicker questions in these courses.  Were students asked to vote on their own before discussion the clicker questions in pairs?  (I don’t think they were, but this isn’t stated in the article.)  Also, what instructions were given to students prior to the peer discussion times?  I’ve seen some evidence (Lucas, 2009) and heard some advice (from Doug Duncan) that the instructions given to students prior to peer instruction can affect the quality of the discussions.  And while it was found here that the type of clicker question did not correlate with the kinds of ideas shared during peer instruction, it would have been informative to know what kinds of clicker questions were used.

What’s not directly address in this article, however, is the assumption that more student participation during small-group discussion of clicker questions leads to greater student learning.  This was an issue I raised in my comments on James’ earlier study, which included data on student performance on final exams.  In response to that study I asked if greater class participation led to greater student learning or if students who knew the material better simply dominated class discussions.  While it’s possible that the latter is true, evidence from non-clicker studies strongly suggests that more active participation in class discussions leads to greater student learning.  I wish this assumption (that participation leads to student learning) had been stated as such in the article.

The takeaway here is that low-stakes grading schemes for clicker questions leads to greater student participation and clicker questions results that more accurately reflect students’ actual understanding (or lack of understanding).  These results have important implications for instructors using clickers to motivate student participation and inform agile teaching choices.

Reference: Freeman, M., Blayney, P., & Ginns, P. (2006). Anonymity and in class learning: The case for electronic response systems. Australasian Journal of Educational Technology, 22(4), 568-580.

Summary: In an effort to investigate the importance to students of the anonymity afforded by classroom response systems, Freeman, Blayney, and Ginns surveyed the 139 students in an introductory management accounting course at the end of the term. Notably, the majority of students in this course were female (73%), 20-22 years old (93%), with non-English speaking backgrounds (82%).

Each three-hour class session in this course featured a number of “formative, mainly rules based, multiple choice questions.”  Students responded to these questions via clickers or hand-raising in alternating class sessions and were encouraged, but not required, to discuss their answers with their peers before responding.  The instructor would announce the correct answer to each question immediately after the distribution of responses was shared with the class and then practice “agile teaching” by using the results of the question to guide subsequent lecture and discussion.  This alternating system of response methods ensured that students were given opportunities to experience a response method that allowed students to remain anonymous (clickers) and one that did not (hand-raising).

Sixty-eight percent of students agreed or strongly agreed with the statement “I preferred answering in-class questions when my answers were anonymous to the instructor.”  Almost as many students (62%) preferred answering questions when their answers were anonymous to their peers.  About the same number of students (63%) agreed or strongly agreed that “anonymity was more important with in-class questions when [they] were uncertain about the answer.”

Additionally, students were asked to rank in order of preference four potential response methods-clickers, hand-raising, volunteering to respond when they knew the answer, and “cold-calling.”  Average student rankings of these four methods indicated a preference for anonymity, as the methods were ranked (on average) in the same order as I just listed them.

The authors conclude from these survey data that students value the anonymity (peer-to-peer anonymity and student-to-instructor anonymity) enabled by a classroom response system.  Since other response methods, such as hand-raising and response cards, are less anonymous, this argues for the use of clickers.

The authors make a couple of additional points about the potential interplay of culture and use of clickers.  One is that peer instruction was not as highly valued by these students as other aspects of learning with clickers, according to survey results.  It’s possible that the cultural diversity of the students (as measured by proxy by the percentage of students with non-English-speaking backgrounds) made peer instruction less useful for these students than for more homogeneous students surveyed in other studies who found peer instruction more useful.

The other is that demographic variables, including native language, had no significant impact on the student responses to survey questions, particularly the questions about the importance of anonymity.  “Using first language as a rough proxy for culture, and in particular openness to criticism, these results might be seen to contradict Banks (2003) who suggested that cultural background could impact preference for ERS usage.”  The authors suggest further study of the role of culture in students’ valuing of anonymity.

Comments: For a relatively simple survey-based study, this article raises some interesting questions.  For instance, given that the instructor of this course practiced agile teaching, altering his instruction based on the distribution of student responses to in-class questions, I would think that response methods that generated more honest responses from students would lead to more useful agile teaching.  That is to say, if students were less likely to indicate confusion about a question when the hand-raising method was used (because their answers would not be anonymous to their peers), then the instructor might overestimate his students’ comprehension of the topic at hand when relying on the response distribution and subsequently spend less time on the topic than might actually be warranted.  If true, then students who prefer anonymous response methods and who have instructors who practice agile teaching might benefit more from an anonymous response method than students whose instructors do no practice agile teaching.  Since I have yet to read a study that compared use of clicker questions with and without agile teaching, it’s unclear at this point how important this issue might be.

The questions raised about the interface of culture, participation, criticism, and anonymity are also very interesting ones.  They are ones that haven’t been on my radar until recently, when I heard a presentation by Parvanak “Pary” Fassihi, who described her use of clickers to engage students in English-as-a-Second-Language (ESL) classes at Boston University and the Boston campus of the Showa Women’s University (of Japan).  She shared that some of her students come from cultures where it is seen as impolite to disagree with others publicly.  These students are often very hesitant to engage in the small-group and classwide discussion of clicker questions that Pary tries to generate in her courses.  (I’ll note here that I interviewed Pary for my book.  She had a lot of interesting things to say about the role of clickers in her language instruction courses, but this observation about cultural views on public disagreement didn’t come up in our interview.)

Pary’s observation is consistent with the finding by Freeman, Blayney, and Ginns that their students, many of whom came from diverse cultures given that few of them spoke English as a first language, did not value peer instruction as highly as might be expected.  I imagine it is also consistent with Banks’ suggestion, mentioned in the article, that “cultural background could impact preference” for clicker usage.  I haven’t read the Banks article, but I hope to do so soon and review it here on the blog.

One criticism I would have of the article at hand is that comparing the clicker and hand-raising response methods doesn’t quite isolate the effect of anonymity since clickers offer a number of advantages over hand-raising that might be at play here.  Given the nature of the survey questions used, this isn’t a significant issue, since student perception of anonymity was the focus, not a control-group-study comparison of two difference response methods.  However, it is possible that the students’ preference for clickers over hand-raising over hearing from student volunteers could be an indication not of their preference for anonymity but of their preference for response methods that encourage more complete and/or more independent participation and/or greater accountability for participation.  Thus, I would argue that those data in particular don’t necessarily imply that students value anonymity when responding to in-class questions.

(Cold-calling also leads to more complete and independent participation and greater accountability for participation since students have to stay on their toes in case they are called upon.  However, I’m comfortable arguing that the stress generated by this method would outweigh any perceived benefits regarding participation or accountability.)

A better comparison might be between having students respond via clickers with and without a display of individual participant responses.  (Many classroom response systems have the ability to display on screen not only a histogram showing the distribution of responses but also a list of individual students and their particular responses.)  That would better isolate the anonymity factor from other advantages of clickers over alternate response methods.  It’s not a perfect solution, in part because students in a large class aren’t likely to know each other’s names and so might not be bothered by having their names displayed next to their responses and in part because this is a feature of classroom response systems that is infrequently used by instructors so its use in this context might seem too artificial.  Also, this plan would not isolate peer-to-peer anonymity from student-to-instructor anonymity.  However, I think it would yield more relevant data than comparing clicker use with hand-raising.

Reference: King, D. B., & Joshi, S. (2008). Gender differences in the use and effectiveness of personal response devices.  Journal of Science Education and Technology, 17(6), 544-552.

Summary: In this paper, King and Joshi present the results of a study of student participation and performance in two semesters of a chemistry course for engineering students with a particular focus on gender differences.  In the first semester, one section of the course used clickers without including clicker questions in the students grades in any way, while the other two sections did not use clickers at all.  In the second semester, only one section of the course was offered.  Clickers were used in this section, and clicker questions contributed to a participation grade for the students (5% of the overall course grade, with full credit awarded to students who answered at least 75% of the clicker questions throughout the semester, correctly or not).

The authors found that in the first semester’s clicker section, when clicker questions were not included in student grades, there was a statistically significant difference in the response rates of male and female students.  Female students answered 62% of clicker questions on average, whereas male students only answered 48% of questions.  In the second semester, when clicker questions were included in students’ grades, there was no significant difference in the response rates of male and female students.

The authors also found that students who were “active participators” (those who answered at least 75% of clicker questions in a semester) had higher final grades than students who were not active participators.  This difference was statistically significant for male students, but not for female students, however.  These results suggest that although male students participated less frequently than female students, male students who were active participators benefitted more from participation via clicker questions than female students.

The differences in final grades between active and non-active participators were consistent whether or not clicker questions were graded.  The authors conclude from this that “while the average grade improvement was the same during each term, the benefit of requiring clicker usage is that a greater number of students receive this benefit when participation is tied to their course grade.”  This argues for grading clicker questions, particularly for male students, who not only participate less when clicker questions aren’t graded, but also appear to benefit more from being active participators.

The authors also looked at student performance on final exam questions that were “related” to clicker questions asked during the semester.  As the authors expected, students who answered clicker questions correctly tended to do better on related final exam questions.  More surprising was that students who answered clicker questions incorrectly also did better on final exam questions than student who didn’t respond to the related clicker questions at all, indicating that class participation via clicker questions helped prepare students for exams.

It is worth noting that the correlation between answering clicker questions incorrectly and doing well on related final exam questions was not observed in the second semester clickers were used.  Recall that in the second semester, clicker questions were included in students’ grades.  The authors argue that this led some students to simply click in to earn participation points without really trying to answer the clicker questions.  Thus including clicker questions in students’ grades is likely to encourage more students to participate, but enough non-participating students are likely to respond to clicker questions (incorrectly in most instances) that the impact of clicker questions on student performance is harder to see in the student data.

Comments: In their literature review, the authors note that there is evidence that teachers tend “to ask questions of and praise male students more than female students.”  This potential bias is another good reason to use the pick-a-random-student feature of some classroom response systems.  Having the system select at random a student from among those who responded to a clicker question prevents this kind of bias.

King and Joshi’s main results–that students who respond to clicker questions (correctly or not) benefit from the participation and that grading clicker questions (on effort) leads to more students (particularly male students) participating in this useful way–are interesting and persuasive.  The authors did a good job of blending a quasi-experimental design (grading clicker questions in one semester, not grading them in another semester) with data collected within a single semester to argue these points.

Given the authors’ comments about students in the second semester just clicking in for participation points, I wonder if asking students for their confidence in their answers would have helped parse out these students to yield more meaningful data from that semester.  For instance, if students who weren’t really trying to answer clicker questions could be persuaded to signify that they had low (and not high) confidence in their answers, one could remove responses with low confidence from the data set to see if answering clicker questions incorrectly still had a positive correlation with success on related exam questions.

I should also add that it was unclear from the article what kinds of clicker questions were used in these courses–difficult ones, easy ones, recall questions, conceptual understanding questions, etc.  It was also unclear if students were asked to discuss clicker questions in small groups or as a class or if the instructors practiced “agile teaching,” responding in meaningful ways to the distribution of responses for particular clicker questions.  More description of these contextual factors would give the authors’ results more meaning.

King and Joshi’s results about gender–that male students tend to participate less frequently than female students (when not motivated by grades) and that male students who do participate benefit more in terms of performance on final exams–are also very interesting.  I’m reminded of findings shared by Hoekstra (2008) that male students tend to prefer to respond to clicker questions on their own, whereas female students tend to collaborate with other students prior to answering when given the option.  Hoekstra found that the male students liked to test themselves by seeing if they could answer a clicker question without external help.  It’s unclear from the King and Joshi article if students were allowed or required to discuss clicker questions prior to responding to them, but if students were given the option of responding on their own, it might be male students who self-test via clicker questions (whether voluntarily or when prompted to do so by including clicker questions in course grades) benefit from doing so, leading to greater learning gains for these participating male students.

Reference: Crouch, C. H., & Mazur, E. (2001). Peer instruction: Ten years of experience and results. American Journal of Physics, 69(9), 970-977.

Summary: In this now-classic article, Catherine Crouch and Eric Mazur present data on ten years of the use of peer instruction in introductory physics courses.  Included is a description of Mazur’s teaching practices for these courses, including ConcepTests (multiple-choice questions that help students develop conceptual understanding independent of computational skills), pre-class reading quizzes (used to motivate students to read their textbooks before class, allowing Mazur to shift the transfer of information outside of class, freeing up classtime to work on the assimilation of information), and peer instruction with and without clickers.

For assessment, Crouch and Mazur compare student performance on pre- and post-tests–the Force Concept Inventory (FCI), a widely used multiple-choice test of conceptual understanding in first-semester physics–before Mazur began using peer instruction and after.  They use normalized gain as their metric, which is determined by the formula (post-pre)/(100%-pre).  Thus, if a student scores a 70% on the pre-test and an 80% on the post-test, their normalized gain is (80-70)/(100-70), which is approximately 0.33.  Another student who moved from a 90% to a 95% would have a gain of 0.5, indicating that the student gained 50% of the improvement s/he could have gained from pre-test to post-test.

Using normalized gain on the FCI as a metric enables Crouch and Mazur to make comparisons to national data.  In Richard Hake’s 6000-student study of “traditional” and “interactive” physics courses, the average normalized gain for students in traditional courses was 0.23, whereas the average for students in interactive courses was 0.48, a very significant difference.  The semester before Mazur started using peer instruction, his normalized gain was 0.25, consistent with Hake’s findings for “traditional” lecture courses.  The first semester Mazur used peer instruction, his normalized gain was 0.49, also consistent with Hake’s findings.

Perhaps most interesting is that as Mazur gained experience with these teaching methods (and made refinements to them, like the replacement of flash cards with clickers in his second year using peer instruction), his normalized gain increased by several percentage points each year, hitting 0.74 the sixth time he implemented peer instruction.  Thus he was, in a sense, three times as effective in helping his students master concepts in first-semester physics.

Comments: I tend to review more recent articles on teaching with clickers on this blog, but I couldn’t resist posting something about this classic article.  Mazur’s peer instruction technique is the most commonly used approach to teaching with clickers, and that’s in large part to the persuasiveness of the data he has collected on its impact in his courses.  This article presents solid evidence that having students read their textbooks before class and grapple with tough conceptual understanding questions during class in small groups is a superior way to teach first-semester physics.

It’s also worth noting that Mazur’s normalized gain improved over time.  I’ll occasionally read an article by an instructor who taught a section of a course with clickers and a section without and student performance in the two sections to find that using clickers had little or no impact on student performance.  These experiments often have a variety of design problems, but, regardless, it’s important to note that instructors can improve in their use of a particular teaching method over time.  Expecting great results the first or second time out is sometimes unrealistic, and big learning gains are sometimes only possible after a few semesters experience.