Clyde Herreid of the University of Buffalo’s National Center for Case Study Teaching in Science has received a $500,00 grant from the National Science Foundation to study the use of clicker questions in case studies used in biology classes, with a particular focus on students’ emotional engagement with science.  (The University of Buffalo seems to be a happening place for clickers.  I mentioned their School of Dentistry’s use of classroom response systems on Monday.)  I haven’t blogged about the use of clickers in case studies yet, but I mentioned the practice in my book, citing Herreid’s paper on the topic (Herreid, 2006) as well as Peggy Brickman’s clicker-enhanced case study on DNA fingerprinting.

I’m glad to see an investigation of this pedagogy receive such a sizable grant, particularly given its emphasis on the affective domain.  However, the press release describes students using clickers like a game show buzzer, which is a bit misleading, now that I think about it.  On Jeopardy, for instance, only the fastest responder is allowed to answer, but when using clickers all students have a chance to weigh in.

Meanwhile, the Faculty Technology Center at Louisiana State University recently hosted a presentation on teaching with clickers by biological sciences professor Steve Pomarico.  Pomarico noted that before using clickers in his 250-student course, attendance would vary from 60% to 30% on any given day.  Now that he uses clickers and awards participation points for students responding to his clicker questions, attendance is never below 65%.  He notes that merely using clickers to take attendance is a poor choice, however.  Asking questions that let students test their understanding and provide instructors with useful feedback on student learning is a better use of the technology.

• Herreid, C. F. (2006). “Clicker” cases: Introducing case study teaching into large classrooms. Journal of College Science Teaching, 36(2), 43-4.

And here’s the fifth and final part of student participation week here on the blog.  In some ways, I’ve saved the best for last.  Today’s paper is by Angel Hoesktra of the University of Colorado-Boulder.  When I interviewed Angel for my book, I got a preview of some of her findings and I was impressed by her use of qualitative research methods to explore student perceptions of learning with clickers, so I was expecting interesting and useful results from her published work.  Having finally gotten around to reading that published work, I’m glad to say my expectations have been met.

Reference: Hoekstra, A. (2008). Vibrant student voices: Exploring effects of the use of clickers in large college courses. Learning, Media, & Technology, 33(4), 329-341.

Summary: Hoekstra investigated student perceptions of learning with clickers in multiple sections of a general chemistry course over a period of three years.  Most of the students in the course (apparently between 200 and 300 students per section) take the course to fulfill degree requirements, but typically less than 10% of the students are chemistry majors.  Survey results of these students indicated that 80-90% of them are “concerned about whether or not they will pass the course” and less than 10% of them “would feel comfortable enough to respond to a professor’s question by raising their hand in the large lecture hall.”

Clicker questions were typically used in the course after 10-12 minutes of lecture to assess student understanding of the material just explained.  “Classic” peer instruction was not used.  Instead, each clicker question was asked once and students were encouraged (but not required or prodded) to discuss the questions with their neighbors before voting.  Instructions given to students for these discussions were fairly vague (e.g. “Feel free to work with your neighbors”).  Also, clicker questions were followed by instructor explanations, not classwide discussion.  Correct answers to clicker questions earned three points each; incorrect answers earned one point each.  Clicker questions contributed only 5% of the students’ overall course grades.

Hoekstra investigated student perceptions of clickers through three semesters of student surveys administered using clickers, observations of 27 class sessions, and in-depth interviews with 28 students averaging 56 minutes in length.  Since students in the initial round of interviews had favorable reactions to students, students with less favorable view of clickers were interviewed in the second round.  Observation note and interview transcripts were analyzed through in vivo coding, “a form of open coding designed to allow conceptual categories to emerge from the data.”

Key results are as follows:

• Interviews indicated strongly that students pay more attention during lecture because they know that clicker questions are asked frequently during lecture.
• Students stated in interviews that “they looked forward to times when they were able to talk with their peers” during clicker questions.
• Observations revealed that students frequently voiced their reactions (positive, negative, surprised) to the display of results and answers of clicker questions.
• Interviewees also indicated that the results displays provided them useful and regular feedback on their learning in the course.  Some even indicated that the clicker questions were most helpful when they answered them incorrectly since these were opportunities to resolve misconceptions.
• About 15-20% of students chose not to engage in peer discussion of clicker questions.  The decision to engage was typically influenced by the difficulty of the clicker question and the student’s “affinity for working with others.”  Interestingly, during more difficult clicker questions, female students were more likely to engage in peer discussion than male students, who tended to use these questions as tests of their own understanding.  Other reasons for working alone included not having done the reading before class, a disinterest in hearing possibly incorrect explanations from their peers when the correct explanation would be forthcoming from the instructor, and the fact that accountability for peer interaction was difficult in the large class.
• Most students found the general noise and activity levels in the classroom during peer discussion stimulating.  Some students found it distracting and would have preferred times of quiet as they answered clicker questions.
• Many students felt that clicker questions increased their anxiety levels during the initial weeks of the course but as they became comfortable with the technology and with peer instruction, they found that the clicker questions decreased their overall anxiety about the course.

Hoekstra uses a quote from Trees and Jackson (2007) to summarize many of her findings: “The success of clickers is in many ways dependent on social, not technological, factors.”

Comments: Where to begin?  This study is a rich source of understanding the many ways that students interact with clicker questions and with each other during times of peer instruction.  I’ve briefly summarized the findings above, but the paper includes details, examples, and very illustrative quotations from student interviews.  I realize that some find qualitative research less meaningful than quantitative research, but I think the scope and rigor of Hoekstra’s work adds much credibility to her findings.

Before commenting on a few specific findings, I thought I might connect the teaching environment described in Hoekstra’s paper with some studies I blogged about earlier in “student participation week” here on the blog.  Given the results of Lucas (2009), the vague instructions given to students for discussing clicker questions might have reduced the quantity and quality of student participation in those discussions.  Since the grading scheme for clicker questions was both high-stakes (because correct answers earned three times as many points as incorrect answers) and low-stakes (because clicker questions only contributed 5% of the students’ course grades), it is unclear from the results of James (2006) and Willoughby and Gustafson (2009) whether or not this grading scheme would have enhanced or inhibited student participation.  Hoekstra’s work did not include a control group of any kind, so one can’t say if student participation in the courses she studied was less than or greater than it would have been under different conditions, but her results seem to indicate that most students engaged in meaningful and productive peer discussions in spite of the vague instructions given to them and the somewhat high-stakes grading scheme.

As for Hoekstra’s specific findings, I think they lend support to a statement I made in my book: “Knowing that a deliverable [a clicker question] may at any time be requested from students can help students maintain attention and engagement during a class session.”  Clickers make it easy to request frequent deliverables of students during class, and Hoekstra’s findings indicate that this is an important reason to use clickers.  Hoekstra’s findings also support other reasons I frequently provide for using clickers: sharing the results of a clicker question can enhance student engagement, clicker questions provide students with useful feedback on their learning, and clicker questions can be useful in structuring class time for students.

Hoekstra’s findings about gender and student participation are thought-provoking.  I’ve debated the importance of an initial, independent vote prior to peer instruction time with several instructors who tend to skip this initial vote (particularly my friends in the math department at Carroll College).  Hoekstra’s findings indicated that for difficult clicker questions, female students might not get as much out of an initial, independent vote as male students, whereas male students might not appreciate jumping straight into peer instruction without a chance to respond independently.

This certainly complicates the debate about initial, independent votes, as well as a teacher’s choices during peer instruction times.  I’ve frequently found it to be the case that the students in my class who hesitate to engage in peer discussion are male students (so Hoekstra’s findings ring true to me), and I typically prod these students to discuss clicker questions with peers.  I may not do that as much in the future given these results.

The situation is further complicated by the finding that some students don’t appreciate the noise and activity levels during clicker questions.  This point reminded me of Richard Felder’s work on learning styles.  Felder’s model distinguishes between active learners, those who prefer to learn through discussion and interaction, and reflective learners, those who prefer to think quietly first.  He makes the great point that traditional lectures do a poor job of supporting both types of learners, since they typically provide students with little time for discussion or quiet reflection.  The “classic” peer instruction model serves both types of learners well, however, since students are invited to respond to clicker questions first on their own, then after discuss them with their peers.  (Larry Michaelsen’s team-based learning model works similarly.)

The finding that some students decide not to engage in peer discussions because they want to wait to hear the correct explanation from the instructor was interesting, as well.  Some instructors (for example, Dennis Jacobs, who teaches chemistry at Notre Dame and is profiled in my book) are very intentional about having students surface and debate reasons for and against all of the answer choices to a clicker question.  The methods these instructors use can help students move away from merely taking notes and memorizing explanations and develop critical thinking skills.  It’s possible that the students in Hoekstra’s study who preferred waiting for their instructor’s explanations to peer discussion might have been motivated to engage in peer discussion and thus sharpen their critical thinking skills with more directive instructions and/or requirements on the part of the instructors.

It’s also worth noting that students who weren’t interested in hearing their peers’ incorrect explanations for clicker questions were also concerned about sharing their own incorrect explanations and thus confusing their peers.  Responding to those concerns might be an important part of motivating these students to engage in peer discussions.

I’ll finish my comments with a response to the quote from Trees and Jackson (2007).  I’ll agree that social factors are likely more important than technological factors in the success of teaching with clickers.  I would qualify that statement, however, to note that (a) the technology can enhance those social factors when used well and (b) the teaching choices that instructors make when using clickers can have a significant impact on those social factors.  As a result, we need not think of those social factors as out of our influence as instructors.

That’s the end of student participation week here on the blog.  That’s also likely the end of five-posts-in-a-week here, too!  I’ll return to my usual format next week, although I have found a few more interesting looking articles on student participation to read soon…

Reference: Lasry, N. (2008). Clickers or flashcards: Is there really a difference? The Physics Teacher 46(4), 242-244.

Summary: Lasry reports the results of a study contrasting the use of clickers and flashcards in facilitating peer instruction in an introductory physics course.  Two sections of the course were taught in the same semester by Lasry.  In one section, students responded to multiple-choice, conceptual understanding questions using clickers; in the other they responded using flashcards.  In both sections, student responses to the questions were used to determine what followed the question–further explanations of the topic at hand by the instructor if most students missed the question, moving on to the next topic if most students answered correctly, or peer instruction otherwise.

Lasry administered the Force Concept Inventory to both sets of students at the start and end of the semester as an assessment of the students’ conceptual understanding.  The normalized gain, (post%-pre%/100-pre%), for the clickers section was 0.486, and for the flashcard section it was 0.520, not a statistically significant difference in this case.

Lasry’s conclusion is that “using peer instruction with clickers does not provide any significant learning advantage over low-tech flashcards.”  He notes that clickers might provide other advantages, such as enabling instructors to analyze student response data for the purpose of improving in-class questions over time and interesting other instructors in experimenting with peer instruction.

Comments: Lasry’s data are certainly interesting and provide some evidence that peer instruction works as well with flashcards as with clickers.  However, he describes the “contributions of clickers” as being “more on the teaching side than on the learning side of the educational equation.”  I find this separation of teaching and learning a little artificial.  The effects on student learning that any instructional technology has depend on how the technology is implemented.  There are a couple of ways of implementing clickers that have the potential to positively impact student learning that don’t appear to be addressed in this study.  These factors might explain the lack of difference in learning gains between the two sections.

For example, since clickers allow an instructor to track individual student responses, they can be used to hold students more accountable for their responses than they would be using flashcards, which has the potential to increase student motivation to participate and engage with questions asked during class.  It’s unlikely that student responses in the clicker section in this study were factored into student grades since tracking individual student responses in the flashcard section would have been impractical and Lasry apparently tried to keep as many aspects of each section constant as he could.  If that’s the case, then students in each section would have been similarly motivated to participate, which might explain the lack of difference in learning gains.  Had student responses to clicker questions been included in student grades in the clicker section, students might have performed better on end-of-semester assessments.

One of the points that Tim Stelzer made in his clicker conference keynote last November was that student participation tended to decrease over time when flashcards were used at the University of Illinois.  I would be interested in finding out if there was any difference in participation in the two class sections in Lasry’s study.  If there was not, then there might have been other factors, such as instructor experience or instructor-student rapport, that kept participation high in the flashcard section and offering another explanation why the clicker section didn’t exhibit greater learning gains.

Another implementation choice that has a potential effect on student learning is “agile teaching,” that is, using response data from clicker or flashcard questions during class to make teaching decisions.  In Lasry’s study, response data were used to determine when to engage students in peer instruction.  Such choices are likely most effective when based on accurate assessments of student learning.  As Stowell and Nelson (2007) showed, the flashcard method can lead to instructors overestimating their students’ comprehension since the method makes it possible for students to see other students’ responses as they select their own responses.  Clickers tend to provide more accurate feedback on student learning since they promote independent answering by students.  It’s possible that in the Lasry study, the flashcard method provided accurate enough assessments for the teaching choices that were made.  Other kinds of agile teaching choices might have benefited from the more accurate data provided by clickers.  The impact of clickers on agile teaching is an issue that hasn’t been studied well to date to my knowledge.

Finally, another way in which clickers might provide benefits over flashcard methods is that clickers make it easy for students to see the distribution of responses to a question.  Flashcards provide this distribution (in rough form) only to the instructor.  Seeing the distribution of responses has a potentially motivating effect on students, particularly when students find out that most of their peers answered a question incorrectly.  It’s unclear from the article the extent to which clicker or flashcard questions were used to generate “times for telling” in this fashion.  It’s possible that in classes where these kinds of questions are asked more regularly, clickers have a bigger impact on student learning because of the easy display of results to the class.

Patrick M. Len recently commented on an earlier post about clicker question banks to share links to his blog, where he regularly posts astronomy and physics clicker questions he has used.  Since he was kind enough to share those links and to make his clicker questions available online for others to use, I thought I would take a look at one of his recent articles on classroom response systems.

Reference: Len, P. M. (2007). Different reward structures to motivate student interaction with electronic response systems in astronomy. Astronomy Education Review, 5(2), 5-15.

Summary: In this study, Len explores the impact of two different “reward structures” used for clicker activities in a medium-to-large astronomy survey course at Cuesta College:

• Introductory questions were asked the start of class. These questions were graded on effort, not accuracy of student responses. Students were allowed to discuss their answers with each other before voting. Some did, and some did not.
• Review questions were asked at the end of class. These questions were graded on effort, as well, but if at least 80% of the class answered the day’s questions correctly, those participation points were doubled. This led to some raucous class-wide discussions about the questions.

Sample questions of each type, many of which are conceptual understanding or application questions, are available online in appendices to the article.

Individual students were identified via their responses to a survey as independent workers (“self-testers” in Len’s terminology) or collaborators during the introductory questions. Two pre/post instruments were used to explore differences in these two types of students: the Survey of Attitudes Toward Astronomy (SATA) and the Astronomy Diagnostic Test (ADT).

One key finding of the study was that collaborators (those students who chose to work together to answer the introductory questions) became less confident in their astronomy knowledge and skills and valued astronomy less over the course of the semester, as measured by the SATA. Collaborators also “reported a lower pretest proficiency in science,” according to the ADT, even though they were as accurate in their answers to introductory questions as their self-tester peers.

Len concludes that this one-semester course in astronomy had a significant, negative impact on the beliefs and attitudes about science of these students. He recommends that since these students are “predisposed toward collaborative behavior,” instructors should think carefully about how to use clickers to structure collaborations in ways that increase student confidence and help them value astronomy more.

One other finding was that the helpfulness of the instructor’s lecture in student learning was rated more highly by the self-testers than the collaborators. This complements other findings (Graham, Tripp, Seawright, and Joeckel, 2007) that students who prefer not to participate find clickers less helpful.

Commentary: There’s a lot of data here to make sense of, but I think Len has successfully argued that students who self-report that they aren’t as good at math and science as their peers (a) prefer to collaborate when given the opportunity and (b) became less confident in themselves and less positive toward science during this course. His recommendation to structure collaborative activities (with or without clickers) in ways that are sensitive to these affective issues is a sound one.

Along those lines, it’s possible that the attitudes and beliefs about science held by the collaborator students would have worsened more over the course of the semester had they not been allowed to collaborate on introductory questions. If they had not been allowed to do so, they likely would have done more poorly on these questions (instead of answering them as accurately as their self-tester peers), which in turn would have discouraged them more.

This issue of students in physics and astronomy courses becoming less interested in science because of these courses has been reported elsewhere in the Physics Education Research (PER) community (notably by Carl Wieman’s research groups at the University of Colorado and the University of British Columbia), and I think it’s an important challenge in science education. I’m glad to see this article by Len helping to explore this issue.

Len’s central question–the impact of different reward structures on students in his courses–is only partially answered, in my opinion. It’s clear that the “success-bonus” reward structure used for the review questions encouraged students to collaborate. However, given the way he describes the class environment when students answer his review questions (“Some students shouted for assistance from the rest of the class; others attempted to coach the rest of the class on how to answer, indicating their answer on the overhead projector using fingers, on the screen using laser pointers, or vocally”) it’s unclear the extent to which critical reasoning, as opposed to persuasion and peer pressure, was a factor in these collaborations. An investigation of more structured approaches to implementing this reward structure would be beneficial.

As usual, your comments are welcome!