Today, as you may have heard, is Ada Lovelace Day, a day in which bloggers are encouraged to write about women in technology they admire.  The day is named in honor of Ada Lovelace, widely regarded as the world’s first computer programmer.  Lovelace worked with Charles Babbage, mathematician, cryptographer, and organ grinder hater, who designed–but did not build–the world’s first computer, the analytical engine.  (As a result of Babbage’s lack of follow-through, Lovelace’s computer programs were, sadly, not actually implemented in her lifetime.)

I would like to take this opportunity to recognize someone I admire in the world of educational technology: Angel Hoekstra, who recently completed a PhD in sociology at the University of Colorado-Boulder.  For her dissertation research, Angel studied student perspectives on learning with classroom response systems in large chemistry courses.  Her first paper on this work came out in 2008:

• Hoekstra, A. (2008). Vibrant student voices: Exploring effects of the use of clickers in large college courses. Learning, Media, & Technology, 33(4), 329-341.

From the abstract:

This study investigates social, educational, and emotional effects of the use of SRSs–clickers–at the University of Colorado at Boulder.  Methods include participant observation, survey data from over 2000 students enrolled in three semesters of General Chemistry, and in-depth interviews exploring the nature of student experiences with clickers.

Angel’s use of not one, but three interesting methods of surfacing student uses of and perspectives on clickers–surveys, classroom observations, and in-depth interviews–give her findings a great deal of weight.  I’m a great fan of this kind of qualitative research.  I find it’s often more useful than the kind of quasi-control group, quantitative research often conducted around teaching with clickers–more useful in the sense that it provides a richer understanding of how students learn both with and without clickers.  Dr. Hoekstra’s work is a significant contribution to the literature on classroom responses in particular and educational technology in particular.  (More of my thoughts on this paper are available in an earlier blog post.)

Beyond her dissertation work, Angel also uses clickers in the sociology courses she teaches.  Her recent paper (with Stefanie Mollborn) on the use of clickers in sociology education is the first and, so far, only item in the “sociology” section of my clickers bibliography.

• Mollborn, S., & Hoekstra, A. (2010). “A meeting of minds”: Using clickers for critical thinking and discussion in large sociology classes. Teaching Sociology, 38(1), 18-27.

Angel will be presenting her dissertation research at the upcoming clickers conference at the University of Louisville, June 4, 2010.  Here’s her abstract:

Drawing from five years of research into the effects of clicker use in higher education, this presentation will make use of quantitative and qualitative data to illustrate the practical experiences of students and teachers who use clickers.  The presentation will compare best practices for clicker use across disciplines, with a focus on effective pedagogy in natural and social science courses.  Data from four disciplines (chemistry, astronomy, sociology, journalism) will be used to illustrate the potential of clicker technology for fostering increased engagement, conceptual understanding, cooperation, and solidarity, while engaging intermediate and experienced clicker users in general and discipline-specific strategies for clicker use.

If you’re heading to the Louisville conference, I encourage you to hear from Angel about her work!  And if you’re not going to Louisville, her talk is another great reason to go.

I’m glad that Angel is bringing a sociologist’s perspective to the growing body of research exploring the effects of teaching with classroom response systems, and I look forward to reading and hearing about her future work in this area.

For more Ada Lovelace Day posts, see the list of posts (over 2000 and still going–March 24th has a few more hours in it depending on your time zone!) at FindingAda.com.  Here are a few posts from blogs I follow:

• ProfHacker – “Today Is Ada Lovelace Day“
• Partially Ordered Thoughts (Mitch Keller) – “Women in Science & Technology: Helen E. Grenga“
• Old is the New New (Rob MacDougall) – “Lovelace and Somerville“
• Great Sufficiency (Michelle M. Hudson) – “Ada Lovelace Day 2010 – Mary Lou Jepsen“
• Future Tense (podcast by Jon Gordon) – “Bloggers Pause to Honor Ada Lovelace“
  
• 2D Goggles (Sydney Padua) – “Ada Lovelace Day!“
  

Reference: King, D. B., & Joshi, S. (2008). Gender differences in the use and effectiveness of personal response devices.  Journal of Science Education and Technology, 17(6), 544-552.

Summary: In this paper, King and Joshi present the results of a study of student participation and performance in two semesters of a chemistry course for engineering students with a particular focus on gender differences.  In the first semester, one section of the course used clickers without including clicker questions in the students grades in any way, while the other two sections did not use clickers at all.  In the second semester, only one section of the course was offered.  Clickers were used in this section, and clicker questions contributed to a participation grade for the students (5% of the overall course grade, with full credit awarded to students who answered at least 75% of the clicker questions throughout the semester, correctly or not).

The authors found that in the first semester’s clicker section, when clicker questions were not included in student grades, there was a statistically significant difference in the response rates of male and female students.  Female students answered 62% of clicker questions on average, whereas male students only answered 48% of questions.  In the second semester, when clicker questions were included in students’ grades, there was no significant difference in the response rates of male and female students.

The authors also found that students who were “active participators” (those who answered at least 75% of clicker questions in a semester) had higher final grades than students who were not active participators.  This difference was statistically significant for male students, but not for female students, however.  These results suggest that although male students participated less frequently than female students, male students who were active participators benefitted more from participation via clicker questions than female students.

The differences in final grades between active and non-active participators were consistent whether or not clicker questions were graded.  The authors conclude from this that “while the average grade improvement was the same during each term, the benefit of requiring clicker usage is that a greater number of students receive this benefit when participation is tied to their course grade.”  This argues for grading clicker questions, particularly for male students, who not only participate less when clicker questions aren’t graded, but also appear to benefit more from being active participators.

The authors also looked at student performance on final exam questions that were “related” to clicker questions asked during the semester.  As the authors expected, students who answered clicker questions correctly tended to do better on related final exam questions.  More surprising was that students who answered clicker questions incorrectly also did better on final exam questions than student who didn’t respond to the related clicker questions at all, indicating that class participation via clicker questions helped prepare students for exams.

It is worth noting that the correlation between answering clicker questions incorrectly and doing well on related final exam questions was not observed in the second semester clickers were used.  Recall that in the second semester, clicker questions were included in students’ grades.  The authors argue that this led some students to simply click in to earn participation points without really trying to answer the clicker questions.  Thus including clicker questions in students’ grades is likely to encourage more students to participate, but enough non-participating students are likely to respond to clicker questions (incorrectly in most instances) that the impact of clicker questions on student performance is harder to see in the student data.

Comments: In their literature review, the authors note that there is evidence that teachers tend “to ask questions of and praise male students more than female students.”  This potential bias is another good reason to use the pick-a-random-student feature of some classroom response systems.  Having the system select at random a student from among those who responded to a clicker question prevents this kind of bias.

King and Joshi’s main results–that students who respond to clicker questions (correctly or not) benefit from the participation and that grading clicker questions (on effort) leads to more students (particularly male students) participating in this useful way–are interesting and persuasive.  The authors did a good job of blending a quasi-experimental design (grading clicker questions in one semester, not grading them in another semester) with data collected within a single semester to argue these points.

Given the authors’ comments about students in the second semester just clicking in for participation points, I wonder if asking students for their confidence in their answers would have helped parse out these students to yield more meaningful data from that semester.  For instance, if students who weren’t really trying to answer clicker questions could be persuaded to signify that they had low (and not high) confidence in their answers, one could remove responses with low confidence from the data set to see if answering clicker questions incorrectly still had a positive correlation with success on related exam questions.

I should also add that it was unclear from the article what kinds of clicker questions were used in these courses–difficult ones, easy ones, recall questions, conceptual understanding questions, etc.  It was also unclear if students were asked to discuss clicker questions in small groups or as a class or if the instructors practiced “agile teaching,” responding in meaningful ways to the distribution of responses for particular clicker questions.  More description of these contextual factors would give the authors’ results more meaning.

King and Joshi’s results about gender–that male students tend to participate less frequently than female students (when not motivated by grades) and that male students who do participate benefit more in terms of performance on final exams–are also very interesting.  I’m reminded of findings shared by Hoekstra (2008) that male students tend to prefer to respond to clicker questions on their own, whereas female students tend to collaborate with other students prior to answering when given the option.  Hoekstra found that the male students liked to test themselves by seeing if they could answer a clicker question without external help.  It’s unclear from the King and Joshi article if students were allowed or required to discuss clicker questions prior to responding to them, but if students were given the option of responding on their own, it might be male students who self-test via clicker questions (whether voluntarily or when prompted to do so by including clicker questions in course grades) benefit from doing so, leading to greater learning gains for these participating male students.

And here’s the fifth and final part of student participation week here on the blog.  In some ways, I’ve saved the best for last.  Today’s paper is by Angel Hoesktra of the University of Colorado-Boulder.  When I interviewed Angel for my book, I got a preview of some of her findings and I was impressed by her use of qualitative research methods to explore student perceptions of learning with clickers, so I was expecting interesting and useful results from her published work.  Having finally gotten around to reading that published work, I’m glad to say my expectations have been met.

Reference: Hoekstra, A. (2008). Vibrant student voices: Exploring effects of the use of clickers in large college courses. Learning, Media, & Technology, 33(4), 329-341.

Summary: Hoekstra investigated student perceptions of learning with clickers in multiple sections of a general chemistry course over a period of three years.  Most of the students in the course (apparently between 200 and 300 students per section) take the course to fulfill degree requirements, but typically less than 10% of the students are chemistry majors.  Survey results of these students indicated that 80-90% of them are “concerned about whether or not they will pass the course” and less than 10% of them “would feel comfortable enough to respond to a professor’s question by raising their hand in the large lecture hall.”

Clicker questions were typically used in the course after 10-12 minutes of lecture to assess student understanding of the material just explained.  “Classic” peer instruction was not used.  Instead, each clicker question was asked once and students were encouraged (but not required or prodded) to discuss the questions with their neighbors before voting.  Instructions given to students for these discussions were fairly vague (e.g. “Feel free to work with your neighbors”).  Also, clicker questions were followed by instructor explanations, not classwide discussion.  Correct answers to clicker questions earned three points each; incorrect answers earned one point each.  Clicker questions contributed only 5% of the students’ overall course grades.

Hoekstra investigated student perceptions of clickers through three semesters of student surveys administered using clickers, observations of 27 class sessions, and in-depth interviews with 28 students averaging 56 minutes in length.  Since students in the initial round of interviews had favorable reactions to students, students with less favorable view of clickers were interviewed in the second round.  Observation note and interview transcripts were analyzed through in vivo coding, “a form of open coding designed to allow conceptual categories to emerge from the data.”

Key results are as follows:

• Interviews indicated strongly that students pay more attention during lecture because they know that clicker questions are asked frequently during lecture.
• Students stated in interviews that “they looked forward to times when they were able to talk with their peers” during clicker questions.
• Observations revealed that students frequently voiced their reactions (positive, negative, surprised) to the display of results and answers of clicker questions.
• Interviewees also indicated that the results displays provided them useful and regular feedback on their learning in the course.  Some even indicated that the clicker questions were most helpful when they answered them incorrectly since these were opportunities to resolve misconceptions.
• About 15-20% of students chose not to engage in peer discussion of clicker questions.  The decision to engage was typically influenced by the difficulty of the clicker question and the student’s “affinity for working with others.”  Interestingly, during more difficult clicker questions, female students were more likely to engage in peer discussion than male students, who tended to use these questions as tests of their own understanding.  Other reasons for working alone included not having done the reading before class, a disinterest in hearing possibly incorrect explanations from their peers when the correct explanation would be forthcoming from the instructor, and the fact that accountability for peer interaction was difficult in the large class.
• Most students found the general noise and activity levels in the classroom during peer discussion stimulating.  Some students found it distracting and would have preferred times of quiet as they answered clicker questions.
• Many students felt that clicker questions increased their anxiety levels during the initial weeks of the course but as they became comfortable with the technology and with peer instruction, they found that the clicker questions decreased their overall anxiety about the course.

Hoekstra uses a quote from Trees and Jackson (2007) to summarize many of her findings: “The success of clickers is in many ways dependent on social, not technological, factors.”

Comments: Where to begin?  This study is a rich source of understanding the many ways that students interact with clicker questions and with each other during times of peer instruction.  I’ve briefly summarized the findings above, but the paper includes details, examples, and very illustrative quotations from student interviews.  I realize that some find qualitative research less meaningful than quantitative research, but I think the scope and rigor of Hoekstra’s work adds much credibility to her findings.

Before commenting on a few specific findings, I thought I might connect the teaching environment described in Hoekstra’s paper with some studies I blogged about earlier in “student participation week” here on the blog.  Given the results of Lucas (2009), the vague instructions given to students for discussing clicker questions might have reduced the quantity and quality of student participation in those discussions.  Since the grading scheme for clicker questions was both high-stakes (because correct answers earned three times as many points as incorrect answers) and low-stakes (because clicker questions only contributed 5% of the students’ course grades), it is unclear from the results of James (2006) and Willoughby and Gustafson (2009) whether or not this grading scheme would have enhanced or inhibited student participation.  Hoekstra’s work did not include a control group of any kind, so one can’t say if student participation in the courses she studied was less than or greater than it would have been under different conditions, but her results seem to indicate that most students engaged in meaningful and productive peer discussions in spite of the vague instructions given to them and the somewhat high-stakes grading scheme.

As for Hoekstra’s specific findings, I think they lend support to a statement I made in my book: “Knowing that a deliverable [a clicker question] may at any time be requested from students can help students maintain attention and engagement during a class session.”  Clickers make it easy to request frequent deliverables of students during class, and Hoekstra’s findings indicate that this is an important reason to use clickers.  Hoekstra’s findings also support other reasons I frequently provide for using clickers: sharing the results of a clicker question can enhance student engagement, clicker questions provide students with useful feedback on their learning, and clicker questions can be useful in structuring class time for students.

Hoekstra’s findings about gender and student participation are thought-provoking.  I’ve debated the importance of an initial, independent vote prior to peer instruction time with several instructors who tend to skip this initial vote (particularly my friends in the math department at Carroll College).  Hoekstra’s findings indicated that for difficult clicker questions, female students might not get as much out of an initial, independent vote as male students, whereas male students might not appreciate jumping straight into peer instruction without a chance to respond independently.

This certainly complicates the debate about initial, independent votes, as well as a teacher’s choices during peer instruction times.  I’ve frequently found it to be the case that the students in my class who hesitate to engage in peer discussion are male students (so Hoekstra’s findings ring true to me), and I typically prod these students to discuss clicker questions with peers.  I may not do that as much in the future given these results.

The situation is further complicated by the finding that some students don’t appreciate the noise and activity levels during clicker questions.  This point reminded me of Richard Felder’s work on learning styles.  Felder’s model distinguishes between active learners, those who prefer to learn through discussion and interaction, and reflective learners, those who prefer to think quietly first.  He makes the great point that traditional lectures do a poor job of supporting both types of learners, since they typically provide students with little time for discussion or quiet reflection.  The “classic” peer instruction model serves both types of learners well, however, since students are invited to respond to clicker questions first on their own, then after discuss them with their peers.  (Larry Michaelsen’s team-based learning model works similarly.)

The finding that some students decide not to engage in peer discussions because they want to wait to hear the correct explanation from the instructor was interesting, as well.  Some instructors (for example, Dennis Jacobs, who teaches chemistry at Notre Dame and is profiled in my book) are very intentional about having students surface and debate reasons for and against all of the answer choices to a clicker question.  The methods these instructors use can help students move away from merely taking notes and memorizing explanations and develop critical thinking skills.  It’s possible that the students in Hoekstra’s study who preferred waiting for their instructor’s explanations to peer discussion might have been motivated to engage in peer discussion and thus sharpen their critical thinking skills with more directive instructions and/or requirements on the part of the instructors.

It’s also worth noting that students who weren’t interested in hearing their peers’ incorrect explanations for clicker questions were also concerned about sharing their own incorrect explanations and thus confusing their peers.  Responding to those concerns might be an important part of motivating these students to engage in peer discussions.

I’ll finish my comments with a response to the quote from Trees and Jackson (2007).  I’ll agree that social factors are likely more important than technological factors in the success of teaching with clickers.  I would qualify that statement, however, to note that (a) the technology can enhance those social factors when used well and (b) the teaching choices that instructors make when using clickers can have a significant impact on those social factors.  As a result, we need not think of those social factors as out of our influence as instructors.

That’s the end of student participation week here on the blog.  That’s also likely the end of five-posts-in-a-week here, too!  I’ll return to my usual format next week, although I have found a few more interesting looking articles on student participation to read soon…

I attended a presentation by Daniel King, a chemistry faculty member at Drexel University, at a recent conference.  He’s been using clickers for several years in both large, introductory courses and small, upper-level courses, and I thought it might be interesting to share some of his approaches to doing so here on the blog.

Daniel shared several types of clicker questions he uses.  He uses clicker questions to assess students’ knowledge of course prerequisites at the beginning of lessons in which those prerequisites will be used.  He likes to stimulate students’ interest in topics before discussing those topics by asking clicker questions that have non-intuitive correct answers, creating “times for telling.”

Daniel also has students predict the outcome of classroom demonstrations as a way to engage them in those demonstrations.  He noted that many students don’t pay attention to demonstrations until something dramatic happens; his prediction questions engage them earlier in the process.  He also frequently uses the think-pair-share / peer instruction method, engaging students in small-group discussions about difficult questions.

As for grading clicker questions, Daniel prefers to grade on effort and not to penalize students for incorrect answers.  This is because (a) his questions are often designed to introduce students to topics and thus aren’t likely to be answered correctly by many students and (b) he doesn’t want his students to worry about their grade when responding; he wants them to be thinking about the chemistry.

The first semester he included clicker questions in his students’ grades, he counted them toward 5% of his students’ grades as a participation grade.  Students would earn these points by answering at least 75% of the clicker questions during the term.  He found, however, that a number of students who ordinarily wouldn’t attend class starting coming to class just to earn these participation points.  This was problematic because they were often disruptive (chatting among themselves instead of paying attention) and because they frequently responded to clicker questions without thinking about those questions, making it difficult for Daniel to interpret the results of his questions.

To alleviate these problems, the next semester, Daniel awarded 5 bonus points to the final exam scores of students who answered at least 75% of the clicker questions.  This reward wasn’t sufficient to motivate students to attend class if they really didn’t want to, but it did reward the effort of those students to came to class and participated regularly.

Daniel provided some insight into his decision-making process regarding when to move on after a clicker question.  He said it depends on the reasons he has for asking the question.  If the question is meant to assess students knowledge of a concept or technique they’ll need to understand in order to follow the rest of class, he’ll spend time discussing the question unless 85% or more of his students answer it correctly.  If only 50% of students answer a question correctly that he thinks they should have answered correctly had they spent some time studying, he’ll tell the 50% of students who missed it to hit the books and move on with his lesson.

Daniel shared several other aspects of his use of clickers, including his use of a couple of clicker questions early in the semester that most students answer incorrectly to teach students that the most popular answer is not necessarily the correct one.  He’s clearly thought a lot about his teaching choices when using clickers, and he did a great job of articulating his reasons for his choices during his presentation.

Reference: Bunce, D. M., VandenPlas, J., & Havanki, K. (2006). Comparing the effectiveness on student achievement of a student response system versus online WebCT quizzes. Journal of Chemical Education, 83(3), 488-493.

Summary: In the context of a 41-student chemistry course for nursing students, after certain topics were introduced during lecture, students were asked to respond to questions about those topics delivered via a classroom response system (CRS).  (In this case, the system used software running on wireless-enabled laptops that were loaned to students for this purpose.)  After class, students were asked to complete online quizzes on class topics prior to the next class.  The impact of these activities on student learning was assessed through performance on instructor-written hourly exams and a standardized final exam provided by the American Chemical Society.

Since some questions on the hourly and final exams featured topics covered in the CRS questions, some in the out-of-class quizzes, some in both, and some in neither, the authors were able to assess the impact on student learning of the two review mechanisms.  Students did better on hourly exam questions tied to the online quizzes than other questions, indicating that the online quizzes did the best job of helping students do well on these exams.  Results from the final exam indicate that neither the in-class CRS questions nor the out-of-class online quizzes helped students do better on the final, however.

The authors report a couple of problems that undercut these results.  One was that as students responded to the in-class CRS questions, they were able to see the bar chart showing the results as they came in.  This meant that, as one student put it, “People who don’t know the answer simply wait for the graph [to enter their responses] and no real learning occurs.”

The other was that the online quiz questions were made available to students after the quizzes for study purposes, and, according to a survey of students, students took advantage of this resource.  The in-class CRS questions, however, were not made available to students for review.

Students reported on a survey that the most useful features of the CRS were that it helped students reinforce what they learned in class and that giving them the opportunity to talk about class material with their peers helped them learn the material.

Comments: This article is a useful example of how difficult it can be to design a study about classroom response systems that provides meaningful results.  I’m glad the authors published this account, even though their results aren’t strong.  I think others designing similar studies are likely to learn from some of the design mistakes made in this study, in part because the authors did a great job of exploring the constraints of their study design in the conclusion of the article.

One of the primary advantages of having students respond via clickers or similar classroom response systems is that students are able to respond to a question independently.  Each student is asked to respond before he or she finds out what his or her peers think.  This can increase the level of engagement of the students with the question and any subsequent discussion.  The “bandwagon effect” seen in this study, in which students wait to find out what their peers think and then pick the most popular response, is why asking for a show of hands can be unproductive.  Since the classroom response system in this study was used in a such a way as to eliminate this primary advantage, it’s difficult to draw meaningful conclusions from the study.

What is clear from this study is that the availability of clicker questions for students to review after class is a potentially important variable, one that should be included in future research on the effects of classroom response systems.  This study presents some evidence that this may be a key ingredient in the impact of clicker questions on student learning.  I hope that future studies take a closer look at this variable.

Studies like this one that compare classroom response systems with alternatives (like online quizzes or asking for a show of hands) can get complicated very quickly.  I thought I might list just a few of the variables at play in this study to illustrate how difficult it can be to isolate the effects of a CRS.

• Independent Answers – In this case, students didn’t have to answer in-class questions independently; they could see their peers answers before responding.  This was probably not the case for the online quizzes; students likely had to respond on their own before seeing their peers’ responses.
• Peer Instruction - Students in this study were required to pair up and provide consensus answers to the in-class questions.  Students worked on their own when responding to the online quiz questions.  This seems like a significant difference between the two methods.
• Class Results - Students were shown the class results to the in-class questions.  It’s not clear if they were shown the class results to the online quiz questions, either online or in subsequent classes.  The impact of seeing these class results is something worth exploring in CRS research since these results aren’t typically immediately available when other response mechanisms are used.
• Immediacy of Feedback – With the in-class questions, students found out immediately whether they answered the questions correctly.  It’s not clear from the article if students had to wait to receive feedback on their online quizzes.
• Agile Teaching - In-class CRS questions that were missed by significant numbers of students were reviewed in some fashion during class, which meant that the “lecture” was responsive to student learning needs evidenced by the CRS.  It’s not clear if online quiz questions influenced class time in a similar fashion in this study.  The ability to practice “agile teaching” is a primary advantage of a CRS, so this is an important variable to consider.
• Question Type – Not much is said in this article about the nature of questions asked in class via CRS or out of class via online quizzes.  The two sets of questions were judged to be of similar difficulty levels by a panel of four chemical educators, however.  The students appeared to answer the online quiz questions correctly about 88% of the time, so the questions weren’t that difficult.  The format, difficulty, and learning goals associated with questions are potentially important variables.
• Availability for Review – The online quiz questions were made available to students for test review; the CRS questions were not.
• Grading Scheme – The article doesn’t state how either the in-class questions or the online quizzes were factored into students’ grades.  Given the role grades play in student motivation, this is a variable worth noting.

These are some of the variables I look for when I read about studies exploring the impact of classroom response systems.  I hope this partial list will be of use to researchers reading my blog.

One of the more challenging aspects of teaching with clickers is coming up with questions that engage students and promote class-wide and small-group discussion.  Fortunately, some instructors have shared online the clicker questions they’ve developed.  Here are a few question banks I’ve found.

• Astronomy ConcepTests, Paul Green, Harvard University – features questions for introductory astronomy courses
• Chemistry ConcepTests, University of Wisconsin – includes questions for general, organic, analytic, inorganic, and physical chemistry, as well as biochemistry
• Earth Science ConcepTests, Science Education Resource Center, Carleton College – features questions on 11 different Earth science topics, including surface processes, solid earth, and time / Earth history
• Physics ConcepTests, University of Colorado-Boulder – includes questions on mechanics, electricity and magnetism, energy and the environment, and the physics of sound and music
• Project Math QUEST, Carroll College – includes questions for precalculus, calculus, linear algebra, and differential equations, as well as links to other mathematics question banks

Do you know of any online question banks, particularly any in the social sciences, humanities, or professional disciplines?