Resources for engaging and assessing students with clickers
21 Jul
The summer meeting of the American Association of Physics Teachers (AAPT) wraps up in Portland, Oregon, today. There were several talks on teaching physics with clickers at the meeting, including one by Ian Beatty of the University of North Carolina at Greensboro physics education research group. Ian was the subject of my first podcast interview, and he’s been doing great work helping science instructors at the K12 and post-secondary levels teach effectively with clickers.
In Ian’s presentation, he identified and addressed several common concerns instructors express about teaching with clickers. For each concern, Ian identifies a belief about teaching and/or learning that likely underlies the concern, as well as an alternate belief that can be adopted to address the concern productively. Ian also includes some practical strategies and example clicker questions for each of these alternate beliefs.
For example, when many instructors hear about teaching with clickers, they’re concerned with having sufficient class time to cover what they need to cover in their courses given the time required by having students discuss and respond to clicker questions. Ian notes that this concern is likely a result of the following belief: “I must explicitly address in class everything students will be held accountable for.” Ian then presents an alternate perspective on this idea: “I can use class time to focus on core ideas and big-picture understanding, and charge students with filling in the details outside class.” This alternate perspective is, perhaps, non-intuitive to many instructors, but it’s a reasonable and useful perspective to have. Adopting this perspective leads to a shift from what Ian calls an understanding of class as a place to present content to an understanding of class as a place to help students digest content. Ian then shares five tips and techniques for implementing this shift in the classroom.
Ian addresses other concerns in a similar manner, including concerns about having enough time to write good clicker questions, concerns about poor student participation during class, and concerns about changing one’s teaching style. His visuals, which use the online presentation tool Prezi, are included below and are well worth checking out.
Ian also includes a couple of nice visualizations of the clickers-facilitated pedagogy he endorses, Technology-Enhanced Formative Assessment (TEFA), including this one:
I really like the graphics he uses to represent the four main components of TEFA: question-driven instruction, formative assessment, dialogical discourse, and meta-level communication. Speaking of visual thinking, I’ll end by noting that this is Ian’s first Prezi, but it’s a great one. He uses the Prezi navigation system (zooming in, out, and around) very effectively.
For more coverage of Ian’s talk as well as other talks at the AAPT conference, see Stephanie Chasteen’s reports (one and two) over on the Active Class blog.
15 Jul
Over on the Vanderbilt University Center for Teaching blog, my Vanderbilt colleague Isabel Gauthier, professor of psychology, has shared her experiences asking her students to write their own clicker questions. I met with Isabel a few years ago and briefly discussed ways to use clickers in her courses, and she’s really taken the technology (and pedagogy) and run with it. She’s got a great handle on how to have students write their own clicker questions, and I’ve been wanting to share her experiences here on the blog for a while. Here’s her article, in her own words:
It is difficult to write meaningful and discriminative multiple-choice questions that students find clear and fair. Years ago, I met with CFT assistant director Derek Bruff, who gave me useful pointers to perfect this skill. But a side effect of this interaction transformed entirely the way I teach: I learned so much by working on writing better questions, surely my students could learn too! Derek said something like, “You know, some teachers ask their students to generate questions…” This idea took me on a path to use this strategy, cautiously at first, and then more boldly, as the central pedagogical and evaluative strategy in some of my courses, including Brain Damage and Cognition and Principles of Experimental Design.
I teach these courses three days a week. On two of these days each week, I lecture on course material. These lectures are informed by questions about the readings posted online by students and issues that emerge from a hands-on, semester long project I assign my students. On the third day, we use clickers to go through student-generated multiple choice questions.
Each week each student is responsible for turning in a single question on the weeks’ readings. Students use a PowerPoint template to submit their questions which facilitates use of the question in my clicker software, TurningPoint. In the notes area of the slide, each student includes their name, the correct answer, the page(s) that inspired the question, and, optionally, a justification for the correct answer. Before class, I concatenate all the questions in a single file and read them, grading each on a scale of 1 to 5. The grade goes in the notes area, and, in a textbox on the slide, I write comments about the question. This allows me to print the slides as a PDF with student names removed so that all questions and comments can be distributed to students. I then reorder the slides to choose the right mix of questions I want to use in class with the clickers.
This provides me in a single step with my preparation for the next class, an idea of what I need to focus on during my lecture days, an evaluation of each student, and a mechanism for providing students with feedback on their learning. This weekly feedback allows students to realize how difficult it is to write a good question, one that raises an important issue clearly and is appropriately challenging for their peers. Students eventually learn to key in on critical concepts and relationships in the readings and sometimes even go beyond the readings in interesting ways. They take a more active part in their own and their peers’ learning, and their questions keep me focused on what is most challenging for these students at each point in the course.
Each week students answer the best of these questions in class using clickers, accumulating points for their answers using a generous but motivating grading scheme. If there’s controversy over the correct answer to a question, the class can decide to eliminate a question or to accept multiple answers as correct, provoking interesting discussions. As needed, I can lecture for a few minutes, but issues are generally clarified in class discussion. Questions are used anonymously in class, but students want their question to be picked and use wit and humor to this effect, making the experience more enjoyable for everyone.
This method completely replaces any exams I used to give: They are no longer needed since my students now share the responsibility to evaluate their own learning throughout the semester.
Isabel and her use of clickers were featured on Nashville’s NewsChannel 5 last year. Here’s the video clip:
21 Apr
Just over a year ago, I shared a story here about a clicker question I used in one of my math courses that didn’t go as planned during class. I titled my post “Flexible Clicker Questions” because I wanted to make the point that clicker questions that seem to be poorly written can turn into real learning opportunities for one’s students. If you put a poorly worded multiple-choice question on an exam, you’re in for a lot of student complaints and regrading. However, in class, a poorly worded multiple-choice clicker question can, with a little agility on your part, turn out great.
I mention this because Mitch Keller recently described a similar incident in his math course over on his blog, Partially Ordered Thoughts. He posed a particular clicker question with what he thought had a single correct answer. His “correct answer” was indeed the most popular student response to the question, but more than 60% of students selected other answers. Mitch wisely had his students discuss the question in small groups and then led a classwide discussion of the question. Not only did he surface the correct reasoning for the “correct” answer, but he discussed the other answer choices, too. It turned out that there were reasonable arguments for not one, but three of his answer choices. Mitch writes:
A natural first reaction to a slip-up in a clicker question is almost always “Drats! I thought I’d done that perfectly.” However, it became a teachable moment. In reality, we were able to discuss far more aspects of generating functions than I intended with the question.
Have you used a clicker question that turned out to be poorly worded, yet resulted in valuable class discussions? Please share below!
Image: “Untitled” by Flickr user Maurizio Polese / Creative Commons licensed
4 Mar
Classroom assessment techniques (CATs) are simple, non-graded, usually anonymous, in-class activities designed to give you and your students useful feedback on the teaching-learning process as it is happening. The standard reference on CATs is Classroom Assessment Techniques: A Handbook for College Teachers by Thomas A. Angelo and K. Patricia Cross (Jossey-Bass, 1993). This book includes 50 CATs, each described in detail with examples from a variety of disciplines. You’ve probably heard of a few of these, such as the minute paper, muddiest point exercise, and background knowledge probe.
CATs provide what is known as formative assessment, something I’ve frequently blogged about. This is assessment of student learning intended to inform future teaching. Formative assessment is often contrasted with summative assessment, which is performed in order to evaluate student performance. Summative assessment comes at the end of a learning experience; formative assessment happens as the students are learning. Feedback from formative assessment can provide instructors with useful insight into what students are understanding, what they are not understanding, and how they might target their teaching to their particular students.
At the recent EDUCAUSE Learning Initiative conference, Charlotte Briggs (University of Illinois-Chicago) and Deborah Keyek-Franssen (University of Colorado-Boulder) presented the results of a very useful study. They combed through all 50 CATs in the Angelo and Cross book and determined that 23 of them could be used with clickers. I’ve long thought of classroom response systems as a sort of “technoCAT,” a technology-enhanced classroom assessment technique, since they provide such useful formative assessment of student learning. Charlotte and I connected via Twitter some time ago, and she had let me know that this analysis of the Angelo and Cross book was in the works. I was very excited to see her work presented at the ELI meeting!
Charlotte and Deborah’s PowerPoint slides are available, as is their handout listing all 23 CATs that can be performed with clickers. In their slides, they provide the following example of a CAT that can be used “as is” with clickers.
Background Knowledge Probe: Which of the following is NOT a characteristic of Romanticism?
- Attention to “the natural”
- Valued “folk” literature, such as fairy tales
- Had a strong geographical center in Düsseldorf
- Referred to “the blue flower” as a central symbol for longing
- Valued medieval literature and art.
You can imagine asking this kind of clicker question at the start of a unit on Romanticism–or a unit that referenced Romanticism but didn’t focus on it. If knowledge of Romanticism is important for participating in the discussion that followed, then this question will let instructors know how much time they need to spend reviewing Romanticism at the start of the unit.
The background knowledge probe CAT is one that I referenced in my book since it’s such a common use of clickers. Where Charlotte and Deborah’s work gets more interesting is in their analysis of the other 49 CATs in the Angelo and Cross book! For instance, they identify 12 other CATs that can be used “as is” with clickers, including such CATs as approximate analogies, problem recognition tasks, self-confidence surveys, and goal ranking and matching. They also identify 10 CATs that can be modified to work with clickers.
For example, Angelo and Cross describe the “one sentence summary” CAT, in which students are asked to write a one-sentence summary of a reading or lecture using the WDWWWWHW format: Who Does What to Whom When Where How and Why. Charlotte and Deborah note that students aren’t able to construct and submit these sentences using clickers. However, they can be given a potential one-sentence summary and asked to identify its flaws. The example they share in their slides is this one:
One-Sentence Summary: Find the errors in WDWWWWHW: A grand jury is a panel of judges (who) that decides if someone should be charged with a crime (does what to whom) when the offense might be a felony carrying prison time (when) if federal courts and most state courts (where) by listening to arguments by attorneys from both sides (how) so common sense and community perspectives are part of the criminal justice system (why).
- Who and Why
- When and Where
- How and Why
- Who and How
- Does What to Whom and How
This clicker serves much the same purpose as a “traditional” one-sentence summary, in part because it’s a “multiple mark” style of question, asking students to identify not one, but two things wrong with the given summary. If your clicker system allows actual multiple-mark questions, allowing students to select as many incorrect elements as they wish, the question becomes even more complex–and thus closer in usage to the one-sentence summary described by Angelo and Cross.
Charlotte and Deborah make a few very good points about modifying CATs to work with clickers. They note that doing so “tends to down-grade the complexity” of the CAT itself. With the one-sentence summary, for instance, you lose the ability to see what surprising things students might say in their constructed sentences. However, Charlotte and Deborah point out that class discussion of the clicker question can restore that complexity. As they write, “Instructors often get the most out of clickers when they are used to prompt discussion,” which is a point I always make when I talk about teaching with clickers.
Here’s one more great example along those lines. Instead of asking students to write down the “muddiest point” of a lecture at the end of class, Charlotte and Deborah suggest in their handout the following:
List potential topics on slide and include an “other” option. Ask students to indicate the topic with which they had the most difficulty. If a significant proportion of the class selects “other”, probe the class to identify other “muddy” issues.
For other ideas on adapting CATs for use with clickers, take a look at their handout. The Angelo and Cross CATs book is well-known in some educational circles (not so much in others, unfortunately), and Charlotte and Deborah’s work serves as a nice introduction to teaching with clickers for those familiar with the book. Conversely, those already teaching with clickers are likely to find a few new ideas for using them as they explore the CATs framework.
5 Feb
Eric Tremblay recently blogged about his plans to have his students write clicker questions for him to use during class. He’s not lazy; he wants his students to think about the material in his course and possible misunderstandings of that material. Student questions will be posted in a class forum, and he’ll select one or more each week to use in class. Students earn participation credit for posting questions and triple credit when their questions are the ones selected.
Having students write exam questions as a way of preparing them to take exams is a time-honored teaching strategy, but I have only lately heard of instructors like Eric having students craft clicker questions as a way to have them engage with course material. Writing clicker questions is difficult, but that’s due in part to the difficulty of predicting student misunderstandings, which is required for constructing good wrong answer choices. I wonder if students might be better able to identify potential misunderstandings since they are not experts in their fields and are thus closer, in a sense, to those misunderstandings.
This idea of having students write clicker questions came up a few times at the recent Joint Mathematics Meetings I attended. Have you tried this? I would be interested in hearing how this plays out.
18 Jan
My post last week pointing to a few collections of best practices for writing clicker questions seemed to go over well, so I thought I would return to that topic. As it happens, I put together my own list of best practices to share at the minicourse on teaching with clickers that Adam Lucas and I led at the Joint Mathematics Meetings in San Francisco last week. The following list summarizes some of the resources from my earlier post and adds a few tips of my own.
What are your suggestions for writing effective clicker questions?
11 Jan
I am frequently asked how one goes about writing effective clicker questions. My usual approach is to share examples of useful clicker questions of different types from a variety of disciplines. That works well, but I believe that some who ask me this question are looking for something more directive. I included many suggestions for writing clicker questions in Chapter 4 of my book, but I was recently asked by a colleague for some other resources on this topic. After putting together a list of a few resources in an email to him, I thought I might share it here, as well.
One flash of insight I had recently is that, at least for me, it’s not really creating questions that’s tough. The hard part is figuring out what I want my students to learn from the class, and casting that in terms of what I want my students to be able to do.
I’ll add that I have an article that should complement the above resources that’s due to come out soon. I’ll be sure to share it here when it’s available.
Other suggestions for succinct advice on writing clicker questions?
Update: In the comments below, Stephanie Chasteen mentions a video produced by the University of Colorado that includes strategies for writing clickers questions. Here’s the video:
21 Sep
One of the questions I’m asked most often when I present about teaching with clickers is the “coverage” question: How do you cover all the content you need to in a course if you spend class time having students think about, vote on, and discuss clicker questions? All that active learning during class must mean you can’t cover all the same content, right?
Although I find the term “cover” problematic, I understand these questions. Particular in courses that are prerequisites for other courses, there’s a need to make sure students learn a certain (usually large) amount of material. In talking with faculty who teach with clickers, I’ve heard several different kinds of responses to the “coverage” question, ones I detail in my book. One response is to move some of the learning that would have taken place during class to out-of-class time. One way to do this is by having our students read their textbooks before class, which I’ve done in my math courses for several years now. This means that students come to class with some exposure and understanding of the material, which allows class time to be spent helping the students make sense of that material and go deeper via clicker questions and other active learning techniques.
However, since studies show that only about 30% of students will read their textbooks before class without some kind of incentive, it’s helpful to have students complete pre-class reading quizzes online. This semester, I’m having my students do so via our course blog. I post three or four open-ended questions about the textbook section we’ll be addressing in class. They respond to those questions in the comments below the blog post. (I’m using the Semi-Private Comments WordPress plugin to make sure student can’t see each others’ responses.) I grade them on effort, and the quizzes count toward a class participation grade. I’ve found these pre-class reading quizzes do the job well. I probably have between 80 and 90% of my students read the textbook before class and make at least some sense out of it judging by their responses to the reading quiz questions.
An added benefit to having students complete pre-class reading quizzes is that I can draw on student responses to open-ended quiz questions to create in-class clicker questions. Here’s an example:
Consider Question 1 on the Introductory Problems handout and Example 1 in Section 1.6. These two problems involve input-output relationships between different sectors of an economy. In what ways are these problems essentially different? Which of the following is the best answer to this question?
- The output from one sector in the example is entirely used up by the other sectors. In the handout, the output is only partly used and a net excess is provided.
- Example 1 asks for the total annual outputs of the coal, electric, and steel sectors. Whereas question 1 is looking for the production levels for an outside demand.
- In the original example, we’re solving the system to meet a single expectation from a foreign country of three demands, whereas the book’s example is looking to maximize the productivity. This means the book’s example has multiple solutions and we’re looking for the best of them, where as our original only has one.
- The two problems are different as the first is trying to find the initial inputs to achieve certain outputs while the second problem is about finding the market price.
The exact same question was posed on the pre-class reading quiz the night before as an open-ended question. The answer choices you see here actual student responses to that open-ended question. During class, I had my students respond to this clicker question, letting them know that four of them should recognize their own words in the answer choices.
The votes were split 30% / 0% / 43% / 26% among the four answer choices, which is a great distribution for generating discussion about the question. It helped that the most popular answer (#3) was partially incorrect. (The book example did not, in fact, deal with maximizing productivity.) The other two answers selected by the students (#1 and #4) are both correct, although #4 gets at the heart of the difference between the two examples more than #1 does.
Some of the students were bothered by the fact that this question doesn’t have a single correct answer. However, since I’m trying to help my students improve their ability to communicate mathematical and technical ideas, it’s worth spending time on a question like this one, where the quality of the explanation plays an important factor.
We had a funny moment when the student who supplied the popular but incorrect answer choice (#3) spoke up after we had discussed what that choice was incorrect. He didn’t directly own up to his answer, but instead said something like, “I think the student who gave that answer probably didn’t catch on to the fact that productivity wasn’t being maximized. He probably has a much better understanding of the example now.”
It can be challenging to write clicker questions with answer choices that align well with student understandings and misunderstandings of a topic. Taking the students’ very own responses as answer choices is one way to get around this. It also communicates to students that the pre-class reading quizzes are an integral part of their learning experience.
19 Sep
I’ve recently become a fan of the “mark all that apply” type of question my classroom response system facilitates. I call these “multiple mark” questions in my book. Here’s one I used in the linear algebra course I’m teaching this fall.
This question is adapted from one of the questions written by Project MathQuest out of Carroll College. Their version of the question wasn’t a multiple mark question. Instead, it included a fifth option, “More than one of the above are possible.” While that option makes the question more interesting and more challenging for the student, it also yields inconclusive data about student learning since students submitting that response may have different ideas about which of the four options are possible. That’s not all bad, of course. Given my interview with Kelly Cline, one of the PIs for Project MathQuest, I can imagine Kelly leveraging that ambiguity into a productive classwide discussion of the question.
However, I decided to turn this question into a multiple mark question by adding the instruction, “Mark all that are possible.” As you can see from the results, 20 of the 20 students present that day indicated that option 3 was possible, 19 of the students indicated options 1 and 2 were possible, and 14 of the students felt that option 4 was possible. This was very useful feedback for me, since I could quickly tell that the class was in agreement on options 1 through 3, but option 4 deserved some further discussion.
I’ll admit, however, that I got a little tripped up on my own question logic here. As it turns out, all four options are possible, which was not my intent when I included this question in my lesson plan. Option 2 is only possible if the third plane intersects the two overlapping planes and option 4 is only possible if the three planes are parallel because they are in fact the same plane. The way I’ve worded the question, these wrinkles aren’t addressed, making all four options possible. As a result, the question doesn’t do a great job at uncovering student understanding of these wrinkles.
Here’s the question I should have asked instead:
Suppose you have a system of 3 linear equations in 3 variables. Which of the following conditions would guarantee that the system has an infinite number of solutions? Mark all that apply.
- All three equations represent the same plane.
- Two of the equations represent the same plane.
- The three equations represent planes that intersect along a line.
- The three equations present parallel planes.
With this question, only options 1 and 3 are correct. With option 2, it could be that the third plane is parallel to but distinct from the two overlapping planes, yielding no solutions instead of infinitely many solutions. With option 4, it could be that the three planes are not the same plane, again yielding no solutions instead of infinitely many solutions. This wording of the question puts the special cases in their proper places.
Have you used multiple mark questions? Do you find them more difficult to write? How do your students respond to them?
9 Sep
Back in January, I blogged about a New York Times article describing MIT’s Technology Enhanced Active Learning (TEAL) classrooms. Just today, Diana Senechal blogged about the article, too, as well as her own experiences as an adult student in a physics class that uses clickers. A few important questions were raised in Diana’s post and in the comments that followed it–questions about the prep time teachers need to teach with clickers and about which students we should be trying to benefit through our teaching. I weighed in on those questions on Diana’s blog post, but I thought I would reproduce my comments here in case my readers would like to weigh in, too.
I’ve taught math courses with clickers for five years now, and (full disclosure) I’ve written a book on teaching with clickers, one that draws upon interviews I conducted with 50 faculty members in different disciplines, including physics. As you might expect, I have a few thoughts about the questions raised here!
The first thing I noticed reading this post and its comments was the juxtaposition of the MIT student’s comment that using clicker-facilitated active learning during class means professors don’t have to prepare as much and Mike Anderson’s comment that using the IFAT quizzes he describes took more, not less, preparation time.
I think Mike’s hit the nail on the head: Figuring out what misconceptions students are likely to have, which is required for coming up with plausible wrong answers to multiple-choice questions, is challenging work. And doing what the MIT physics professors are doing–designing intensive learning experiences that help students resolve misconceptions and build their knowledge–is even more challenging. It requires a great deal of understanding of student learning and motivation.
Speaking of student motivation, the question was raised above asking which students are benefited by more active classroom learning experiences. I would argue that as teachers, we have a responsibility to try to motivate and teach all our students, not just the ones that are self-motivated or the ones who learn best by listening to a lecture. I think it’s great that Diana enjoys and benefits from a great lecture. Evidence points to the fact that such students are in the minority. Combining lectures with more participatory learning experiences is likely to benefit more students’ learning.
I’ll also point out that the pedagogy behind Mike’s IFAT quizzes is very similar to the pedagogy behind effective instruction with clickers–getting students to actively engage with problems and to discuss those problems with peers and their instructors, and providing instructors with useful feedback on student learning, feedback that can inform future instruction. As Ricki points out, it’s the pedagogy that counts more than the technology.
That being said, clickers provide a few advantages that other technologies don’t. Clickers allow me to hold my students accountable for their class participation since the system tracks individual student responses. However, clickers also provide students with a level of anonymity since their peers can’t see who they responded, making it safer for them to take risks and be wrong. (Asking a question to a class of students and taking the first student response privileges those students who are quicker, more confident, and more experienced. It leaves all the other students out of the loop, unfortunately.) And the instant display of results (in the form of a bar graph) provides the instructor with useful information for making on-the-fly teaching choices and can have an impact on student motivation. If, for instance, students see that most of them answered a question incorrectly, they’re more likely to pay attention to the explanation that follows.
So, dear readers, what say you? Any thoughts on the prep time issue or the question of which students are most benefited from active engagement teaching techniques?