Continuing my reports from the contributed paper session on teaching with clickers I helped coordinate at the Joint Mathematics Meetings back in January…

“Using Personal Response Systems (Clickers) in Liberal Arts Mathematics Courses to Support a Lecture Format,” Janet A. White, Millersville University of Pennsylvania [Slides]

Just like Jean McGivney-Burelle and Kimberly Burch, Janet White shared her experiences teaching with clickers in a “liberal arts” mathematics course taken by non-majors. Unlike Jean and Kimberly, who teach relatively small sections of this kind of course, Janet teaches in a large lecture hall with 75 students per section. Janet had used clickers in courses for pre-service math teachers in the past and found them useful, so when it was her turn to teach this larger course, she decided to use them again. A classroom response system was hardly the only technology Janet used in this course: She also had students complete online homework and quizzes and she annotated her PowerPoint lecture slides using an Interwrite Mobi.

Janet used clickers on a daily basis in her course, usually either to assess students’ prior knowledge or to assess their understanding of a topic taught during lecture.  Her questions came from a bank of multiple-choice questions provided by her textbook publisher.  She counted the clicker questions as part of her students’ participation grades, but in a low-stakes manner.  Given her use of the questions as well as the source of the questions, many were on the lower levels of Bloom’s taxonomy, aimed at recall and application of procedural knowledge.  She shared an example of a prior knowledge question that asked students to find the measure of an angle that complements a 36 degree angle.  A slightly harder question aimed at assessment of something taught during the course asked students to identify the cut edge in a given graph (or to assert that the graph had no cut edge).

Student survey results indicated that 85% of Janet’s students who used clickers regularly liked using them, and 71% said that using clickers helped them learn the material.  Students who used clickers regularly during the course ended up with higher grades in the course than students who didn’t, but, of course, that can’t necessarily be attributed to the use of the clickers.  (And since clicker questions were factored in the course grade, students who participated more frequently in clicker questions would almost certainly have higher grades in the course anyway.)

Student comments about the clickers were generally positive.  My favorite one was, “I liked getting the wrong answer anonymously.”  Other comments addressed the usual points that students like about clickers: They liked the interactivity, they liked discussing questions with classmates, they liked seeing where they stood relative to their peers, and they liked the feedback on their own learning the clicker questions provided.  The only significant negative aspect for the students was the cost, about $50 in Janet’s case.

Janet found that having students discuss clicker questions in small groups led to very engaged students, even in the large auditorium environment.  In the future, she plans to write more of her own questions, instead of relying on ones from the textbook’s question bank.  She hopes to write more difficult questions that will generate even more engaged discussion during class.  She’s also hoping to find ways to reduce the technology cost to the students, either by selecting a different vendor or facilitating the resale of clickers after each semester to students taking the course the next semester.

Also, Janet mentioned that the earth science faculty at Millersville are big users of clickers.  Earth science instructors looking for advice on using clickers might want to investigate!

Image: “Recursive Daisy” by Flickr user gadl / Creative Commons licensed

Classroom assessment techniques (CATs) are simple, non-graded, usually anonymous, in-class activities designed to give you and your students useful feedback on the teaching-learning process as it is happening.  The standard reference on CATs is Classroom Assessment Techniques: A Handbook for College Teachers by Thomas A. Angelo and K. Patricia Cross (Jossey-Bass, 1993). This book includes 50 CATs, each described in detail with examples from a variety of disciplines.  You’ve probably heard of a few of these, such as the minute paper, muddiest point exercise, and background knowledge probe.

CATs provide what is known as formative assessment, something I’ve frequently blogged about.  This is assessment of student learning intended to inform future teaching.  Formative assessment is often contrasted with summative assessment, which is performed in order to evaluate student performance.  Summative assessment comes at the end of a learning experience; formative assessment happens as the students are learning.  Feedback from formative assessment can provide instructors with useful insight into what students are understanding, what they are not understanding, and how they might target their teaching to their particular students.

At the recent EDUCAUSE Learning Initiative conference, Charlotte Briggs (University of Illinois-Chicago) and Deborah Keyek-Franssen (University of Colorado-Boulder) presented the results of a very useful study.  They combed through all 50 CATs in the Angelo and Cross book and determined that 23 of them could be used with clickers.  I’ve long thought of classroom response systems as a sort of “technoCAT,” a technology-enhanced classroom assessment technique, since they provide such useful formative assessment of student learning.  Charlotte and I connected via Twitter some time ago, and she had let me know that this analysis of the Angelo and Cross book was in the works.  I was very excited to see her work presented at the ELI meeting!

Charlotte and Deborah’s PowerPoint slides are available, as is their handout listing all 23 CATs that can be performed with clickers.  In their slides, they provide the following example of a CAT that can be used “as is” with clickers.

Background Knowledge Probe: Which of the following is NOT a characteristic of Romanticism?

1. Attention to “the natural”
2. Valued “folk” literature, such as fairy tales
3. Had a strong geographical center in Düsseldorf
4. Referred to “the blue flower” as a central symbol for longing
5. Valued medieval literature and art.

You can imagine asking this kind of clicker question at the start of a unit on Romanticism–or a unit that referenced Romanticism but didn’t focus on it.  If knowledge of Romanticism is important for participating in the discussion that followed, then this question will let instructors know how much time they need to spend reviewing Romanticism at the start of the unit.

The background knowledge probe CAT is one that I referenced in my book since it’s such a common use of clickers.  Where Charlotte and Deborah’s work gets more interesting is in their analysis of the other 49 CATs in the Angelo and Cross book!  For instance, they identify 12 other CATs that can be used “as is” with clickers, including such CATs as approximate analogies, problem recognition tasks, self-confidence surveys, and goal ranking and matching.  They also identify 10 CATs that can be modified to work with clickers.

For example, Angelo and Cross describe the “one sentence summary” CAT, in which students are asked to write a one-sentence summary of a reading or lecture using the WDWWWWHW format: Who Does What to Whom When Where How and Why.  Charlotte and Deborah note that students aren’t able to construct and submit these sentences using clickers.  However, they can be given a potential one-sentence summary and asked to identify its flaws.  The example they share in their slides is this one:

One-Sentence Summary: Find the errors in WDWWWWHW: A grand jury is a panel of judges (who) that decides if someone should be charged with a crime (does what to whom) when the offense might be a felony carrying prison time (when) if federal courts and most state courts (where) by listening to arguments by attorneys from both sides (how) so common sense and community perspectives are part of the criminal justice system (why).

1. Who and Why
2. When and Where
3. How and Why
4. Who and How
5. Does What to Whom and How

This clicker serves much the same purpose as a “traditional” one-sentence summary, in part because it’s a “multiple mark” style of question, asking students to identify not one, but two things wrong with the given summary.  If your clicker system allows actual multiple-mark questions, allowing students to select as many incorrect elements as they wish, the question becomes even more complex–and thus closer in usage to the one-sentence summary described by Angelo and Cross.

Charlotte and Deborah make a few very good points about modifying CATs to work with clickers.  They note that doing so “tends to down-grade the complexity” of the CAT itself.  With the one-sentence summary, for instance, you lose the ability to see what surprising things students might say in their constructed sentences.  However, Charlotte and Deborah point out that class discussion of the clicker question can restore that complexity.  As they write, “Instructors often get the most out of clickers when they are used to prompt discussion,” which is a point I always make when I talk about teaching with clickers.

Here’s one more great example along those lines.  Instead of asking students to write down the “muddiest point” of a lecture at the end of class, Charlotte and Deborah suggest in their handout the following:

List potential topics on slide and include an “other” option. Ask students to indicate the topic with which they had the most difficulty. If a significant proportion of the class selects “other”, probe the class to identify other “muddy” issues.

For other ideas on adapting CATs for use with clickers, take a look at their handout.  The Angelo and Cross CATs book is well-known in some educational circles (not so much in others, unfortunately), and Charlotte and Deborah’s work serves as a nice introduction to teaching with clickers for those familiar with the book.  Conversely, those already teaching with clickers are likely to find a few new ideas for using them as they explore the CATs framework.

Maybe this is obvious to others, but I hadn’t thought of this particular use of numeric-response clicker questions, shared with me by a humanities professor recently: In a class that deals with history, ask students to identify the year in which a particular event happened using a numeric-response clicker question.

This question type is typically used in math and science classes to have students respond with their answers to open-ended computational questions, but it can just as easily be used in a humanities class to have students respond with dates (e.g. 1776, 2010).  Sure, one could ask students to respond to a multiple-choice date question, but the free-response format might surface some wrong answers you wouldn’t predict.

This kind of question isn’t limited to events, of course.  You could also ask students to identify the year a piece of literature was written or an artwork was created.  This type of question need not be a factual recall question, either.  You could present to students a piece of art, for instance, they haven’t likely seen before and ask them to analyze the artwork and estimate when it was created.

Some classroom response systems allow you to set a range for the correct answer to a numeric-response question.  With that feature, you could give students a little wiggle room in their answers (“To within 5 years, in what year did X occur?”) or have them respond to the nearest decade.

(By the way, I’ve just signed up for the twitterfeed service, so a tweet about this post should automatically appear in my Twitter stream in the next hour.  Fingers crossed!)

I couple of weeks ago on this blog, I shared a tweet by Colin Morris, a student at Kent State University in Ohio.  His comment (via Twitter) was, “44% OF MY U.S. HISTORY CLASS THINKS WATERBOARDING IS A SURFING TERM. I take back everything I’ve said about these ‘clickers’ being useless.”  After I shared this tweet on my blog, a few interesting things happened.

1. Colin Morris, the student who tweeted this (on the left above), found out about my blog post via Twitter and commented on my blog post, indicating that he saw pedagogical value in clickers but objected to the cost of his clicker, particularly as a senior who won’t use it in future courses.
2. Then Twitter user @iclickercrs, apparently affiliated with the i>clicker company, tweeted about my blog post.  Another Twitter user, @JonathanRose, a professor at Queen’s University in Ontario, Canada (on the right above), saw this tweet and decided to see how many of his (Canadian) students knew what waterboarding is.  He contacted Colin Morris, who directed Jonathan to the Kent State University instructor who posed this clicker question during class, John Jameson.  Jonathan Rose then used the same question in his introductory political science course.
3. Jonathan then posted the results of both questions–the student responses from Kent State and the ones from his Queen’s University students.  Here are the results [PDF].  As you can see, only 28% of his students thought that waterboarding is a surfing term.  Also, more of his students than the Kent State students viewed waterboarding more as torture than an interrogation technique.
4. To bring things full circle, Jonathan Rose tweeted about these results and @iclickercrs re-tweeted Jonathan’s tweet.  I saw this re-tweet, then tracked down Jonathan.  He let me know about items 2 and 3 above, filling in the gaps in my knowledge of this whole social media process.

Watching this all unfold has been very interesting, not only for the interesting uses and reactions to clicker questions, but for the way that Twitter has facilitated connections that might not have happened otherwise.

One Last Update: Colin Morris blogged about this, too, noting the importance of keeping in mind potential audiences when using social media.

I just had to share this tweet I saw a few weeks ago.

@colinmorris: 44% OF MY U.S. HISTORY CLASS THINKS WATERBOARDING IS A SURFING TERM. I take back everything I’ve said about these “clickers” being useless.

Here’s another example of a slideshow that combines great visuals with a few short sentences and phrases to explain something.  The “something” in this slideshow is teaching with clickers, naturally.

This slideshow is by Sidney Eve Matrix of Queen’s University.  She does a great job of using visuals in her presentation.  My favorite image is probably the overflowing glass on slide 19.  It’s a beautiful image that works very well as a metaphor for Sidney’s point on that slide.

I thought I would share a story about a clicker question I used yesterday in my linear algebra course.  Although not all of my readers will follow the mathematics, I hope they’ll all appreciate some of the pedagogical ideas mentioned below.

This semester I’ve brought a bucket into class with me every day.  At the end of each class, I encourage my students to write questions they have on the day’s material on slips of paper and drop these questions in the bucket.  This gives me useful feedback on a regular basis regarding what my students find confusing.  I try to respond to a couple of the more common or interesting bucket questions at the start of the next class.

It occurred to me a few weeks ago that a given bucket question might be of use to a few students but not all of my students, making it less useful to review during class.  So I’ve started trying to turn bucket questions into clicker questions, so that they provide me with a better sense of how many students share a particular student’s confusion.  I’ve been pleased with how this has worked out.  For example…

On Monday, a couple of my students asked related bucket questions that I turned into the following clicker question:

Is it possible for the standard matrix of a linear transformation not to have an eigenvalue?

1. Yes – High Confidence
2. Yes – Low Confidence
3. No – Low Confidence
4. No – High Confidence

You’ll notice that I’ve include confidence level in the answer choices.  I’ve been doing this regularly this semester for questions with only two answer choices (Yes/No, True/False).  I find that knowing, say, 65% of my students answer a True/False question correctly doesn’t provide me with very useful information on their understanding since half of the students with no idea about the question are likely to answer correctly anyway.  Including confidence level provides me with a better sense of how difficult my students find a question.

After my students had a chance to think about this question independently, they voted:

As you can see, the class was almost evenly split on this question, with the majority of them not very confident in their answers.  This told me that we should spend more time on the question, so I had them discuss the question in pairs and re-vote.  During the pair discussion, one of my students, let’s call him Jack, asked, “When you say eigenvalue, you mean just real eigenvalues, right?”  That kind of gave away the question, since I was asking this question to see if students would remember that eigenvalues can be complex numbers.  We saw a linear transformation in the previous class that had no real eigenvalues but did have complex eigenvalues, so this should have been on their radar.  However, we also saw a useful way to visualize the effects of real eigenvalues.  That method doesn’t work for complex eigenvalues, so it’s likely that some students weren’t considering complex eigenvalues when they answered this question since we didn’t have a tool for visualizing them.

Here are the results of the second vote, “tainted” by Jack’s question:

As you can see, most of the students went with Jack on this one, asserting that the standard matrix must have (possibly complex) eigenvalues and feeling confident in this assertion.  This is a reasonable assertion because every matrix of size n x n must have exactly n eigenvalues, counting complex ones and allowing for multiplicity.  That was an assertion I made a couple of class sessions ago.

So far, so good.  I was disappointed that Jack had “spoiled” the question, but it still helped make the point I had intended it to make.  Then one of my other students, let’s call her Juliet, asked, “What if the standard matrix isn’t square [that is, what if isn't n x n]?  Then it wouldn’t have any eigenvalues, right?” Good point, Juliet.  I had commented on Jack’s question that the clicker question should be read as stated, so that complex eigenvalues should be allowed.  Juliet essentially called me on that, noting that the clicker question doesn’t specify if the linear transformation in question has a square standard matrix.  In fact, linear transformations need not have square standard matrices and non-square matrices don’t have eigenvalues, so the correct answer to the clicker question is “Yes.”

This question ended up working better than I had hoped since the fact that non-square matrices don’t have eigenvalues wasn’t clear to the students based on past class sessions.  In fact, several students had asked about that issue in their bucket questions at the end of the last class session.  Juliet’s question presented us with a great opportunity to clear that issue up, and I was able to enlist a couple of students (including Juliet) in helping me prove the result about non-square matrices at the chalkboard.  As a result, what started as more of a recall question (Do students remember that every square matrix has eigenvalues?) turned into more of a concept question (Do students understand where eigenvalues come from well enough to argue that non-square matrices can’t have them?).

What are the takeaways from this story?

1. Clicker questions need not be written as well as exam questions.  This question had two wrinkles to it–one that I planned and one that I didn’t plan.  That would have been a minor disaster on an exam question, but it worked very well for a clicker question since it helped me surface not one, but two, student misconceptions and helped generated useful class discussion about those misconceptions.
2. Handling a clicker question like this took some agility on my part.  I probably could have done a better job facilitating this question (particularly in my response to Jack’s question), but it certainly helped that I was willing to react on my feet to the discussion as it emerged.
3. Writing good clicker questions can be as easy as adapting questions posed by students prior to class.  My question bucket has yielded several useful and engaging clicker questions just in the last couple of weeks.

Have you asked any clicker questions that didn’t go as planned but turned out to be useful nonetheless?

I’m back from the Inaugural Conference on Classroom Response Systems, hosted by the Delphi Center for Teaching and Learning at the University of Louisville.  I had a great time meeting people who I knew only by their research, and some of the sessions were very well done.  Thanks to the Delphi Center, particularly director Gale Rhodes and associate director Marianne Hutti, for putting together such an enriching conference.  I’m already looking forward to next year!

Tim Stelzer, research associate professor of physics at the University of Illinois and one of the founders of i>clicker, presented the morning keynote address.  His presentation was very engaging, featuring a nice blend of information and humor.  His slides were particularly impressive, having been designed in the Presentation Zen style.  Many of his slides consisted primarily of one fullscreen image and just a few words of text.  This put the focus on Stelzer and his message while reinforcing that message visually.  He included a few well-chosen video clips and animations that helped him make his points (with a little humor) as well as an ongoing clicker-enabled game that kept his audience engaged.

One point Stelzer made that stood out to me was that in the past, being highly educated was correlated strongly with remembering lots of facts.  This is still true today to some extent.  Consider Ken Jennings, the guy who won all those Jeopardy gameshows.  He’s considered highly intelligent, but not for higher-order thinking skills (problem solving, critical thinking, etc.), just for remembering lots of trivia.

Stelzer made the point that with all the information available to students via the Internet, factual recall doesn’t play the same role it used to play in learning.  The challenge now in higher education is to develop students’ higher-order thinking skills, and Stelzer feels that classroom response systems can facilitate pedagogies that help teachers meet that challenge.  This is a valid point, and it’s one of the reasons I included in my book so many examples of clicker questions aimed at higher-order thinking skills.

Stelzer also described the genesis of the i>clicker classroom response system.  The first electronic system he used at Illinois (after abandoning the flash card method due to poor student participation, caused by lack of accountability) involved hardwiring jacks in all of the seats in a lecture hall.  The students connected their TI-83 calculators to these jacks.  This system allowed a couple of neat features not available in current systems to my knowledge.  One was that it allowed the instructor to call up a seating chart showing each student’s name and how that student voted in response to a clicker question.  This allowed an instructor to say, for instance, “John, I see that you answered B but the students sitting next to you answered C.  Why don’t you discuss this question with them and see if you can come to a consensus?”

Another feature of the system was that when a student answered a multiple-choice question, the system could be programmed to send the student a response determined by the student’s answer choice.  So if a student selected choice A, the system might reply, “Have you considered X?”, where X would be some example or concept relevant to the answer choice.  I imagine this feature would be very useful in helping students think more deeply about their answer choices.  It’s also a feature that could be implemented in some of the systems that use cell phones, smart phones, and laptops available now.  In fact, this might already be a feature of some of these systems.  If you know that to be the case, please let me know.

(Coincidentally, Friday night at the conference reception, I met Kevin Patton of St. Charles Community College in Missouri.  He and I were talking about clickers and somehow hit upon this very same idea–giving feedback to students after their answers right on their clickers, feedback tailored to their particular answers.)

One of the research findings Stelzer shared was particularly interesting, too.  According to surveys of students at the University of Colorado at Boulder, where over 17,000 clickers are in use, the factor mostly highly correlated with negative student reactions to clickers was “sporadic use.”  If clickers aren’t used very often, students tend not to like them.  That’s pretty good evidence that students see some value in the use of clickers, I think.

Stelzer’s talk was videotaped and will be posted online, probably at the i>clicker site.  His top ten tips for using clickers are also available at the i>clicker site.

I have more thoughts from the conference I’ll be posting in the next few days.