Back in January I gave a keynote talk at the Health Professionals Education Research Symposium hosted by Nova Southeastern University.  Part of my preparation for that talk included reading some of the articles from related disciplines in my clickers bibliography.  Shortly after the conference, I blogged about one great article about using clickers to promote critical thinking in nursing (Debourgh, 2008), and I’ve been meaning to post some notes about the other articles I read.  Let’s get started…

Reference: Cain, J., & Robinson, E. (2008). A primer on audience response systems: Current applications and future considerations. American Journal of Pharmaceutical Education, 72(4), 77.

Notes: The literature review is the highlight of this article.  It’s not as comprehensive as other lit reviews, but it does a great job of describing a few studies of the use of clickers in the health professions with particularly positive results.  For example, Slain et al. (2004) report that students in clicker sections of two pharmacy courses scored significantly higher on exams than students in non-clickers sections.  Similar results were found by Schackow et al. (2004) in classes for family medicine residents and by Pradhan, Sparano, and Ananth (2005) in classes for obstetrics and gynecology residents.  These references are listed in my bibliography.  Hopefully, I’ll find some time to read and blog about them soon.

Cain and Robinson also include a useful exploration of some of the logistical aspects of teaching with clickers.  Instead of making recommendations, they describe the various choices a department might make and their pros and cons.  They note that any clickers initiative should make sense given an institutions teaching philosophy and technology plan.

For example, a pharmacy school with a mandatory laptop program may highly value an ARS that can utilize laptops as response devices, rather than basing the decision on other features.

They also recommend purchasing a set of clickers available to faculty and staff to check out for one-shot events, like continuing education programs and faculty meetings.

The section on recommendations for future research is a strong one.  Cain and Robinson write, “Any effects from using an instructional medium do not come from the use of the media itself, but from the instructional methods employed.”  That’s something I’ve argued here before.  Cain and Robinson call for research that explores the effects of very particular instructional strategies involving clickers, including strategies useful for facilitating discussion about matters of ethics and morality.  While ethical issues are present in every discipline, they are often particularly important in professional education.

Cain and Robinson make an interesting statement in their section on student considerations: “Finally, appropriate application of the ARS in the curriculum should be defined and encouraged.”  I understand the interest in encouraging instructors to use clickers in appropriate ways.  It’s the “defining” piece that makes me wonder if pharmacy education is a bit more top-down than the kinds of programs you find in, say, colleges of arts and science.  I find that faculty members in undergraduate liberal arts departments tend to have a high degree of autonomy when it comes to their teaching decisions.  They might not be comfortable having appropriate uses of clickers “defined” for them.  Am I reading too much into this word choice?  Does your department (whatever your discipline) set policy on educational technology use?

Image: “Rx, San Antonio, TX” by Flickr user Tadson / Creative Commons licensed

I was quoted this morning in “At Universities, Is Better Learning a Click Away?“, an Associated Press story on the future of classroom response systems by AP reporter Eric Gorski.  The story features Michael Dubson, who teaches physics with clickers at the University of Colorado-Boulder.  CU-Boulder, and its physics education research group in particular, has been very active in the world of clickers (including contributing to these great videos), and I was glad to hear Michael Dubson’s perspectives on the technology in the AP piece.

CU-Boulder is an i>clicker campus, and Dubson makes the case in the AP story that a simple, dedicated clicker device is preferable in most instances to more flexible systems based on smart phone apps.  Indeed, i>clicker devices have only six buttons–an on/off button and buttons labeled A, B, C, D, and E.  This is a very simple system, but, as inventor Tim Stelzer argued at the Louisville clicker conference back in 2008, multiple-choice questions with five answer choices work very well for the kinds of formative assessment and peer instruction many instructors use clickers to implement.

Gorski places me on the other side of a somewhat-artificial divide:

Derek Bruff, assistant director of Vanderbilt University’s Center for Teaching, said simple clickers are great at multiple choice questions. But he’s more excited about using smart phones, which allow students to ask questions of instructors, hold back-channel discussions with each other and respond in their own words.

Regular readers of this blog know that I’m definitely excited by the possibilities of using smart phones as “super-clickers” or to facilitate backchannel discussion in the classroom.  It’s true that I’m more excited by smart-phone systems than I am by simple clickers like i>clicker, but that’s largely because I’ve been involved in teaching with clicker with several years and I’m eager to leverage that experience to consider new kinds of technology-facilitated classroom dynamics.  (For one thoughtful perspective on those potential dynamics, consider Sean Seepersad’s recent post on moving away from clickers.  I hope to blog about Sean’s post soon!)

I’ve spent plenty of time thinking about the pedagogy of multiple-choice questions (while writing my book, blogging about clickers here, and giving talks on the subject around the country), and I think the multiple-choice format is often underrated.  I even have an article coming out (soon, I hope!) titled, “Multiple-Choice Questions You Wouldn’t Put on a Test: Promoting Deep Learning with Clickers.”  So I definitely get where Michael Dubson is coming from: Five-answer multiple-choice clicker questions are incredibly useful in all kinds of courses.

All this to say that one of the principles I attempted to uphold when writing my book was that everyone’s teaching context is different–different students, different disciplines, different institutions, different teaching styles and experiences.  I’m interested in helping instructors think more intentionally about their teaching choices, exploring the pros and cons of choices both traditional and innovative.  So while I may be more excited myself about smart phone systems, I always encourage instructors to select technologies and teaching practices that make the most sense in their particular teaching contexts.

I’m glad for clickers to receive the attention of the Associated Press.  The story has been all over Twitter today, and I hope it makes its way into print and online newspapers across the country.  And I’m glad that I could help Eric Gorski out as he was researching this story.  Eric also contributed to a short video piece to accompany his article, and he blogged about the story on the AP’s Facebook page.

Thoughts on the AP story?

Image: “The Nabla System (Forgotten Seed)” by Flickr user Syntopia

At today’s TLT Group seminar on teaching with clickers, a few of the participants were interested in advice on the vendor selection and adoption process.  Given how quickly classroom response system vendors release new hardware and software, I decided not to include in my book some kind of chart comparing the features of popular vendors.  Instead, I described several possible categories for such a chart in the hope that these categories would be a useful starting point for campuses beginning the vendor selection process.

Below you’ll find a short version of this list of categories, along with a few useful questions to ask for each category.  There’s a more detailed version in my book, but given the interest in this topic at the TLT Group seminar, I thought I would share this short version here.  Feel free to share additional factors to consider in the comments below.

Cost

• What are the costs to students for hardware or registration fees?
• What are the costs to your institution for hardware or software?
• What discounts are available if an adoption / exclusivity agreement is signed?

Hardware

• What kind of displays do the clickers have? What kinds of batteries do they use?
• How easy will it be for instructors to set-up receivers before class? How easy will it be for receivers to be permanently installed in classrooms?
• What frequencies does the system use? Radio frequencies are generally superior to infrared frequencies.

Software

• Does the software function differently when running on Macs?
• What needs to be installed on the classroom computer-the software itself or other programs such as PowerPoint?

Accessibility

• How easily can students with physical disabilities, including sensory impairments, use the system?

Registration Methods

• Can students register their clickers’ serial numbers using the local online course management system? How easy is this process for students and instructors?
• Can students register their clickers’ serial numbers during class? How easy is this process?
• Can the clickers be used fully anonymously so that not even the instructor can identify student responses?

Delivery Modes

• How does the system integrate with PowerPoint and other slideware programs? Can the instructor use a remote to control both PowerPoint and the classroom response system?
• Does the system provide a floating toolbar mode allowing instructors to display questions and answers in any way they choose?
• Does the system make it easy to ask questions on the fly during class?
• Does the system provide a student-paced mode in which students respond to questions asynchronously?

Question Formats

• Does the system permit the use of free-response questions? If so, how does the system display the results of a free-response question?
• Are other question formats possible, such as ranking questions or multiple-mark questions?

Reporting and Grading

• What kinds of reports can the system generate from clicker data?
• What options does the system provide instructors for assigning point values to questions?
• Does the system allow instructors to upload grades to the local online course management system? If so, how easy is this process to use?

Back in December, I shared some thoughts on Michael Bugeja’s Chronicle of Higher Education essay titled “Classroom Clickers and the Cost of Technology.”  My post generated a lot of comments.  Bugeja’s essay generated other responses as well, including three letters to the editor published last month in the Chronicle.  All three letters point to educational research conducted on teaching with classroom response systems that Bugeja chose not to mention in his essays, which was my response to the essay, as well.

For instance, Stephanie Chasteen of the University of Colorado-Boulder wrote:

Mr. Bugeja hypothesizes that students would vote against the use of clickers because the costs outweigh the benefits. Research suggests otherwise. In our own large introductory-physics courses, 95 percent of students stated that clickers helped them learn the material. Studies in other disciplines suggest that students are more likely to value clickers when they’re used to promote discussion, rather than to ask simple questions or take attendance.

Doug Duncan, also of the University of Colorado-Boulder, wrote:

Most of the practices [Bugeja] describes are what our research shows to be worst practices. We see them fail, too. When instructors use clickers as part of peer instruction and explain to students that they will attend class more, work harder, learn more, and be rewarded for that, peer instruction and clickers produce learning gains. When instructors ask low-level memorization questions and don’t explain why they are using clickers, students call them dumb and worthless.

I’ll continue to explore and discuss the research on teaching with clickers here on this blog.  Given the clear learning gains that clickers can facilitate as well as the cost of the technology, it’s important to give due consideration to both sides of the cost-benefit discussion.

Further thoughts on Bugeja’s essay or on these responses to it?

This week’s Chronicle of Higher Education includes an essay by Michael Bugeja, director of the Greenlee School of Journalism and Communication at Iowa State University, titled “Classroom Clickers and the Cost of Technology.”  (You’ll need a subscription to the Chronicle to use that link, unfortunately.)  In his essay, Bugeja expands on a few points he made about clickers in a prior essay.  I thought I would respond to a few of his points here.

I agree with some of Bugeja’s takeaways from his institution’s experiences with clicker vendors.  He argues that students should be involved in decisions about instructional technology, that chief information officers should be consulted by departments making such decisions, that faculty adopting technologies should be aware of not-so-obvious costs of using these technologies, and that administrators should be prudent when conducting cost-benefit analyses of new instructional technologies.

Those are all very sensible points.  However, I see some problems in the ways Bugeja uses clickers as an example in support of these points.  The fundamental weakness of the essay is that Bugeja seems to be doing a cost-benefit analysis on clickers without paying much attention to the benefits portion of that analysis.  As well-referenced as the cost portion of his analysis is, he fails to consider any of the research looking into the impact of teaching with clickers on student learning.

For instance, he quotes Ira David Socol of Michigan State University as saying, “The idea of wasting money on a device no more sophisticated pedagogically than raising your hand drives me nuts…”  However, there’s strong evidence (Stowell and Nelson, 2007) that when the hand-raising method is used, fewer students participate and students are more hesitant to answer questions honestly than when a classroom response system is used.  Those are significant differences and to ignore them is to fail to accurately describe key benefits of using clickers.

Bugeja also writes that had students at his institution been asked to weigh in on the cost-benefit question regarding clickers, “they probably would have said no because of excessive student fees.”  I can’t speak for students at Iowa State, but a number of published studies of student perceptions of clickers, including Trees and Jackson (2007), MacGeorge et al. (2007), and Kaleta and Joosten (2007), indicate that students respond positively to clickers, particularly when clickers are used in ways that engage them in class and provide them with feedback on their learning.  It should be noted that in the studies I just listed, students were required to purchase their own clickers.  Thus, there is evidence that students see the benefits of clickers outweighing the costs.

A second weakness of Bugeja’s argument is that he discusses the cost side of the cost-benefit analysis by focusing on the cost to install and maintain infrared-based classroom response systems.  IR systems are indeed costly to install and maintain and a bit of a pain for faculty and students to use.  However, arguing that classroom response systems aren’t worth the cost because infrared-based systems are costly is a bit like arguing that automobiles aren’t worth purchasing because steam-powered cars are a pain to use.  Very few colleges and universities are still using infrared-based clicker systems.  The radio frequency systems now in common use eliminate almost all of the installation, maintanence, and usage problems of the infrared systems.

As Bugeja points out, at Iowa State relatively few faculty members used clickers when the infrared system was the only one available.  When the easier-to-use and more-reliable radio frequency system was made available, “users then multiplied throughout the university.”  Bugeja makes good points about the costs involved in supporting early versions of clicker systems, but given how usage increased when more mature technologies were made available, I think a stronger takeaway is that institutions should be cautious when implementing new technologies.  Waiting for “version 2″ can help institutions avoid costs.  Universities now in the process of rolling out clickers widely can take advantage of the more mature radio frequency technologies and thus avoid all the hassle of the older systems.

There’s more I could say about this essay, but I’ll stop here for now.  I encourage you to respond to Michael Bugeja’s essay as well as my thoughts in the comments section below.

On the final day of the EDUCAUSE Annual Conference, I attended a session titled “Growing and Sustaining Student Response Systems at Large Campuses: Three Stories” presented by Christopher Higgins of the University of Maryland, Nancy O’Laughlin of the University of Delaware, and Michael Arenth of the University of Pittsburgh.  The presenters’ slides are available, and Inside Higher Ed ran a story on the session, too.

University of Maryland

There were three classroom response systems in use at the University of Maryland as of a few years ago, so the IT office got together with Undergraduate Studies and the Center for Teaching Excellence to form a review committee that recommended the adoption of the TurningPoint system.  Key factors included keeping student data on campus (because of FERPA), cost to students, integration with PowerPoint and Maryland’s course management system, and reporting options.

They now have over 12,000 clickers in the system with at least 75 faculty members using clickers, mostly with courses in business and the natural sciences.  (Some departments purchased their own sets, so IT isn’t sure how many faculty are using clickers in these departments.)  I haven’t spoken with many business faculty about how they use clickers, although the business and management section of my bibliography is one of the larger ones.  I might try to track down a couple of Maryland business faculty to find out how they are using clickers.

Challenges included registering student clickers, which required two different registration systems for a while.  Also, the software isn’t as robust on Macs, which poses a problem for some faculty.  They also went from 10 classrooms with receivers and software to 150 in a single semester which was challenging!  TurningPoint’s receivers also needed upgrading last academic year, which posed some logistical problems.

Currently, the IT office handles technical support for faculty using clickers, while the Center for Teaching Excellence handles training and promotion.  The two units seem to work well together, offering joint training sessions that have gone over well.  IT finds it necessary to have a staff member devoted almost entirely to clicker support at the start of a semester.

Christopher Higgins is particularly excited about TurningPoint’s new ResponseWare Web system, which enables any Web-enabled device (laptop, iPhone, etc.) to function as a clicker.  He likes the fact that the system leverages existing hardware that can also perform other functions, as well as the fact that the Web system is cheaper–$20 per student per year or $40 per student for four years.  Christopher found that many students took advantage of an Apple promotion this fall to purchase iPod Touches and iPhones along with their Mac laptops so a lot of students at Maryland have devices that can run the new TurningPoint system.

University of Delaware

The adoption committee at Delaware included not only faculty members and IT staff, but staff from the assessment offices and, I think, students, as well.  (I may have misheard that last point.)  They standardized on Interwrite PRS and spent the summer of 2006 training faculty and installing receivers and software in all classrooms with at least 75 seats.  (They now have receivers and software in all classrooms with at least 35 seats, which is most of the classrooms on campus.)  By the fall semester about 3,600 students and 40 faculty were using clickers.  More faculty started using clickers in the fall of 2007, but this year there are relatively few faculty new to clickers since most faculty have heard about them and decided whether or not to use them.

Clickers are popular in courses in the natural sciences, as well as psychology, political sciences, and nursing.  Many first-year undergraduate courses use clickers, which means that faculty teaching “downstream” courses are now more likely to use clickers, as well, since most of their students already own the devices.

Clickers are used in non-academic settings on campus, too.  Residential Life uses them to collect information on student experiences and opinions in the dorms.  The library and the office of assessment use them, as well.

Challenges to the support of classroom response systems on campus included a move to a new unique student identifier.  The Interwrite PRS system allows students to enter and store their unique identifiers on their clickers, but it took some work to have all the students request a new unique identifier on the Delaware Web site.  Other challenges included handling new versions of the software and a move from one course management system (WebCT) to another (Sakai).

One process Nancy mentioned that I particularly liked is that when faculty request clickers for their courses from the bookstore, there’s a checkbox on the form that asks them if they are new to using clickers.  Faculty who check this box are then sent resources by Nancy’s office and added to Nancy’s mailing list.  This helps faculty connect to useful pedagogical and technical resources and helps Nancy know who’s using clickers on campus.

Nancy also mentioned that she’s found it helpful to give faculty members their own receivers so they can practice as much as they need to outside of the classroom.  She finds that students know when their teachers aren’t comfortable with a technology, so time for practice is important.

Another point Nancy made was that the code of student conduct at Delaware has been amended to mention clickers.  Students are to respond for themselves, not on behalf of other students.  She indicated that faculty appreciate having this clause in the code since it means there’s a process they can follow if they suspect students of cheating by bringing other students’ clickers to class.

University of Pittsburgh

Things at Pittsburgh have been a little more chaotic.  A review committee consisting of IT staff, facilities staff, and registrar staff decided in 2003 not to adopt a single system on campus.  As a result there are now a few systems in use on campus now.  There’s now some move toward standardizing on eInstruction, but there doesn’t seem to be a central decision-making office that enforces that decision so faculty are still free to use other systems.

Clickers are popular in biological sciences, physics, nursing, and pharmacy.  Also, the School of Social Work uses them frequently in their gambling addiction counselor program.  I wouldn’t mind talking to some of those faculty to find out how they use clickers in that setting.

Michael Arenth named a few challenges they’ve faced at Pittsburgh, including managing faculty expectations (particularly for faculty who get excited by clickers but don’t plan on the time necessary to learn the systems), cheating (students who bring other students’ clickers to class to cheat on attendance grades), and set-up between classes since until recently, they haven’t been installing systems in classrooms.

I believe Michael said that Pittsburgh has still been using infrared clicker technologies until fairly recently switching to radio frequency.  (Most people I’ve talked to made the switch a couple of years ago.)  He noted that the IT group on campus had to approve the use of radio frequencies for this purpose.  I hadn’t heard of this kind of approval before, so I found this point interesting.

Common Issues

All three campuses have surveyed faculty and students about clickers, and they used some common questions to enable comparisons among the three campuses.  They found that faculty frequently use clickers to measure student comprehension, measure student opinion, obtain anonymous responses, monitor attendance, and facilitate quizzes.  The presenters spoke only briefly about these results, and it was unclear to me the extent to which faculty use comprehension or opinion questions to generate small-group or classwide discussion or to practice “agile teaching” by responding to the results of clicker questions during class.  I was, however, happy to see that clickers were used more for formative assessment (measuring comprehension and opinions) than summative assessment (quizzes and tests) since I think that’s where clickers really shine.

An audience member at the presentation asked about the student response to clickers.  The panel indicated that students like the interactivity that classroom response systems provide.  They confirmed what I’ve now heard from multiple sources, that students want to see some value added to their learning experience as a result of the clickers.  If a faculty member just asks a question and quickly moves on, there’s no interactivity and little impact on student learning.  Students don’t respond well to this.

Finally, I spoke with Danny Sohier of Université Laval in Québec after the session.  His school is using clickers to conduct end-of-semester course evaluations during class.  They found that online course evaluations resulted in low response rates, a problem I’ve heard about from many institutions.  They now use clickers to collect student responses to multiple-choice evaluation questions during class in some courses, inviting students to respond to open-ended questions online outside of class.  Danny indicated that this arrangement is working pretty well.  I might follow up with him to learn more about this process.

That’s it for my notes on this session.  I was glad to see a clicker session on the agenda at EDUCAUSE.  I was a little surprised at the number of audience members who asked questions at the end of the session and at the nature of those questions.  It seems there are a lot of institutions that are still just starting to work on adoption and support issues.  That indicates to me that use of classroom response systems will continue to grow over the next few years.

The Chronicle of Higher Education recently posted an essay by Michael Bugeja titled “Could You Be a Hoopla-dite?” in which Bugeja criticizes those at colleges and universities who advocate for instructional technologies without thinking critically about the hidden costs (training, technical support, etc.) of those technologies or without assessing those technologies to see if they are worth their cost.  Bugeja takes a pretty harsh tone in his essay, but he provides some sensible suggestions for institutions adopting instructional technologies.  Check out his seven recommendations at the end of his essay.

Bugeja uses clickers as an example of a technology that wasn’t properly vetted at his institution, Iowa State University.  Unfortunately, this is the only example of an instructional technology he describes in his essay, which makes it a little too easy to see his essay as directed at those who would uncritically advocate the use of clickers.  I don’t think that’s his intention, but as someone who advocates the use of clickers (not uncritically, I like to think), I would have appreciated an example or two of the adoption of other instructional technologies.

Bugeja writes, “We need to assess technology to weed out ineffective systems and applications and preserve those that truly enhance our educational missions.”  This sounds sensible to me.  However, as Steve Ehrmann said on this blog, “When evaluating educational impact, it’s not possible (or desirable) to separate technology from the pedagogy that makes use of it.”  It’s not possible to “assess the product” (one of Bugeja’s recommendations) without assessing the instructional choices teachers make when using the product.  This complicates the task of assessing instructional technologies like clickers. Fortunately, in the case of clickers, there’s a growing body of research that explores the impact on student learning of the ways in which instructors make use of clickers.

What steps has your institution taken to assess the impact of the use of clickers on your campus?  What challenges have you encountered in undertaking assessment efforts?