Student evaluations of teaching (SET): unreliable

Bias Against Female Instructors (Inside Higher Ed, January 11, 2016)

     There’s mounting evidence suggesting that student evaluations of teaching are unreliable. But are these evaluations, commonly referred to as SET, so bad that they’re actually better at gauging students’ gender bias and grade expectations than they are at measuring teaching effectiveness? A new paper argues that’s the case, and that evaluations are biased against female instructors in particular in so many ways that adjusting them for that bias is impossible.
     Moreover, the paper says, gender biases about instructors -- which vary by discipline, student gender and other factors -- affect how students rate even supposedly objective practices, such as how quickly assignments are graded. And these biases can be large enough to cause more effective instructors to get lower teaching ratings than instructors who prove less effective by other measures, according to the study based on analyses of data sets from one French and one U.S. institution.
. . .
     “Student Evaluations of Teaching (Mostly) Do Not Measure Teaching Effectiveness,” was published last week in ScienceOpen Research. Philip B. Stark, associate dean of the Division of Mathematical and Physical Sciences and a professor of statistics at the University of California at Berkeley and co-author of a widely read 2014 paper questioning the reliability of evaluations, co-wrote the paper with Anne Boring, a postdoctoral researcher in economics at the Paris Institute of Political Studies, and Kellie Ottoboni, a Ph.D. candidate in statistics at Berkeley.
. . .
     Stark said he doubted the new study would be the “nail the coffin” for student evaluations of teaching, but said he hoped it will “bring us closer to ending any use of SET for employment decisions.” Still, he said, pretending that such evaluations are strong measures of teaching effectiveness remains “irresistible” to some, for a variety of complicated reasons.
     But could the tide be turning? Stark said he expected class action lawsuits against universities that rely on these evaluations for employment decisions will start this year, and that there’s evidence to support such cases.
     “Our analysis would support an argument that the use of SET has adverse impact on female instructors, at least in the two settings we examined,” he said. “Replication of this kind of experiment and analysis elsewhere would strengthen the argument. Eventually, lawsuits will lead universities to do the right thing, if only to mitigate financial risks.”