Neither student course evaluations nor student results on curriculum-aligned standardized tests are good predictors of professor quality. In fact, even though students tend to give better reviews to instructors who have higher test scores, both of these performance evaluation tools are severely flawed. It turns out that professors with lower test scores and more negative student reviews are the ones adding the most value to college students’ educations.
If this seems completely illogical to you, you are well-suited for reading the rest of this post.
The conclusions above were reached by two professors—Scott E. Carrell of UC Davis and James E. West of the U.S. Air Force Academy—in a new research paper, “Does Professor Quality Matter? Evidence from Random Assignment of Students to Professors,” featured in the June 2010 edition of the Journal of Political Economy. As the title suggests, 10,534 students who attended the U.S. Air Force Academy from the fall of 2000 to the spring of 2007 were randomly assigned to required introductory courses. During their time at the USAFA, their academic performance in each course, as measured by standardized course exams, was tracked as they progressed through the curriculum.
To minimize any bias in this experiment, all instructors teaching core courses used common syllabi and all students took standardized course exams that were graded by several professors (Professor A graded problem 1, Professor B graded problem 2, etc.). In the study, Carrell and West set out to determine which instructors caused the greatest amount learning for students, both in the “contemporaneous course” (i.e., the course taught by the instructor) and in “follow-on courses” (i.e., sequence courses taken after the contemporaneous course). An example of a contemporaneous course would be Calculus I, and some of its follow-on courses are Calculus II and aeronautical engineering.
Using advanced statistical methods, Carrell and West created a value-added model for instructors, which allowed them to isolate each instructor’s contribution to student achievement in the actual course that she taught and to her students’ achievement in follow-on courses. They found that instructors who cause higher student achievement in the courses they teach are less experienced, less likely to hold a terminal degree, and mostly not tenured professors. These instructors also receive better evaluations from their students. On the other hand, their model finds that instructors who cause less student achievement in the actual courses they teach, cause more achievement in follow-on courses. These instructors are more experienced, more likely to hold a terminal degree, and mostly tenured faculty.
So, how can one possibly make sense of this paradox? Apparently, it can be explained away by experienced, tenured professors providing students with some kind of enigmatic “deeper learning”:
Results show that there are statistically significant and sizable differences in student achievement across introductory course professors in both contemporaneous and follow‐on course achievement. However, our results indicate that professors who excel at promoting contemporaneous student achievement, on average, harm the subsequent performance of their students in more advanced classes. Academic rank, teaching experience, and terminal degree status of professors are negatively correlated with contemporaneous value‐added but positively correlated with follow on course value‐added. Hence, students of less experienced instructors who do not possess a doctorate perform significantly better in the contemporaneous course but perform worse in the follow‐on related curriculum.
Student evaluations are positively correlated with contemporaneous professor value‐added and negatively correlated with follow‐on student achievement. That is, students appear to reward higher grades in the introductory course but punish professors who increase deep learning (introductory course professor value‐added in follow‐on courses). Since many U.S. colleges and universities use student evaluations as a measurement of teaching quality for academic promotion and tenure decisions, this latter finding draws into question the value and accuracy of this practice.
As you can see from the excerpt above, it’s pretty clear that Carrell and West are searching for an intellectual argument to undermine student course evaluations and the common sense method of judging instructors by their students’ achievement on material from the actual courses they teach. If you’re like me, you are utterly perplexed by a system that would mostly determine the quality of a Calculus I instructor by students’ performance in a Calculus II or aeronautical engineering course taught by a different instructor, while discounting students’ mastery of Calculus I concepts.
The trouble with complex value-added models, like the one used in this report, is that the number of people who have the technical skills necessary to participate in the debate and critique process is very limited—mostly to academics themselves, who have their own special interests. I hope that this report isn’t used by the media or lobbyists to claim something like, “Research shows that tenured faculty are more effective instructors because they provide students with a deeper understanding of concepts. Course-aligned standardized exams and instructor evaluations by students are poor tools for assessing professor quality.”
To the skeptical consumer of higher education, this research report appears to be an attempt to erode support for evolving professor evaluation tools, while replacing these tools with evaluation methods that are designed by (and can only be understood by) the elite suppliers of higher education. The authors must better communicate their findings to the public and policy makers to displace this perception.
Thanks to Greg Mankiw for directing me to this research report. He calls the findings “fascinating.”
Click Image To Enlarge
5/11/2012 | U.S. News & World Report highlights ideas from "Moving On Up: How Tuition Tax Breaks Increasingly Favor the Upper-Middle Class." 
5/11/2012 | In Diverse, Steve Burd comments on the Obama administration's plans to address student debt. 
5/11/2012 | Kevin Carey talks to Salon about the government's role in rising student debt.
{ 3 comments }
The worm in this pudding, I suspect, is Mr. Hinton’s view of higher education as a consumerist commodity market. (“To the skeptical consumer of higher education….”)
I would also suspect that rather than “seeking to undermine” the latest fashion in classroom consumption, i.e., Taylorist-flavored quality control evaluations that reassure anxious consumers that they are getting their money’s worth with an expensive commodity (which itself can be leveraged into making the commodity more expensive and desirable to consumers), Carrell and West were speaking to, and from, an entirely different epistemology of value at a very fundamental level, and despite the study’s framing as “political economy.”
My own experience in the Ed Biz over four decades led me to conclude that the enduring value of a quality education is not measured, or even measurable, in any consumerist schema, and surely not over the short term. The point of higher and advanced education is not to make the student succeed. The point is to challenge the student, and to impart imperfect, but gradually perfectible habits and skills, to a very imperfect, but gradually improveable, human. The point is to plant seeds that will sprout and grow over a lifetime, and to give to the student both the skills and perspectives that put that gardening in their hands.
This plays out differently for, say, the empirical sciences and the humanities, and that is a flaw in many studies of “evaluating effective teaching.” Effective teaching in engineering imparts skills that mean the bridge or building won’t fall down. Effective teaching in the humanities imparts skills that mean that when the student falls down, it will have inner resources for righting itself. Sadly, we have come around to a view of the individual that they are helpless in the face of mistakes or ill fortune, and cannot self-repair. We have come around to a view that disrespects their resilience, and instead gives them the out of victimhood and dependency on experts to fix them.
For many students, what they need the most in their teens isn’t to feel good about their education, but less good about what they think they know. It’s like parenting; they’ll hate you for years, and thank you eternally when they mature. In my view students shouldn’t be allowed to evaluate teachers until their 20th reunion.
Teenage social popularity has become the model for both teaching AND parenting. The result is several generations of young people—including many now in tenured teaching positions—who are helpless outside of that constrained and artificial model of interaction. Faced with any challenge to their certainty, skills or self esteem, they crumble spectacularly…or demand the system change to accommodate them. This is contributing to an increasingly adolescent (read: consumerist) social system.
I turned down multiple tenure-track teaching positions, despite being (I was told) an excellent teacher. I knew that I could not remain an excellent teacher within tenured academe. I saw what happened to my friends and colleagues, how their ability to challenge students quickly turned into a tango of seduction and appeasement both of the students and their bill-paying parents. When I say I was told I was an excellent teacher, my student evaluations were usually about 1/2 “Great class, great teacher, worth my time,” and 1/2 like this: “I didn’t like this class, I didn’t like the teacher, I learned so much and someday I’ll appreciate it more than I do now. Don’t take this course if you want an easy B.” Also, I had very low dropout rates (4 in hundreds). That was my #1 metric. It’s like a marriage or a family; in any given time you may not like it, you may find it ridiculously difficult, but it may be the most worthwhile thing you ever do. What matters is that we motivate each other to hang in there. But again, I measured teaching as a human and personal relationship, not an economic one.
Since higher education has been remade as the bootcamp of class privilege and class leapfrogging, students have been turned into focus groups, which is really what all this talk of “student evaluations to measure teaching effectiveness” boils down to. Did they like it, will they buy it again, and how will we redefine the classroom experience in terms of that satisfactory consumer experience, so we can sell more and more of the product.
I responded to Michael’s blog post here http://permut.wordpress.com/2010/06/11/professor-quality-and-professor-evaluation/#comment-337
I blogged about this paper as well here: http://permut.wordpress.com/2010/06/11/professor-quality-and-professor-evaluation/ I’ll add a link here from my post.
I think this is a really interesting paper but I agree that we should consider multiple potential explanations for the result.
Comments on this entry are closed.
{ 4 trackbacks }