How Does Question Difficulty Order Affect Evaluations of Test Performance?
By Yana Weinstein
(Cover image by kinkate from Pixabay)
If we want to improve student learning, we also need to worry about students' attunement with their own memory performance. That is, if students can't gauge how well they did on a test, they're going to have more trouble preparing adequately for the next one.
Last year, I wrote about about how difficult it is to improve the accuracy of students’ self-evaluations - particularly when those students are not performing well in a class. The low-performing students tend to be overconfident, and this overconfidence is very hard to correct (1), (2). Today I wanted to talk about another judgment bias - one that appears across the board, regardless of whether students are high or low-performing.
It turns out that something as simple as the order in which questions are presented on a test affects students' evaluations of their own performance. I was inspired to pursue this line of research 10 years ago, when a talk by Nobel Prize winner Daniel Kahneman sparked the idea in my mind.
Daniel Kahneman, image by Buster Benson (CC BY-SA 2.0) from Flickr
Kahneman was talking about his research on judgment bias, including his study of colonoscopy patients (bear with me - this really does link back to learning eventually!). Kahneman's team discovered that when patients had a colonoscopy and were later asked about how painful it was, their judgments did not accurately reflect the total amount of pain they had reported feeling during the procedure (3). Instead, their overall post-hoc pain judgments were heavily influenced by the very end of the procedure - if it ended very painfully, that was what they remembered most. (The actual results of that study are a little more complicated, but this aspect of the data made the lightbulb go off in my head).
I started to wonder: are tests like a colonoscopy? Will students remember the test as being more painful (i.e., harder), if it ends with difficult questions, thereby leading to lower performance evaluations than when that same set of questions is reversed and therefore ends with easy questions? This seemed plausible, but research from another domain suggested the opposite hypothesis.
Over in social psychology, researchers were looking at impression formation by exposing participants to lists of adjectives describing a person. Here, the finding was that adjectives presented at the beginning of a list weighed more heavily on impressions than did adjectives presented at the end of the list (4). That is, if I tell you someone is greedy, arrogant, rude, and also insightful, honest, and funny, you're less likely to think they're a great person than if I flipped those two sets of qualities. This suggests the opposite hypothesis to the colonoscopy studies: perhaps if difficult questions appear at the beginning, that will lead to students to think they did more poorly on the test, because their impression of the test will be more negative.
I'm going to end this blog post on a cliffhanger, because for those of you who are not familiar with this line of my research, it would be fun to see what you think.
I'll answer in my next blog post!
References:
(1) Hacker, D. J., Bol, L., Horgan, D. D., & Rakow, E. A. (2000). Test prediction and performance in a classroom context. Journal of Educational Psychology, 92, 160-170.
(2) Kruger, J., & Dunning, D. (1999). Unskilled and unaware of it: how difficulties in recognizing one's own incompetence lead to inflated self-assessments. Journal of Personality and Social Psychology, 77, 1121-1134.
(3) Redelmeier, D. A., Katz, J., & Kahneman, D. (2003). Memories of colonoscopy: a randomized trial. Pain, 104(1-2), 187-194.
(4) Anderson, N. H. (1965). Averaging versus adding as a stimulus-combination rule in impression formation. Journal of Experimental Psychology, 70, 394-400.
