A book given to people taking the GRE exam advises them that “Exercise great caution if you decide to change an answer. Experience indicates that many students who change answers change to the wrong answer.”
This advice represents a widespread belief that our instincts, our intuitive senses, are the most reliable guides to decisions. It is based on the assumption that instincts are based on prior knowledge and experiences and that our brains integrate all these things to enable us to make quick judgments that tend to be sound. This is the idea heavily promoted by Malcolm Gladwell in his book Blink.
But is it true?
I was interested in this study (published in the Journal of Personality and Social Psychology, 2005, Vol. 88, No. 5, 725–735 by Justin Kruger, Derrick Wirtz, and Dale T. Miller) titled Counterfactual Thinking and the First Instinct Fallacy (subscription required but you can see a fairly detailed discussion of the paper here) that examined the answers given by 1561 students in response to a multiple choice test. By looking at the erasures and comparing them with the final answer, they found that 51% changed their answers from wrong to right, 25% from right to wrong, and 23% from wrong to wrong. In other words, when people changed their minds, they were twice as likely to go from wrong to right as from right to wrong.
In fact, the authors point out that research over a period of 70 years provides overwhelming evidence that these ‘second guesses’ are more likely to be correct. So given this clear evidence over a long period of time that second guessing leads to better outcomes, why do people still think that sticking with the first choice is the best option?
As I have written before, our initial reactions are based on System I thinking that “operates automatically and quickly, with little or no effort and no sense of voluntary control” and does not engage the higher order thinking skills that are brought in by System 2 which is slower because it “allocates attention to the effortful mental activities that demand it, including complex computations” (Thinking Fast and Slow, Daniel Kahneman, 2011, p. 20). Most of the time when confronted with novel problems, we tend to form an initial response based on the quick intuitive System 1 thinking instead of taking the time to engage the slower but more analytical System 2 thinking because it requires less intellectual effort. We tend to use System 2 only when we are forced to do so, even though it gives better results. But when we ‘second guess’, we are more likely to be using the more reliable System 2.
But there is more to the desire to stick with our first guess than the tendency to avoid intellectual effort. In a separate experiment that involved only 23 students (but still has p<0.05), Kruger, Wirtz, and Miller found that people feel worse when they change a correct answer to an incorrect one (74%) than when they stick with an original incorrect answer (0%!). 26% felt that they would feel no difference in regret. Furthermore, students who switched from a correct answer to an incorrect one were far more likely to remember it later. All these make changing right answers to wrong more memorable than a wrong-to-right change and since we tend to assign higher probability to more easily recalled events, this makes it seem far more probable that second guesses lead to wrong decisions than it really is.
This might also explain the psychology behind the anomalous results of the Monty Hall problem. Most people are aware of this game. There are three closed doors, behind one of which is a valuable prize, while the other two have nothing. People are asked to pick a door. After they have chosen, the door is not opened but one of the remaining two doors is opened to reveal that there is nothing there. The contestant then has the option of sticking with the first choice or switching to the remaining unopened door.
Statistically, you are better off switching than sticking with your first choice. This is because there is a 2/3 chance that your first choice was wrong (since you had three options then) but only a ½ chance that switching would result in the wrong door (since you now have only two options) and so it is twice as likely that the prize was behind one of the other doors. Opening one of them later does not change those odds.
But most people stick with their first choice. Why? Part of it is undoubtedly due to a misunderstanding of statistics. They think that since there are only two choices after one of the doors is opened, it is 50-50 as to which of the two remaining closed doors conceals the prize so it does not really matter. But even then, you would expect half to stick with the first choice and half to switch but that is not the case.
One can put this down to inertia. Why switch if there is nothing to be gained? But there is more to it than that and the psychology behind not switching is not hard to understand. It is because people regret more those things they had and then lost than those things they never had in the first place. This desire to avoid loss is very strong. In the Monty Hall problem, if it is revealed that they switched from the right choice to the wrong one, this is accompanied by the feeling that they had the valuable prize in their grasp and then threw it away, whereas if they stick with their choice and it is wrong, they feel that they never really had the prize in the first place.
So the reason that we are reluctant to change our minds once we have made even a quick and hurried decision is driven by our desire to avoid the great feeling of regret that we had it right and then blew it.
“there is a 1/3 chance that your first choice was wrong (since you had three options then), but only a ½ chance that switching would result in the wrong door (since you now have only two options). ”
Don’t you mean a 2/3 chance that your first choice was wrong?
I haven’t taken the all electronic version of the GRE, but back in the pencil and paper days, whether it was a good idea to change an aswer or not depended on the position of the question in a group of questions.
Suppose there are nine questions in a group. The first six will be straightforward, though the second group of three in that six can be pretty challenging. The last three won’t really be any more difficult, but the question may be harder to interpret and will always contain a decoy answer, an answer choice that is easy to think correct based on an incorrect interpretation of the question.
Different test prep guides recommended slightly different strategies for when to switch and when to not switch. Kruger et al. may have done some solid work on multiple choice tests in general, but unless their test had decoy answers like the GRE at least used to, there may be more going on with the GRE than there was in this study.
Still, it’s good to see someone doing some solid research into this.
Yes, you are right and I have corrected it. Thanks!
What I think is being missed in this discussion is that the GRE test is against the clock. Thus, if too much time is spent second guessing oneself over too many questions, one may not be able to finish before the clock ends the test and a number of questions might not be answered at all.
Good point. A timed test may be deliberately testing your System 1 thinking and they are discouraging the engagement of System 2 thinking. OTOH, they may be selecting people with particularly fast System 2 thinking.
You’re right that switching is statistically prudent, but your reasoning is off a little bit. You have a 1/3 chance of initially picking the correct door, but once the second door is opened you don’t have a 1/2 chance of picking the right door if you switch. You actually have a 2/3 chance, because you had a 2/3 chance of being wrong in the first place.
Thanks for the correction. The reasoning was very sloppily worded. I have made a change that I think is better.
I have seen published arguments between mathematicians about the statistics of the ‘Monty Hall’ game which have ended without reaching a resolution. That is because the problem is, routinely, not presented with all the necessary information.
It is critical, to justify the course of action that you recommend to the contestant, to understand that the game show host knows which door conceals the prize and opens the door *knowing* there is nothing behind it.
If, OTOH, the host is not ‘in the know’ and his choice between the two remaining doors is perfectly random (as I assumed when the problem was, inadequately, presented to me!) then the odds have not been altered and there is no advantage to changing the original choice made.
Mano, isn’t it a tiny bit at odds with your post from yesterday telling us to “trusting our instincts”?
Yes, it does seem like a bit of a contradiction at first blush. The difference is that there the advice was to follow one’s instincts in the face of contradictory advice if that advice was not authoritative. After all, we did take Spock’s advice. There you were urged to give preference to your instincts if there was no convincing reason to do otherwise.
The instinct being referred to in this post was the first reaction one had to a situation or problem, before one had thought through the alternatives. If the alternatives are not convincing, you should go with your instincts here too.
Mano -- Just to draw your attention to what Colin said above, the only way the chances of picking the right answer would go up by switching is if someone who knows the right answer crosses off one of the remaining wrong answers for you. If there is no new information entering the “system”, your odds do not change…right?
Yes, that’s right.
I have always heard of the Monty Hall problem formulated as one in which the host knows which door has the real prize and makes sure that the door that is opened does NOT have the prize.
I am not sure how the game could function properly otherwise since if the host did not have such knowledge and control, the door that has the prize may be opened. Then what happens to the contestant?
Back in the stone age when I did the GRE and other multiple choice tests, I would go through the whole test quickly, while also putting a mark against those that I was doubtful about and would return to once I had attempted all. That way, I could second guess without the worry you mention..
Isn’t that strategy given to all test-takers?
If the host, randomly, opens the door with the prize, it simply means that the contestant has lost. (Although, given the history of rigged game shows, she would do well to demand that he also opens the door she had chosen).
I think that my assumption (that the host didn’t know where the prize was) was perfectly reasonable – it would still add to the general ‘excitement’. Perhaps my ‘problem’ is that I haven’t watched a TV game show since Pick-a-Box 45 years ago. Or, as my wife, generally good-naturedly, suggests, that I am somewhat Asbergerish!
‘Lies, damned lies …etc ……’
Back in the stone age when I also took the GRE and earlier the SAT, I did the same thing. It works, provided that there aren’t too many such questions hanging out there. The tests back then were designed so that speed of response was tested as well as accuracy. The tests were designed so that the average student (e.g. who would score 1000 in the SAT) would barely be able to finish in the time allotted.
Another issue that arises from multiple choice tests is that a wrong answer results in a negative score for that question (e.g. if there are 5 choices and one chooses a wrong answer, 1/4 of a point is deducted from the total score, whereas leaving it blank results on a net of 0 for that question). The rule of thumb was that, if one or more choices could be definitively ruled out, guessing amongst the remaining choices was advantageous, otherwise leaving the question blank was the correct choice.