-
Target Test Prep 20% Off Flash Sale is on! Code: FLASH20
Redeem
Debunking the “First 7 Questions” Myth
I dont even need to say what the myth is. Everyone already knows - that's how pervasive it is. Ever since the GMAT and GRE CATs launched in the 1990s people have believed that the earlier questions are worth more, that if we could get the first 7 (or 5, or 10) questions in a row right, wed be guaranteed a really high score.
And youve likely also heard that this is a myth from me, from other teachers, from Dr. Lawrence (Larry) Rudner, Chief Psychometrician of GMAC (the organization that makes the GMAT). And yet so many people still talk about it and believe it so who should we believe?
Lets talk about this and, hopefully, lay the myth to rest once and for all.
What is the Myth?
Different people talk about different details if we get the first 5, or 7, or 10 questions right, then well get a high score no matter what else happens, or a higher score than we would otherwise get. (And, conversely, if we do poorly on the first 5, or 7, or 10 questions, then our score will be terrible no matter what, or lower than we would otherwise get.)
How Did the Myth Get Started?
Item Response Theory (IRT), the concept on which the GMAT is based, has been around for more than 50 years. In the 1990s, the GMAT decided to switch from the old-fashioned paper-based format to a new computer-adaptive testing (CAT) format based upon IRT.
During that timeframe, the Educational Testing Service was responsible for producing test items for both the GRE and the GMAT (now, the GMAT test items are produced by another organization, ACT). As ETS developed its new GRE CATs, it did initially have a format that emphasized the earlier questions. The test prep world soon figured that out and ETS redesigned the format so that this would no longer happen. That was the start of the myth: it wasnt actually a myth at first!
But why did it persist? Everyone knew that the early form had been cracked and that ETS responded accordingly by changing things.
In 1999, ETS researchers presented a white paper at the annual meeting of the National Council on Measurement in Education. The researchers, Manfred Steffen and Walter D. Way, presented the results of a number of scenarios they ran to test the IRT-based algorithm and show how it worked. Im going to give you some of that data below, but I want to point something out first: this paper is why the rumor persisted.
The paper itself is completely correct and accurate; in fact, its an example of quite good research. But a lot of that research was misinterpreted by people in the test prep industry misinterpreted to mean that the earlier questions still were worth more and that students should spend a lot more time on those earlier questions. The paper, however, shows just the opposite.
One thing the paper tells us is that, for a true examination level of 650, answering the first two questions correctly vs. incorrectly results in a 31 point score difference (658 vs. 627). Note, though, that this assumes all the other items are identically answered in other words, the examinee who answered the first two questions correctly didnt have to guess on any questions at the end. How does that happen? The examinee doesnt run out of time because the examinee didnt take extra time (or not much, anyway) in the first place. (Also: no, you cant really score 658 or 627 on the test; these are the average results of many simulations.)
One scenario they ran was what happened to the true scoring level if someone got the first, first two, or first three in a row right, or wrong, vs. the final question, the final two, or the final three. The results are very interesting.
For someone with a true scoring level of 750, getting the first 3 questions right (and leaving everything else the same) results in a 10-point lift to 760. Great! We should spend more time on the first three questions, right?
Not so fast. Getting the last three questions in a row wrong results in a 20-point drop to 730. What do these two data points really mean? If you can answer those first three questions correctly without sacrificing later questions, then great. Do it. But chances are pretty good that youd have to spend extra time and then your score will drop at the end. (And this is how the myth was perpetuated people forgot that there are consequences for trying to get the first X number of questions right. You have to take extra time!)
At the 650 true-scoring level, getting the first 3 right results in a 16-point jump, to 666. Getting the last three wrong, however, results in a 15-point drop, to 635. Six of one, half a dozen of the other it doesnt really matter.
I will concede that, at lower scoring levels (sub-550), a strategy that involves getting the first 3, or 5, or 7 in a row right works in theory because there isnt much of a drop at the end for getting a similar number of questions wrong. Theres only one problem with this strategy. What are the chances that someone at a true scoring level of 500 is going to get the first 5 (or even 3) questions in a row right? Think about what happens each time you get a question right.
Next, the study talks about scenarios in which someone gets the first ten questions right or the last ten questions right. Heres an interesting statistic: if a test-taker with a true scoring level of 670 gets the first 10 questions right, the study showed that the resulting score would be 728. Clearly, we should spend that extra time on the first 10 questions!
Except for one little detail. That part of the study assumed that the test-taker didnt have to rush on any other questions. In other words, the study assumed that the test-taker didnt need any significant extra time in order to get those first 10 questions right. (And, again, we missed this in the 90s when interpreting this data.)
By the way, what can you do if you simply dont know how to do a problem? Will spending an extra minute or two help? The vast majority of the time, no. If you cant do it in the normal time (or perhaps about 30 seconds above normal time), then this just means that you dont actually know what youre doing, since there is a normal time solution. Spending even more time, then, is not going to do much (except blow time).
And finally we get to the portion of the report that mimics real-life conditions the best: what the report calls the early care / late guessing condition. In this scenario, the test-taker takes additional time on some early number of questions and then has to guess on questions towards the end in order to finish the test on time. Theres one more not-so-minor detail. This scenario assumed that, for the early care (extra time) situation, the tester would answer every single question correctly. That is you spend more time, you automatically get it right. I dont need to point out that life doesnt really work that way, right? :)
If a test-taker with a true scoring level of 500 answers the first 5 questions correctly, then that test-taker is likely to end up with a higher-than-500 score as long as he doesnt guess on more than 6 questions at the end. That sounds pretty good until you remember that this requires a 500-level tester to answer the first 5 questions in a row correctly. Again, remember what happens when you answer questions correctly.
What about a tester at the 780 level? This tester has a pretty good shot at answering the first 5 in a row correctly. In this scenario, however, the tester cannot guess on more than 1 question at the end without the score dropping below 780. Only 1 question!! If this tester answers the first 10 questions in a row correctly, he can guess on no more than 3 questions before the score level drops below 780.
What does all of this mean?
If youre going for a 750+ score, then the strategy actually boils down to this: get the first 5, or 7, or 10 in a row right while spending barely any extra time so that you have to guess on zero or very few questions at the end otherwise, your score will actually go down. (And if you can actually do this, then you dont need to worry about trying to get certain questions right based on their position in the test. You just do all the questions normally.)
What about a 500-ish score? Were allowed to guess more at the end but wed still have to get 3+ questions in a row right at the beginning of the test, and we all know how the test works. Im going to start the test with a medium-level problem. If my level really is around 500, theres no way Im going to get 3 in a row right, because that third question is going to be way too hard for me no matter how much time I spend (and possibly the second one as well).
What about in the 600 range or right at 700? The final scenario in the study (early care / late guessing) didnt provide data for these specific scores. But look at all of the data given collectively in the paper. We havent found one case in which spend more time on the early ones actually works. Basically, weve got a tug of war between how many questions we really could get right in a row and how many times wed have to guess at the end as a result and the data shows that theyve figured out how to balance this in such a way that gaming the test is just going to backfire in the end.
Takeaways
1. It really is a myth. Dont spend lots of extra time on any one problem anywhere in the test. Its not worth it.
2. The real strategy that derives from the research is: get everything right that you can without spending extra time. (Extra = more than about 30 seconds above the average for the question type, since youll also have some that are a bit faster than the average.)
3. Read these two articles: In It To Win It and Time Management.
All data points cited from Test-Taking Strategies in Computerized Adaptive Testing, Steffen, M. and Way, W. D., Educational Testing Service, April 1999. Presented at the annual meeting of the National Council on Measurement in Education.
Recent Articles
Archive
- April 2024
- March 2024
- February 2024
- January 2024
- December 2023
- November 2023
- October 2023
- September 2023
- July 2023
- June 2023
- May 2023
- April 2023
- March 2023
- February 2023
- January 2023
- December 2022
- November 2022
- October 2022
- September 2022
- August 2022
- July 2022
- June 2022
- May 2022
- April 2022
- March 2022
- February 2022
- January 2022
- December 2021
- November 2021
- October 2021
- September 2021
- August 2021
- July 2021
- June 2021
- May 2021
- April 2021
- March 2021
- February 2021
- January 2021
- December 2020
- November 2020
- October 2020
- September 2020
- August 2020
- July 2020
- June 2020
- May 2020
- April 2020
- March 2020
- February 2020
- January 2020
- December 2019
- November 2019
- October 2019
- September 2019
- August 2019
- July 2019
- June 2019
- May 2019
- April 2019
- March 2019
- February 2019
- January 2019
- December 2018
- November 2018
- October 2018
- September 2018
- August 2018
- July 2018
- June 2018
- May 2018
- April 2018
- March 2018
- February 2018
- January 2018
- December 2017
- November 2017
- October 2017
- September 2017
- August 2017
- July 2017
- June 2017
- May 2017
- April 2017
- March 2017
- February 2017
- January 2017
- December 2016
- November 2016
- October 2016
- September 2016
- August 2016
- July 2016
- June 2016
- May 2016
- April 2016
- March 2016
- February 2016
- January 2016
- December 2015
- November 2015
- October 2015
- September 2015
- August 2015
- July 2015
- June 2015
- May 2015
- April 2015
- March 2015
- February 2015
- January 2015
- December 2014
- November 2014
- October 2014
- September 2014
- August 2014
- July 2014
- June 2014
- May 2014
- April 2014
- March 2014
- February 2014
- January 2014
- December 2013
- November 2013
- October 2013
- September 2013
- August 2013
- July 2013
- June 2013
- May 2013
- April 2013
- March 2013
- February 2013
- January 2013
- December 2012
- November 2012
- October 2012
- September 2012
- August 2012
- July 2012
- June 2012
- May 2012
- April 2012
- March 2012
- February 2012
- January 2012
- December 2011
- November 2011
- October 2011
- September 2011
- August 2011
- July 2011
- June 2011
- May 2011
- April 2011
- March 2011
- February 2011
- January 2011
- December 2010
- November 2010
- October 2010
- September 2010
- August 2010
- July 2010
- June 2010
- May 2010
- April 2010
- March 2010
- February 2010
- January 2010
- December 2009
- November 2009
- October 2009
- September 2009
- August 2009