OK, a friend and I have been trying to figure out how this GMAT scoring system works by playing around with GMATPrep many many times. Here are some thoughts so far (all based on the quant section only):
1. Have you ever noticed that sometimes GMAT throws you a ridiculously easy question? Eg. on one test I got the first 12 questions correct. Question 11 was a hard geometry PS, and question 12 was a moderately hard DS question. I answered both of these correctly. Question 13 was: What is (-3)^(-2), a question that we can pretty much do in our heads. What's the explanation for this? My mate and I think that it can be one of two things: (1) GMAT sometimes throws you a weird question (we consider this unlikely); (2) it is an experimental question (more likely).
It is our belief that these experimental type of questions come up when GMAT believes it "knows" your difficulty level. Eg. by getting the first 12 correct, GMAT is sufficiently confident about your level and starts to throw experimental questions at you. However, if your performance in the first 15 questions was patchy, you will only start to see these "weird" questions from 15 onwards.
2. What of the belief that later questions are worth less than earlier questions? Some GMAT tutors believe this but others don't. However, I will say this: if my conclusion in 1 above is correct, then we can naturally assume that there are less experimental questions (which are not scored) in the first half compared to the second half. All other things being equal, this makes the first half more valuable than the second half.
To prove this point, I have been trying to deliberately do two tests: get the first 10 questions wrong, but the remainder incorrect and compare this with the situation where I get everything correct except for the last 10 questions. I haven't managed to do this properly yet, however in my experimentation I have found that if you do poorly on the first 10 questions, you will most likely score lower than if you had not.
3. Lastly, a friend of mine after doing the test maybe 8 or 9 times believes that this is rougly a guide to how you will score in the quant section. Note that this assumes you only make a maximum of 3 mistakes in the first 10 questions.
6-10 incorrect = 50
10-12 incorrect = 49
12-15 incorrect = 48
After this, it becomes a little bit unpredictable.
Anyway, please note that I am not saying this is gospel. I just wanted to share my observatgions with the group and get some feedback.
Finally, I borrowed Cracking the GMAT 2007 from Princeton Review a few days ago. They say that their instructors regularly sit the test, try to reverse-engineer the scoring system, etc, etc. The view is that earlier questions count more than later questions.
5-Day Free Trial
5-day free, full-access trial TTP
Available with Beat the GMAT members only code
MORE DETAILS
Some reflections on GMAT algorithm - what do you guys think
This topic has expert replies
Hi Leo,
This is Nauman. I and my friend agree with your conclusions about GMATprep. We also did some experiments and reached the same conclusions. But the mystery remains the same: why some people feel that the GMATPrep results are inconsistent with the actual GMAT? This cannot be resolved unless until we get some consistent observations about the actual GMAT or the GMAC/Pearson starts giving feedback on the actual test just as ETS does for GRE. I think we should ask GMAC/Pearson to start giving feedback on actual tests so that one could address his or her weaknesses. If one feels that he/she prepared a lot and did better on the test then GMAC/Pearson should give feedback on the actual test. This will save our individual and collective resources by not experimenting with the GMAT again and again. I have talked with the GMAC official and told them my observation about the GMATPrep inconsistency with the actual test and sent them an email. Let’s see what they come up with.
This is Nauman. I and my friend agree with your conclusions about GMATprep. We also did some experiments and reached the same conclusions. But the mystery remains the same: why some people feel that the GMATPrep results are inconsistent with the actual GMAT? This cannot be resolved unless until we get some consistent observations about the actual GMAT or the GMAC/Pearson starts giving feedback on the actual test just as ETS does for GRE. I think we should ask GMAC/Pearson to start giving feedback on actual tests so that one could address his or her weaknesses. If one feels that he/she prepared a lot and did better on the test then GMAC/Pearson should give feedback on the actual test. This will save our individual and collective resources by not experimenting with the GMAT again and again. I have talked with the GMAC official and told them my observation about the GMATPrep inconsistency with the actual test and sent them an email. Let’s see what they come up with.
GMAT/MBA Expert
-
Stacey Koprince
- GMAT Instructor
- Posts: 2228
- Joined: Wed Dec 27, 2006 3:28 pm
- Location: Montreal, Canada
- Thanked: 639 times
- Followed by:694 members
- GMAT Score:780
Straight from the lips of the guy who makes the GMAT algorithm: experimental questions are randomly distributed. They can occur anywhere. You can't have a lot in a row - so if you pick any 5 questions in a row, you know they won't all be experimental. But they can be early, late, or anywhere in between.
The earlier questions are NOT worth any more than the later questions. I have written about this extensively - do some searches on the forums. A myth arose because some people misinterpreted an academic study on IRT (Item Response Theory, the algorithmic theory on which the GMAT is based). I actually wrote a post explaining how the myth came about and refuting it. Read it.
The nutshell is that, though there are slightly wider "swings" at the beginning of the test in difficulty level from question to question, there are severe penalties at the end for leaving questions blank or answering the final questions incorrectly without any correct answers to "break up" the string of wrong answers. These two things cancel each other out.
The experiment you are trying to do (get the first 10 questions wrong) is not illustrating what you think it is illustrating. The issue is not where the questions are - the issue is that they are all in a row. If you get 10 questions wrong in a row ANYWHERE on the test, you will do more poorly than you otherwise would. Of course, this is theoretical anyway... because you won't get 10 questions wrong in a row! Be careful about trying to draw conclusions using extreme (read: would never happen that way) theory.
And, by the way, the test prep company you discuss is absolutely wrong about the earlier questions being worth more. I used to work there, full time, in product development and marketing. I know exactly what they base this claim on - the academic study I referred to above. I used to say the same thing when I worked there, because I just repeated what I was told and didn't read the study for myself.
Now, I've read it. They have misinterpreted the results - in fact, the results actually showed that, if you have to guess on only 4 questions at the end as a result of spending more time on the first 7 supposedly "worth more" questions at the beginning, it will actually result in a score decrease - the more questions on which you guess, the more of a decrease you'll see. The official study result was really: if you can get the first 7 questions right in a row without spending extra time, it can result in a score increase. Someone who can actually do that is already scoring in the 99th percentile - so there's no practical usefulness for us from this theoretical study.
In addition, I was at a conference in October and heard the guy who is in charge of the GMAT algorithm speak extensively about his work. He once again thoroughly refuted the myth. Representatives from the other major test prep companies were also in the room. It baffles me why anyone in that room would keep repeating this urban legend - more than that, it actually angers me that people who should know better keep perpetuating the myth, because it messes you guys up. Don't drink the Kool-Aid! (For those who don't get my little pop culture reference: don't fall for it.)
Thanks for listening to my little rant.
The earlier questions are NOT worth any more than the later questions. I have written about this extensively - do some searches on the forums. A myth arose because some people misinterpreted an academic study on IRT (Item Response Theory, the algorithmic theory on which the GMAT is based). I actually wrote a post explaining how the myth came about and refuting it. Read it.
The nutshell is that, though there are slightly wider "swings" at the beginning of the test in difficulty level from question to question, there are severe penalties at the end for leaving questions blank or answering the final questions incorrectly without any correct answers to "break up" the string of wrong answers. These two things cancel each other out.
The experiment you are trying to do (get the first 10 questions wrong) is not illustrating what you think it is illustrating. The issue is not where the questions are - the issue is that they are all in a row. If you get 10 questions wrong in a row ANYWHERE on the test, you will do more poorly than you otherwise would. Of course, this is theoretical anyway... because you won't get 10 questions wrong in a row! Be careful about trying to draw conclusions using extreme (read: would never happen that way) theory.
And, by the way, the test prep company you discuss is absolutely wrong about the earlier questions being worth more. I used to work there, full time, in product development and marketing. I know exactly what they base this claim on - the academic study I referred to above. I used to say the same thing when I worked there, because I just repeated what I was told and didn't read the study for myself.
Now, I've read it. They have misinterpreted the results - in fact, the results actually showed that, if you have to guess on only 4 questions at the end as a result of spending more time on the first 7 supposedly "worth more" questions at the beginning, it will actually result in a score decrease - the more questions on which you guess, the more of a decrease you'll see. The official study result was really: if you can get the first 7 questions right in a row without spending extra time, it can result in a score increase. Someone who can actually do that is already scoring in the 99th percentile - so there's no practical usefulness for us from this theoretical study.
In addition, I was at a conference in October and heard the guy who is in charge of the GMAT algorithm speak extensively about his work. He once again thoroughly refuted the myth. Representatives from the other major test prep companies were also in the room. It baffles me why anyone in that room would keep repeating this urban legend - more than that, it actually angers me that people who should know better keep perpetuating the myth, because it messes you guys up. Don't drink the Kool-Aid! (For those who don't get my little pop culture reference: don't fall for it.)
Thanks for listening to my little rant.
Please note: I do not use the Private Messaging system! I will not see any PMs that you send to me!!
Stacey Koprince
GMAT Instructor
Director of Online Community
Manhattan GMAT
Contributor to Beat The GMAT!
Learn more about me
Stacey Koprince
GMAT Instructor
Director of Online Community
Manhattan GMAT
Contributor to Beat The GMAT!
Learn more about me
-
aim-wsc
- Legendary Member
- Posts: 2469
- Joined: Thu Apr 20, 2006 12:09 pm
- Location: BtG Underground
- Thanked: 85 times
- Followed by:14 members
I like when people play with GMAT prep test. Thanks for making the analysis and sharing it with the forums.
I too had spent quite a much time doing that, just to end up concluding that it's a shear waste of time. :mrgreen:
GMAT prep test and the real GMAT test consist of experimental questions which are RANDOMLY PLACED throughout the test.
And thus we can't comment as whether first few questions are important or not. We also cannot make a conclusion as GMAT gauges ones ability and level and then throws experimental questions, since as I said ( & Stacey in her last post mentioned) e-questions are randomly distributed.
I too had spent quite a much time doing that, just to end up concluding that it's a shear waste of time. :mrgreen:
GMAT prep test and the real GMAT test consist of experimental questions which are RANDOMLY PLACED throughout the test.
And thus we can't comment as whether first few questions are important or not. We also cannot make a conclusion as GMAT gauges ones ability and level and then throws experimental questions, since as I said ( & Stacey in her last post mentioned) e-questions are randomly distributed.
Getting started @BTG?
Beginner's Guide to GMAT | Beating GMAT & beyond
Please do not PM me, (not active anymore) contact Eric.
Beginner's Guide to GMAT | Beating GMAT & beyond
Please do not PM me, (not active anymore) contact Eric.
-
smartgmat
- Newbie | Next Rank: 10 Posts
- Posts: 1
- Joined: Sat Mar 21, 2009 3:11 am
- Location: bangalore
i think managing director is right,,,
gmat gives experimental questions in random,,i mean,,depending on ur ability in test.if u keeping on doing wrong questions then it wont give u any experimental questions....i tried to figure out gmat prep but resulted in scraching my head,
: : :
gmat gives experimental questions in random,,i mean,,depending on ur ability in test.if u keeping on doing wrong questions then it wont give u any experimental questions....i tried to figure out gmat prep but resulted in scraching my head,
: : : hi,,i want to meet people who want are intersted in gmat n want to pursue mba in usa and canada.
thank you for sharing! I have also tried to figure out, how this damn thing works......
I prepared with princeton review and assumed that early questions are worth more. My observation from playing around with GMATPrep was that there are some questions that reappear at some point regardless of your performance....so I think the e-questions are thrown in randomly. Still I am not sure whether you get penalized for unanswered e-questions in the end, i.e. you miss the last question and that happens to be an e-question.
While playing around with GMATPrep I observed that score variance for the same number of incorrect questions depends more on the number of incorrect questions in a row and not so much on where the incorrect-chain appears in the test.
Still, if the first 5 are correct and the remaining are altering correct/incorrect, you get a higher score than in the opposite case (first ones correct/incorrect and last 5 correct. But this observation could be biased by e-questions appearing randomly.
However, in retrospective I should have focused on studying instead of trying to figure out how the algorithm works. The knowledge about how it works won't get you higher scores anyways, because in the actual GMAT you get scores for answering questions correctly and not for predicting how an incorrect/guessed answer could affect your score.
I prepared with princeton review and assumed that early questions are worth more. My observation from playing around with GMATPrep was that there are some questions that reappear at some point regardless of your performance....so I think the e-questions are thrown in randomly. Still I am not sure whether you get penalized for unanswered e-questions in the end, i.e. you miss the last question and that happens to be an e-question.
While playing around with GMATPrep I observed that score variance for the same number of incorrect questions depends more on the number of incorrect questions in a row and not so much on where the incorrect-chain appears in the test.
Still, if the first 5 are correct and the remaining are altering correct/incorrect, you get a higher score than in the opposite case (first ones correct/incorrect and last 5 correct. But this observation could be biased by e-questions appearing randomly.
However, in retrospective I should have focused on studying instead of trying to figure out how the algorithm works. The knowledge about how it works won't get you higher scores anyways, because in the actual GMAT you get scores for answering questions correctly and not for predicting how an incorrect/guessed answer could affect your score.
