• 7 CATs FREE!
    If you earn 100 Forum Points

    Engage in the Beat The GMAT forums to earn
    100 points for $49 worth of Veritas practice GMATs FREE

    Veritas Prep
    VERITAS PRACTICE GMAT EXAMS
    Earn 10 Points Per Post
    Earn 10 Points Per Thanks
    Earn 10 Points Per Upvote
    REDEEM NOW

The GMAT Algorithm and initial estimate

This topic has expert replies
Master | Next Rank: 500 Posts
Posts: 223
Joined: 17 Apr 2009
Location: Philly, USA
Thanked: 76 times
Followed by:4 members
GMAT Score:750

The GMAT Algorithm and initial estimate

by zuleron » Thu May 14, 2009 2:35 pm
I have read a few posts about how you should spend more time on the first 10 questions and if you don’t you run the risk of getting “locked-in” to a low score. I think the consensus among instructors is that there is no such thing as being locked in. However, the question remains: how is it that many well prepared people have reported being frustrated (while they were doing the exam) by the easiness of the questions they were seeing and by their inability to consistently see the tougher questions, and though they did not flunk the GMAT, they are inevitably disappointed with their performance?

Could it be, then, that the initial estimate is not “locked-in”, but rather it acts like a center of gravity to which you are drawn for the remaining 27 questions? So if your initial estimate is 600 then clearly it is difficult to get to 750 but should you get to level 750 questions, it is very difficult to STAY at 750 because the algorithm wants to draw you down to 600. And if your initial estimate is 750 then it is easier to get back to 750 should you drop to 600.

For example, if I get 6 out of the first 10 wrong and so at Q11 the computer estimates me as a 550. Then I get the next 5 correct and it raises me to 650 at Q16. But then I get the Q16 and Q17 wrong, is it not reasonable that the computer would ‘think’ that I really am a 550 guy and my hot streak from Q 11 – 16 might have been a little lucky so we’ll discount a few points and so for Q 18 we’ll give him a 570 level instead of a 600 level? So if your initial estimate is low you cannot afford to make mistakes coz they are costlier.

Conversely, if I get the first 10 correct, then presumably I’m scoring at the 800 level. But then I get the next 5 wrong which drops me to, say, the 670 level. But then I get the next 2 correct, so at Q 18 is it not reasonable that the algorithm will think he really is an 800 guy and his flubs between Q11 and Q16 were an aberration and so for Q18 we’ll give him a 770 level question instead of a 730 level question. So if your initial estimate is high, you can afford to make some mistakes coz they are less costly.

In other words, throughout the exam, algorithm wants to pull you to your initial estimate. So should you stray far from your initial estimate it will only take a couple of correct or incorrect answers in a row to bring you back to your initial estimate. So if your initial estimate is low, it takes only two or so incorrect responses in a row to bring you back down even if you have managed to raise your level very high. And that if your initial estimate is very high and you go through a rough patch, then a few correct in a row will shoot you right back to your high initial estimate. So the GMAT algorithm is really a species of weighted average with a bias towards the initial estimate.

Is this not a reasonable reading of how the algorithm works? And if this is true, then it should also be true that establishing a high initial estimate is an important test-taking strategy, thus you should spend more time on the 1st 10 questions?

Thoughts?

Senior | Next Rank: 100 Posts
Posts: 84
Joined: 04 Nov 2008
Location: Boston, MA
Thanked: 50 times
Followed by:9 members
GMAT Score:760

by canada_sms » Thu May 14, 2009 4:31 pm
Interesting theory...

In my opinion the GMAT algorithm more about momentum than anything else.

In my experience, your score accelerates upwards when you get multiple questions right in a row. Conversely, getting multiple questions wrong in a row causes you to drop steeply. Doing well is all about putting together strings of correct answers with incorrect errors sprinkled throughout a section but not clumped in pockets of failure.

As a result, spending more time on the front end of a section is doing a disservice to yourself because you're reducing your time to answer questions on the back end and increasing the likelihood of having pockets of multiple incorrect answers due to rushing and guessing.

I'm not saying I'm right about this. No one knows what the algorithm is except GMAC and until someone does a brute force disassembly of the GMAT Prep decision tree we're never going to know.
MBA Candidate 2012
MIT Sloan School of Management

User avatar
Site Admin
Posts: 2567
Joined: 01 Jan 2009
Thanked: 712 times
Followed by:550 members
GMAT Score:770

Re: The GMAT Algorithm and initial estimate

by DanaJ » Thu May 14, 2009 9:18 pm
zuleron wrote:However, the question remains: how is it that many well prepared people have reported being frustrated (while they were doing the exam) by the easiness of the questions they were seeing and by their inability to consistently see the tougher questions, and though they did not flunk the GMAT, they are inevitably disappointed with their performance?

One thing I noticed when I took the real thing was that questions that do not seem difficult are actually pretty tricky. There were at least 5 instances in the test when I changed my answer right before hitting "Next", since I'd missed some insignificant case that flipped the whole thing around.

This is one more reason why I believe that the makers of the GMAT know what they're doing: some of the higher level questions are designed in such a way that they seem easy, but it's very tempting to overlook an essential detail.

So when a well-prepared test-taker is disappointed with his/her score, this may be because he/she mistook some hard questions for easy ones.

That is not to say that your theory isn't plausible. However, as pointed out before, you shouldn't spend that much time on the first few questions, since you risk not finishing the thing and getting a nasty penalty for blanks.

In the end, this is all speculation, however. You can't really say it's this or that, since only by tearing down GMATprep can you be sure how the algorithm works. I personally believe that big prep companies have already tried this: there are a lot of computer engineers out there who have surely seen worse, and I bet it isn't that expensive to hire a team of 10 or something...

Master | Next Rank: 500 Posts
Posts: 223
Joined: 17 Apr 2009
Location: Philly, USA
Thanked: 76 times
Followed by:4 members
GMAT Score:750

by zuleron » Fri May 15, 2009 5:23 am
canada_sms wrote:Interesting theory...

In my opinion the GMAT algorithm more about momentum than anything else.


In my experience, your score accelerates upwards when you get multiple questions right in a row. Conversely, getting multiple questions wrong in a row causes you to drop steeply. Doing well is all about putting together strings of correct answers with incorrect errors sprinkled throughout a section but not clumped in pockets of failure.


As a result, spending more time on the front end of a section is doing a disservice to yourself because you're reducing your time to answer questions on the back end and increasing the likelihood of having pockets of multiple incorrect answers due to rushing and guessing.

I'm not saying I'm right about this. No one knows what the algorithm is except GMAC and until someone does a brute force disassembly of the GMAT Prep decision tree we're never going to know.
I completely agree that it's about momentum but I was really asking what role, if any, the initial estimate plays. I mean GMAC themselves bring it up, so it must play some role...

I also agree that the ideal sequence of correct, wrong, correct etc. is sth like 4 correct 1 wrong, 4, 1, 5, 2, 3, 1, 4, 1, 4, 1 etc.

but it still doesn't answer the role of the initial estimate... I realize that this is perhaps an academic question but because it is one of those widespeard beliefs about the GMAT it is probably worth considering.

Master | Next Rank: 500 Posts
Posts: 223
Joined: 17 Apr 2009
Location: Philly, USA
Thanked: 76 times
Followed by:4 members
GMAT Score:750

Re: The GMAT Algorithm and initial estimate

by zuleron » Fri May 15, 2009 5:54 am
DanaJ wrote:
zuleron wrote:However, the question remains: how is it that many well prepared people have reported being frustrated (while they were doing the exam) by the easiness of the questions they were seeing and by their inability to consistently see the tougher questions, and though they did not flunk the GMAT, they are inevitably disappointed with their performance?

One thing I noticed when I took the real thing was that questions that do not seem difficult are actually pretty tricky. There were at least 5 instances in the test when I changed my answer right before hitting "Next", since I'd missed some insignificant case that flipped the whole thing around.

This is one more reason why I believe that the makers of the GMAT know what they're doing: some of the higher level questions are designed in such a way that they seem easy, but it's very tempting to overlook an essential detail.

So when a well-prepared test-taker is disappointed with his/her score, this may be because he/she mistook some hard questions for easy ones.

That is not to say that your theory isn't plausible. However, as pointed out before, you shouldn't spend that much time on the first few questions, since you risk not finishing the thing and getting a nasty penalty for blanks.

In the end, this is all speculation, however. You can't really say it's this or that, since only by tearing down GMATprep can you be sure how the algorithm works. I personally believe that big prep companies have already tried this: there are a lot of computer engineers out there who have surely seen worse, and I bet it isn't that expensive to hire a team of 10 or something...
I'm with you. I am very wary of the trap questions esp. DS that try to push you to choose C when either A or B alone is suffecient, or to pick C when E is the best answer. But I was basing my theory on my experience tinkering with GMAT Prep I. Admittedly, my experiments were not and could not be exhaustive but the initial estimate as a center of gravity was my intuition based on the limited tinkering I did. Getting the first 10 wrong requires that you almost have no back to back errors thereafter coz the level of difficulty plummets. And getting the 1st 10 correct allows you to have more back to back errors. In fact on one actual test, I scored 8/10 in the first 10 questions and 7 out of 10 in the 2nd 10 questions. and then I got 28 to 33 all wrong and still managed to get a 45 on the quant and a 710 over all.

i guess what I'm saying is that a high initial estimate gives you significant room for consecutive errors whearas a low initial estimate gives you almost no room for consecutive errors. So to this extent, the initial estimate is important...

GMAT/MBA Expert

User avatar
GMAT Instructor
Posts: 2583
Joined: 02 Jun 2008
Location: Toronto
Thanked: 1090 times
Followed by:355 members
GMAT Score:780

by Ian Stewart » Fri May 15, 2009 6:00 am
There's no such thing as an 'initial estimate' in the GMAT scoring algorithm; that's a phrase invented by some prep companies. The only 'weighting' in the calculation of your score is based on the difficulty level of a question; it doesn't matter where you saw it on the test (well, that's a bit of a simplification - questions have two other statistics besides difficulty level, but I'll ignore that for simplicity). If you have a 500-level question at the start of your test, and a 500-level question at the end of your test, getting that first question wrong and the second right will give you the same score as getting the first right and the second wrong, all else being equal.

There are details of the scoring algorithm that are at the discretion of the test developer to determine, but the mathematical basis of CAT test scoring is not a secret. It is, however, very complicated; so complicated, in fact, that you can't tell just from glancing at the formulas how it works. What canada_sms says above describes well how scores are adjusted - strings of bad answers can hurt you a lot, while strings of good answers can help you a lot. And early questions are slightly more important than later ones, if only because the test is finite in length; if you do badly on enough early questions, you'll run out of opportunities to recover. But if you invest a lot of time early in the test to answer questions correctly, you'll be risking a bad string of answers at the end, which can cause your score to plummet.
If you are looking for online GMAT math tutoring, or if you are interested in buying my advanced Quant books and problem sets, please contact me at ianstewartgmat at gmail.com

Master | Next Rank: 500 Posts
Posts: 223
Joined: 17 Apr 2009
Location: Philly, USA
Thanked: 76 times
Followed by:4 members
GMAT Score:750

by zuleron » Fri May 15, 2009 6:08 am
Thanks!

Master | Next Rank: 500 Posts
Posts: 223
Joined: 17 Apr 2009
Location: Philly, USA
Thanked: 76 times
Followed by:4 members
GMAT Score:750

by zuleron » Fri May 15, 2009 7:04 am
Ian Stewart wrote: And early questions are slightly more important than later ones, if only because the test is finite in length; if you do badly on enough early questions, you'll run out of opportunities to recover. But if you invest a lot of time early in the test to answer questions correctly, you'll be risking a bad string of answers at the end, which can cause your score to plummet.
Let me press you a little on this point. If doing badly enough on early questions suggests that you'll run out of real estate to recover, then shouldn't the reverse be true: that doing well enough early cuts you some slack towards the end of the test?

I suppose the question is really pick your poison. If getting Q5, 6, and 7 wrong is as costly as getting Q35, 36, and 37 wrong which would you prefer to get wrong? Personally, I'd prefer to get Q35, 36, and 37 wrong. As such, I'd be willing to spend upto 3 mins on the first 10...

GMAT/MBA Expert

User avatar
GMAT Instructor
Posts: 2228
Joined: 27 Dec 2006
Location: Montreal, Canada
Thanked: 639 times
Followed by:693 members
GMAT Score:780

by Stacey Koprince » Fri May 15, 2009 10:03 am
Received a PM asking me to respond. First, what Ian and canada_sms said - right on. It is unbelievably difficult to estimate a problem's difficulty level even when you're an expert and have unlimited, unstressed time to examine the problem. Trying to estimate during the heat of the test? No way - even experts have major trouble doing that.

Second, to the question in the last post. Sure, doing well enough early on cuts you some slack towards the end, as long as you didn't need extra time to do well early on. That's a pretty big caveat. If you have to guess on 4 (or more) questions in a row at the end as a result of spending extra time to try to get the first 7 in a row right, then statistically your score will be lower than it would have been if you'd just moved steadily through the test.

(Yes, a study has actually been done on just this scenario - a bunch of scenarios were set up and run on a computer. The study showed that the penalty for a string of wrong answers was higher than the benefit for a same-length string of right answers, and also that a string of wrong answers more negatively affected a higher scorer than a lower scorer. Finally, the study showed that for a 500-level test taker, if they can get the first 3 in a row right but have to guess on 3 at the end, the benefit is completely offset by the penalty. If they can get the first 6 in a row right, they can guess on as many as 10 at the end before offsetting the benefit - but read that more carefully. What do you think the chances are of a 500-level test taker getting the first 6 in a row right? At the 780 level, we have to get the first 5 in a row right in order to offset the penalty of guessing on a single question at the end! In other words, we can't take extra time to get these questions right.)**

In a nutshell: in a vacuum, sure, the earlier questions can have more of an impact on your score than the later ones, but this assumes that there are NO consequences to spending extra time on the earlier ones. There are, in fact, heavy consequences for this - the penalty for strings of wrong answers or for running out of time completely.

And let's not even talk about the extreme unlikelihood of getting the first 7 (or 10) in a row right on the real test. Unless you can score 770+, it's not going to happen. (And if you can score 770+, then you're not exactly worried about "gaming" the test, are you? :))

It's also extremely unlikely, by the way, to get the first 7 (or 10) in a row wrong. So you can run tests with GMATPrep in which you try to do these things, but what you are doing is creating an artificial, completely extreme scenario that wouldn't actually happen in the real world.

I know it's tempting to try to pick apart this algorithm, but your time is better spent studying the test problems. We (the experts, the test prep companies, etc.) have already done it. The best strategy is to move steadily through the test, giving your best shot within an appropriate time frame on every question.

** All data from "Test-Taking Strategies in Computerized Adaptive Testing" by M. Steffen and W. Way, Educational Testing Service, April 1999.
Please note: I do not use the Private Messaging system! I will not see any PMs that you send to me!!

Stacey Koprince
GMAT Instructor
Director of Online Community
Manhattan GMAT

Contributor to Beat The GMAT!

Learn more about me

Master | Next Rank: 500 Posts
Posts: 223
Joined: 17 Apr 2009
Location: Philly, USA
Thanked: 76 times
Followed by:4 members
GMAT Score:750

by zuleron » Sat May 16, 2009 10:20 am
Stacey Koprince wrote:(Yes, a study has actually been done on just this scenario - a bunch of scenarios were set up and run on a computer. The study showed that the penalty for a string of wrong answers was higher than the benefit for a same-length string of right answers, and also that a string of wrong answers more negatively affected a higher scorer than a lower scorer. Finally, the study showed that for a 500-level test taker, if they can get the first 3 in a row right but have to guess on 3 at the end, the benefit is completely offset by the penalty. If they can get the first 6 in a row right, they can guess on as many as 10 at the end before offsetting the benefit - but read that more carefully. What do you think the chances are of a 500-level test taker getting the first 6 in a row right? At the 780 level, we have to get the first 5 in a row right in order to offset the penalty of guessing on a single question at the end! In other words, we can't take extra time to get these questions right.)**
Wow! This answers all my questions. If the penalty for 4 in a row wrong is higher than the benefit for 4 in a row correct then you absolutely CANNOT spend more time on the first ten. This should be advertised more! And it probably makes sense that the higher your score is the more devastating a string of incorrect answers would be. Sort of like the credit score where one late payment will drop a a score of 750 to 700 whereas one late payment will drop a score of 620 to maybe 615.

Thanks Stacey for the insights and info! Methinks the matter has been put to rest!

User avatar
Legendary Member
Posts: 519
Joined: 27 Jan 2007
Location: India
Thanked: 31 times

by Neo2000 » Sat May 16, 2009 11:08 am
I think GMAC patented their algorithm right? So somebody with a strong CS background could take a look at the algorithm?? I realise that GMAC has tweaked their algorithm tremendously over the years accounting for all kinds of variables such as time spent, correct/wrong streaks, position, diff level but i think it would still make for interesting reading.

Any volunteers?? :lol:

User avatar
Site Admin
Posts: 2567
Joined: 01 Jan 2009
Thanked: 712 times
Followed by:550 members
GMAT Score:770

by DanaJ » Sat May 16, 2009 11:29 am
Neo2000 wrote:I think GMAC patented their algorithm right? So somebody with a strong CS background could take a look at the algorithm??
Any volunteers?? :lol:
I saw CS over there and immediately thought of Counter Strike :D

User avatar
Legendary Member
Posts: 519
Joined: 27 Jan 2007
Location: India
Thanked: 31 times

by Neo2000 » Sat May 16, 2009 11:32 am
DanaJ wrote:I saw CS over there and immediately thought of Counter Strike :D
Aahh Youth :roll:

p.s. :lol:

Master | Next Rank: 500 Posts
Posts: 124
Joined: 30 Apr 2008
Location: Los Angeles
Thanked: 5 times

by skang357 » Sat May 16, 2009 1:51 pm
I'm taking the test this Saturday so I'm sort of crunched for time and didn't really read your post or the rest of the thread in detail.

But I get the gist of what you're trying to figure out. It seems to be a common topic that test takers wrangle over.

I try to stay away from such theorizing and how scores are calibrated, and how the algorithm of the adaptive test is calculated, etc.

To me, I like to keep it simple. Basically, there is a cap to how hard these questions can get. I have not seen a question that I find it unreasonable to answer in at most 5 minutes.

In that sense, the GMAT is all the same to me in the Quantitative.

Here is basically what reading all of the posts, trolling through this site, the hours,days of research boils down to...

I want to get a Q score of 48 or higher.

If I spend more than 2 minutes for any question on the Q, I'm screwed. If I spend more than 3 minutes and I can't find an answer, I'm just gonna cover my eyes, say "eeny miny mo" and pick an answer.

It doesn't matter what the question is.

Lastly, If I don't get 30 questions right on the Q, then my score is going to be less than high 40's.

That's the bottom line. At the very worst I have to miss no less than 10 questions out of the 37 to not completely bomb it.

I'm not gonna discriminate between what kind of question it is, what level, etc.

That's why there are resources like GMATCLUB.COM, BTG, and MGMAT.

So that we may be able to handle most if not all of everything that is thrown at us in the test.

I just try to get less than 10 wrong on the Q and that's all I care about. Doesn't matter the question.

I'd rather skip a few hard questions in the middle and then try to answer all of them at the end with ample time than try to get the first 10 right or the first 5.

I don't discriminate on the problems, I just try to solve as much of them as I can and as fast as I can. If you can do that, I think that's all that matters.
Impossible is nothing

Master | Next Rank: 500 Posts
Posts: 124
Joined: 30 Apr 2008
Location: Los Angeles
Thanked: 5 times

by skang357 » Sat May 16, 2009 2:04 pm
If you take enough of these timed practice tests, you naturally figure out if you spent too long on a question.

Just remember if it feels like you spent too much on one question, that means you did. Then guess and move on.

I find it impossible that the GMAT can throw you more than a handful or more questions where you were totally stumped. If you practice enough, you should be able to answer least high 20's number of questions in 2 minutes without guessing. Hopefully you will get those alright, and of the ones you guessed, there are blind guesses, and there are educated guesses.

With educated guesses you have a 50/50 shot more or less, and with blind guesses you have a 1 in 5 chance. Always pick round numbers.

With those odds, odds are you will get another 3 or 4 questions right just by guessing. Which adds another 20 or 30 points to your score. Sweet!
Impossible is nothing