• Take advantage of our featured Black Friday/Cyber Monday deals

    LEARN MORE
  • Take advantage of our featured Black Friday/Cyber Monday deals

    LEARN MORE

R&D on GMAT prep

This topic has expert replies
Legendary Member
Posts: 2326
Joined: 28 Jul 2008
Thanked: 173 times
Followed by:2 members
GMAT Score:710

by gmatmachoman » Sun Dec 05, 2010 2:14 am
@gmat1011;

Can u tell me what questions are that 2 wrong belong to?

I just now noticed Anand's snapshot he made 3 mistake like : @ 17 : RC ; 30 : CR ; 41 : SC and scored 45.

So once u tell the wrong ones, I can understand better.....

Actually this part of analysis is getting trickier.

Is that just one point difference can make a substantial difference in scaled scores??

@Rishab :
U said u got 9 incorrect and got 40...

In one of my case studies, i got 14 incorrect and got 28..That's a whopping 12 point difference in scaled score.

So understanding the pattern/algorithm is not EASY task....

Master | Next Rank: 500 Posts
Posts: 160
Joined: 05 Apr 2010
Thanked: 7 times

by gmat1011 » Sun Dec 05, 2010 6:13 am
both the incorrect ones - Q. 9 and Q. 39 were CR questions...

one thing I'd really like to know is how the experimental questions come into the equation on the real GMAT. 25% of the questions overall across the 2 sections are "experimental" and not graded... what if I get all the exp. ones right and get the ones which count wrong... wonder what will happen then...

Junior | Next Rank: 30 Posts
Posts: 13
Joined: 29 Jun 2010

by kola_member » Sun Dec 05, 2010 7:10 am
I am sure your did not get two SCs wrong. If you did, you will not end up with 48,
I guess the guy above had some SCs wrong there.

Junior | Next Rank: 30 Posts
Posts: 13
Joined: 29 Jun 2010

by kola_member » Sun Dec 05, 2010 7:14 am
kola_member wrote:I am sure your did not get two SCs wrong. If you did, you will not end up with 48,
I guess the guy above had some SCs wrong there.
There you go ! Getting Scs wrong pushes your score down more than getting RC or CR wrongs, generally speaking.

User avatar
Senior | Next Rank: 100 Posts
Posts: 72
Joined: 13 May 2009
Location: New York
Thanked: 7 times
Followed by:2 members

by sashish007 » Mon Dec 13, 2010 12:18 pm
rishab1988 wrote:Hi guys

Now I don't remember exactly the data of 8 GMATPrep CATs that I taken,but I do remember that I never got more than 2-3 SC incorrect 1-2 CR incorrect.

For my split used to :

2 SC 1 CR 3 RC -44
1 SC 1 CR 2 RC -46
2 SC 1 CR 4 RC - 41

3 SC 3 CR 6 RC -39 [shocking isn't it].
3 SC 3 CR 7 RC -38 !

After reading some people's posts that they got bombed after crossing 10 incorrect question mark,I asked those guys how many were RC and invariably most answered that the no of incorrect questions were RC.

My logic goes this way:

Since SC is the most learn-able part in verbal ,it is highly likely that the SC question that you got wrong might have been answered correctly by others.

Next comes CR.You can build accuracy in CR after a lot of practice.So getting a CR question wrong penalizes less than a SC.

Next comes RC.I believe that this is the least learn-able part in the verbal.A persons' RC ability 90% of the times stays the same no matter how many questions he practices.That's why getting a RC incorrect penalizes you the least.Remember 50% of the test takers[not givers as most international students say.lol] are non-native people! Most of those people can improve their SC ability by using prep guides such as MGMAT SC strategy guide etc.,but most of the people find it difficult to improve their RC skills.You can't improve them overnight!

Btw did I mention that I got a 40 on verbal with even 9 questions incorrect!



the above analysis is very close to the actual answer. for instance, many people tend to score better on quant than they do on verbal. if you had, say, 9-12 incorrect in quant, your score hovers in 45-49 region. this is because you attempted harder questions and then got them wrong. but on the verbal, you need to have very few incorrect (single digit) to get into the 35+ zone.

generally speaking, whether it is quant or verbal, if you make mistakes early on or on really stupid stuff (sc or ps), you get penalized a lot more than you would if you answered a harder question (rc or ds) wrong - quite logical!

my 2 cents.

-Ashish
Ashish
Share not just why the right answer is right, but also why the wrong ones are not.

Junior | Next Rank: 30 Posts
Posts: 12
Joined: 15 Mar 2010
Followed by:1 members

by msingh » Mon Jan 17, 2011 6:39 am
I did not read each and ever post on this thread but i can share my experience, specially in Verbal.

My experence in one line - If you get 3-4 consecutive wrong answers, you are doomed. Here is break-up of my max score in Verbal. In this test i did not get more than 2 consecutive wrong answers.

Score in verbal = 34

Break up -

#Questions answered = 38 out of 41
# Correct Answers = 29
# Wrong Answers = 9

Here is breakup of my correct answers by taking sample set of 10 questions.

SC Total SC CR Total CR RC Total RC Qs# Range
4 5 2 2 3 3 1-10
3 4 3 3 3 3 11-20
1 3 1 3 2 4 21-30
4 4 1 1 2 3 31-41

User avatar
Senior | Next Rank: 100 Posts
Posts: 80
Joined: 09 Jan 2011
Thanked: 4 times
Followed by:5 members

by powerpuff » Mon Jan 17, 2011 6:57 am
Amazing insights here on this thread. Thank you guys for the great stuff!

User avatar
Legendary Member
Posts: 1079
Joined: 13 Dec 2010
Thanked: 118 times
Followed by:33 members
GMAT Score:710

by bblast » Thu Feb 03, 2011 4:55 am
Gmat prep-1 verbal 40 - 9 incorrect,

u just gotta nail the 1st rc folks,and try to make the least mistakes in RC.
rest assured a 35+ if u keep making mistakes randomly and not back to back,

more than 10 mistakes in verbal(v<35) hasta la vista babay !!



Image
Cheers !!

Quant 47-Striving for 50
Verbal 34-Striving for 40

My gmat journey :
https://www.beatthegmat.com/710-bblast-s ... 90735.html
My take on the GMAT RC :
https://www.beatthegmat.com/ways-to-bbla ... 90808.html
How to prepare before your MBA:
https://www.youtube.com/watch?v=upz46D7 ... TWBZF14TKW_

Legendary Member
Posts: 2326
Joined: 28 Jul 2008
Thanked: 173 times
Followed by:2 members
GMAT Score:710

by gmatmachoman » Sat Feb 05, 2011 11:20 pm
Going by the responses from my fellow co aspirants who wanted to beat/molest/crash/ridicule the monster called GMAT,
I certainly feel this thread has done Justice and has shared quite insightful experiences.

The following inferences are purely based on the GMAT prep & fellow aspirants experiences & their shared data.

Inferences on Verbal:

To get more than 40 (scaled score)

1. Dont make more than 8 mistakes.
2.Avoid consecutive mistakes
3. Mistakes at the beginning is penalised heavily than the mistakes at the end of the section.
4. As the rule says, all sections SC,CR,RC are important. But when it comes to scoring,as per our data collected RC weighs more.

To get more than 44+
1. Make no more than 4 mistakes.

One aspirant was asking me about the influence of experimental section on the scores, the answer is we need to "assume" some variables in the study. Since this study is purely a empirical one, it is given to some assumptions.

Bottom line:
www.mba.com /GMAC has said in its website that they follow the same algorithm for the real test, so on that note we started this thread and deduced the results.

Specials thanks to all the participants & Rajat & e-gmat team for their seed of this thought process!!

Thanks for the cooperation. With out your contributions,this much data could have not been generated. Thx

Rgds
Govi.

User avatar
Master | Next Rank: 500 Posts
Posts: 163
Joined: 24 Nov 2009
Location: Philadelphia
Thanked: 13 times
Followed by:4 members
GMAT Score:660

by chendawg » Tue Mar 22, 2011 7:29 am
I'm not sure if this post actually helps or hurts me from a psychological point of view on test day. I'm gonna pretend I didn't read this post lol

User avatar
Senior | Next Rank: 100 Posts
Posts: 72
Joined: 13 May 2009
Location: New York
Thanked: 7 times
Followed by:2 members

by sashish007 » Thu Mar 24, 2011 12:36 pm
i thought of sharing this verbal ONLY report: V31
what's special about this is that despite the fact that i was able to get 13 questions right in a row later (24-36), i still scored low.
i acknowledge the fact that there are few mistakes initially, and 4 places with consecutive wrong answers.
Attachments
vprep.jpg
v31
Ashish
Share not just why the right answer is right, but also why the wrong ones are not.

Master | Next Rank: 500 Posts
Posts: 184
Joined: 14 Apr 2007
Location: Madison, WI
Thanked: 17 times

by ldoolitt » Thu Mar 24, 2011 3:21 pm
This is a neat thread. Am following.

Random question:

Has anyone thought of the more direct approach of just decompiling the gmatprep software and reading the algorithm from code? I mean as a software engineer thats where my first thought goes. Its purely a client application and I'm guessing its not encrypted so with enough time and willpower...

Junior | Next Rank: 30 Posts
Posts: 19
Joined: 12 Feb 2011
Thanked: 1 times
GMAT Score:680

by beatitin11 » Sun Apr 10, 2011 7:13 am
One of my observations in Section 3.4 Interpretive Guide of the Diagnostic Test of OG12:

9 out of 17 correct answers (8 mistakes) in CR is considered above avg and 11 out of 18 correct answers (7 mistakes) in SC is above average, but only 14 out 17 correct answers (3 mistakes) in RC is above average. This too indicates to me that RC questions have higher weightage over SC and CR questions in verbal.

Does this make sense to you guys?

GMAT/MBA Expert

User avatar
GMAT Instructor
Posts: 2583
Joined: 02 Jun 2008
Location: Toronto
Thanked: 1090 times
Followed by:355 members
GMAT Score:780

by Ian Stewart » Mon May 16, 2011 3:22 pm
I just saw this thread for the first time. I hate to be negative about the studies performed above, but there's no chance you'll be able to reverse-engineer the GMAT scoring algorithm from response patterns in GMATPrep. A few things about the scoring algorithm:

* despite what you might read in books, the algorithm is not a secret. It's based on a field of test theory called Item Response Theory that you can learn about on wikipedia;

* that said, the mathematics behind the algorithm is horrendously complicated. Unless you have an undergraduate degree in probability theory or statistics, you shouldn't even look at it;

* there are two components to the algorithm that work hand-in-hand: the scoring algorithm, and the question selection algorithm. All of the research above is inherently ignoring the importance of the question selection algorithm, and without understanding question selection, you can't understand how GMAT scoring works.

Oversimplifying the math, the algorithm basically does this: at the end of your test, the algorithm looks back over all the questions you answered and tries to determine what difficulty level of question you can handle comfortably, and what level is too hard for you. So if the test looks back over your answers and sees that you get most of your sub-600 level questions right, but that you don't get very many questions right that are above the 600-level, the algorithm will conclude that you're most likely a 600-level test taker. When the algorithm does this, it has no idea when you saw each question. If you get a 500-level question wrong early in your test, that hurts you just as much as getting a 500-level question wrong at the end of your test. The algorithm is completely blind to question 'position'.

One consequence of this is that it hurts you considerably more to get a 250-level question wrong than to get an 800-level question wrong. If you answer an 800-level question wrong, that is only evidence that you might not be an 800-level test taker, but you still might be a 750-level test taker. On the other hand, getting that 250-level question wrong is evidence that you can't handle even the lowest level questions, and you'll need to do some work to persuade the algorithm that your wrong answer there was an anomaly. It's for this reason that your number of correct answers is only weakly correlated with your score. If you were to take a test consisting of only 200-level questions, you'd need to get almost all of them right to get a respectable score. If instead your test consisted only of 800-level questions, you could get a ton of them wrong and still get a good score. For example, the Diagnostic Test at the beginning of OG12, mentioned in the post above, is full of hard questions, so you can get a lot of them wrong and still get a good score according to their score table.

All of that said, your answers to early questions play a role in determining what questions you see later in your test. As you proceed through the GMAT, the test is constantly updating its estimate of your ability, and it attempts to give you questions around your estimated ability level. This is why it is better to get the first 27 questions right, then the last 10 wrong, than to get the first 10 wrong and the last 27 right. If you get the first 10 questions wrong on your test, that's worse than a chimpanzee would normally do just entering random answers - after all, you'd expect, guessing randomly, to get 20% of your questions right. So if you get your first 10 questions wrong, the algorithm will think that you're completely hopeless. From that point forward, it will be giving you very easy questions, and you'll need to get a lot of those right just to persuade the algorithm that you can handle 200 or 300-level questions. If, on the other hand, you get your first 10 questions right, the algorithm will be persuaded that you're a very high-level test taker. You'll then be getting mostly 750 to 800-level questions, and even if you get a lot of those wrong, the algorithm can still think of you as a 700-level test taker, at least up to a point - if you get too many of these wrong, the test will keep lowering its estimate of your ability.

So in one sense, early questions are more important than later ones, since answering them correctly gives you the opportunity to see harder questions later on. But if you're a genuine 600-level test taker, then even if you answer your first 10 questions correctly (something, I should add, that a 600-level test taker will almost never be able to do) and start seeing a lot of 700-level questions, eventually you'll start getting most of your questions wrong (if you didn't, you wouldn't be a 600-level test taker, you'd be a 700-level test taker). You won't be able to 'outsmart' the algorithm by some exceptional response pattern early in the test.

There are several further complicating factors:

* when the test selects your next question, its estimate of your ability is only one factor that determines which question you see. The test also has content restrictions to ensure you see the same number of, say, word problems as any other test taker. There is also a security requirement; the test does not want any question to be given to too many test takers, just in case some questions are (illegally) publicized - then knowing about that question in advance would give a test taker an unfair advantage if he or she was highly likely to see that question on the test. Because the question pool is not infinite, that means there is usually only a relatively small number of questions the test has available at each point. So even if you're a 700-level test taker, you'll see some questions which count which are either well below or well above your level late in your test; the test simply doesn't have an infinite supply of questions precisely at your level that it can give you. This is why all of the research above can't possibly reach any exact conclusions: on any test you take there will be a different question pool, and you won't be able to predict the difficulty level of questions at each point in the test, even if you answer using the same response pattern. It is also why it can hurt you to guess at the last few questions; some of those questions might be well below your level, and it will hurt you to get those wrong. Long story short: a wrong answer early in your test might not hurt you much at all because the question might be way above your level, and a wrong answer late in the test can hurt you a lot because the question might be way below your level. Unless you have a ton of information about the underlying question database, you can't determine how often this will happen in a given test, which is why it's impossible to look at GMATPrep results and reach any reliable conclusions about what will happen on a real GMAT.

* further complicating things is the presence of experimental questions. Some of your questions simply don't count, and you won't have any way to know which those questions are (where they occur on your test is determined completely at random). If GMATPrep models the presence of experimental questions correctly, then it will be the case that some wrong answers have no influence at all on your score.

* the observation above that wrong answers in RC hurt more than in SC or CR is likely due to the fact that RC is the least 'adaptive' question type. RC passages are always accompanied by a group of questions, and while on average these questions might be hard or easy, each question individually will have its own difficulty level - even in a batch of RC questions which is hard overall, one or two of the questions might be easy. So if you answer several RC questions incorrectly, you're more likely to be answering easy questions incorrectly, which hurts you more than answering difficult questions incorrectly.
If you are looking for online GMAT math tutoring, or if you are interested in buying my advanced Quant books and problem sets, please contact me at ianstewartgmat at gmail.com

User avatar
GMAT Instructor
Posts: 2193
Joined: 22 Feb 2010
Location: Vermont and Boston, MA
Thanked: 1186 times
Followed by:512 members
GMAT Score:770

by [email protected] » Mon May 16, 2011 5:34 pm
That is a well-researched very thorough posting that indicates that time would be better spent learning the concepts and practicing the questions rather than trying to reverse engineer the system.

Great stuff!!
Veritas Prep | GMAT Instructor

Veritas Prep Reviews
Save $100 off any live Veritas Prep GMAT Course