Knewton Scoring -- Really skewed

This topic has expert replies
Senior | Next Rank: 100 Posts
Posts: 71
Joined: Wed Mar 17, 2010 1:31 pm
Thanked: 5 times
Followed by:1 members

Knewton Scoring -- Really skewed

by newton9 » Mon Apr 12, 2010 7:37 pm
Took a knewton diagnostic test today.

Verbal : 32 / 63%
Quant : 51 / 98%
Total : 700 / 93%

Got all the questions correct in quant, so no complaints.

But I just got 3 questions wrong in Verbal and my score has been pegged back to 32(63%) . Does that make sense? And the question numbers are 16(RC),19(SC) ,20(CR). I know it my reduce by score by answering 3 of them wrong, but does 32/63% makes sense to you guys?

I guess they might be skewing their scores to account for 50 point money back guarantee.

I will be happy if anyone from Knewton, who claim to have the actual people who made the test on their payroll answer this.

Any of you have similar experiences??
Source: — GMAT Strategy |

User avatar
Legendary Member
Posts: 1275
Joined: Thu Sep 21, 2006 11:13 pm
Location: Arabian Sea
Thanked: 125 times
Followed by:2 members

by ajith » Mon Apr 12, 2010 7:53 pm
newton9 wrote:
I will be happy if anyone from Knewton, who claim to have the actual people who made the test on their payroll answer this.

Any of you have similar experiences??
I would like to hear from them as well ...
Always borrow money from a pessimist, he doesn't expect to be paid back.

Senior | Next Rank: 100 Posts
Posts: 71
Joined: Wed Mar 17, 2010 1:31 pm
Thanked: 5 times
Followed by:1 members

by newton9 » Mon Apr 12, 2010 8:04 pm
ajith wrote:
newton9 wrote:
I will be happy if anyone from Knewton, who claim to have the actual people who made the test on their payroll answer this.

Any of you have similar experiences??
I would like to hear from them as well ...
Yes I am also waiting. I really wonder if 3 wrong questions can bring down the score to 32.

Interesting !!!

Newbie | Next Rank: 10 Posts
Posts: 1
Joined: Thu Apr 29, 2010 4:30 pm

by stormer » Thu Apr 29, 2010 4:36 pm
The same thing has happened to me, on my first Knewton CAT I got:

Verbal: 34/68% (I got 14 questions wrong)

but on my second CAT I got:

Verbal: 27/41% yet I only got 7 questions wrong....

I really would like to hear from them as well

Senior | Next Rank: 100 Posts
Posts: 51
Joined: Thu Mar 04, 2010 9:14 pm
Location: New York, NY
Thanked: 4 times
Followed by:3 members
GMAT Score:800

by gmatninja » Fri Apr 30, 2010 6:51 am
Wow, that's really interesting stuff. I'd also be extremely curious to hear their explanation for that one.

For what it's worth, a writer who helped develop Knewton's questions warned me against using their material. That's complete hearsay and should be taken with a bucketload of salt, but it's interesting that evidence of a badly flawed scoring algorithm is starting to appear online.

Seems like Knewton has a great concept, it would be unfortunate if the execution is lacking...
helping students kick the GMAT in the nuts since 2002... https://www.gmatninja.com/

GMAT Instructor
Posts: 1578
Joined: Thu May 28, 2009 8:02 am
Thanked: 128 times
Followed by:34 members
GMAT Score:760

by Osirus@VeritasPrep » Fri Apr 30, 2010 7:50 am
.....
Last edited by Osirus@VeritasPrep on Fri Apr 30, 2010 11:03 am, edited 1 time in total.
https://www.beatthegmat.com/the-retake-o ... 51414.html

Brandon Dorsey
GMAT Instructor
Veritas Prep

Buy any Veritas Prep book(s) and receive access to 5 Practice Cats for free! Learn More.

User avatar
GMAT Instructor
Posts: 2
Joined: Thu Aug 13, 2009 6:04 pm
Location: New York City
Followed by:1 members

by david@knewton » Fri Apr 30, 2010 10:31 am
Hey everyone.

Wanted to chime in.

We're all used to tests being graded by the number of questions we answer correctly. This is the kind of scoring we've experienced all through school. Even a test like the LSAT is scored by counting the number of operational questions you answered correctly, and then applying a function to transform that number-correct score to the reported scaled score - in the case of LSAT, from 120 to 180. And on a test like the LSAT, every question contributes the same amount to your overall score. But a CAT like the GMAT works very differently. While missing a lot of questions will of course affect your score, there are a couple of additional factors to keep in mind.


First, all tests like the LSAT and GMAT present some number of questions to you that are not part of the operational test. These are often called "pretest" or "experimental" questions. On the LSAT, the pretest questions are all in separate sections from the operational questions. On the GMAT, they are interspersed among the operational questions. Your performance on pretest questions does not affect your overall score. These questions are on the test so that the test-maker can gather data to determine whether the questions are good enough to use operationally. So sometimes you might miss several questions on the GMAT that don't actually count. In those cases, it looks like you missed a lot of questions, but your score is still pretty high. (By the way, there's no point in trying to guess which are the pretest questions - just try your best on every question.)

Second, it is important to remember that the purpose of the test is to sort people along the score scale. The test is designed to try to determine what score is most likely to be the one that most closely matches your current skills and abilities. It makes use of two properties of test questions in order to do that. One property is how hard or easy a question is. We all know about that one. But the other is something most people don't know, called "discrimination". Not all questions are created equal in their ability to sort people. Think of test questions as fences you have to climb. Some are low (adequate sorters), some are of medium height (good sorters), and some are really high (great sorters). Some of the fences are wooden split rails (these are the easy questions), others chain-link (medium difficulty) and still others are razor-wire (these are the really hard questions). Your score is determined not just by what kind of fences you climb, but also by how high the fences are. Roughly speaking, the higher the fence, the more accurate the sorting.

With me so far? People often think that it is more important to get the hard questions right than the easy questions, but it's not quite so simple. Imagine you are making your way through the test. After each question the CAT engine is estimating your score, based on (a) how you answered, (b) what kind of fence it was, and (c) how high it was. Say it thinks your score should be way up there, based on your performance so far. Now you come to a wooden fence that's really high, and for some reason you just get it wrong. The CAT says "Hey, my last estimate was a pretty high score. This person really should have answered this question correctly, but they didn't." BAM! it lowers its estimate of your ability, a lot more than it would have if you had failed to climb a low razor-wire fence. The higher the fence, the more it is likely to affect your score. Miss a few high wooden ones that you really should have gotten over, and your score will definitely suffer, particularly toward the end of the test, when there is little time left to pull it back up again.

Getting over high razor-wire fences causes the CAT's estimate of your ability to go up a lot. Missing high wooden fences causes it to go down a lot. All other things being equal, it does more damage to your score to miss an easy question than a hard one, and it helps your score more to get a hard question right than an easy one.

Like with pretest items, however, there's no point in trying to figure out which questions are hard or easy (fence type), and which questions sort especially well (fence height). It is really almost impossible to tell. If it could be done accurately and reliably, test-makers wouldn't need to pretest questions - they could just tell when they wrote them how hard they'd be, and save themselves a lot of money. But they can't, even though they spend all day every day writing and reviewing test questions. So don't waste your time on it - focus instead on determining what the question is really asking you to do, and then do it.

So, to reiterate: on a CAT like the GMAT, because of the way CATs work, and because of the fact that the test includes pretest questions that don't contribute anything, the raw count of questions you answer correctly/incorrectly is not what determines your score, but rather the characteristics of the specific operational questions you faced and how you answered them.

If you want a little more information about CATs and how they work, please take a look at our CAT FAQ at https://www.knewton.com/GMAT/faq

Best,
Dave
David Kuntz
Vice President of Research, Knewton
https://www.knewton.com

GMAT Instructor
Posts: 1578
Joined: Thu May 28, 2009 8:02 am
Thanked: 128 times
Followed by:34 members
GMAT Score:760

by Osirus@VeritasPrep » Fri Apr 30, 2010 11:02 am
Thanks for the response. I didn't think anyone would. I will remove my original post.
https://www.beatthegmat.com/the-retake-o ... 51414.html

Brandon Dorsey
GMAT Instructor
Veritas Prep

Buy any Veritas Prep book(s) and receive access to 5 Practice Cats for free! Learn More.

User avatar
GMAT Instructor
Posts: 613
Joined: Thu Mar 22, 2007 6:17 am
Location: madrid
Thanked: 171 times
Followed by:64 members
GMAT Score:790

by kevincanspain » Fri Apr 30, 2010 11:50 am
I have seen over 1000 GMATPrep exam results, and in these results I have never seen a verbal score less than 42 that corresponded to only 3 mistakes, but then again, GMATPrep does not have any experimental questions, I would imagine. It will be interesting to see how well scores on Knewton´s first practice test correlate with real GMAT scores. I'm sure Knewton will continue to perfect its algorithm as it gets more data from potential MBA students , in the event its algorithm needs perfecting, of course. I advise my students to take non-GMATPrep practice exam scores with a grain of salt: look at practice exams as a way to shore up weaknesses and increase your stamina.
Kevin Armstrong
GMAT Instructor
Gmatclasses
Madrid