GMAT Strategy

Do you guys have any experience with Knewton CATs?
I'm currently taking the Knewton course. Great instructors, great material, great questions...TERRIBLE scoring on CATs.
Here's my experience so far:
CAT 660 Q:45/V:36(Q: 13 incorrect, V:13 incorrect)
CAT 610 Q:46/V:27(Q: 7 incorrect, V: 12 incorrect)
Please note that incorrect answers are evenly distributed on both CATs

How is it possible to get a score that is 50 points lower even though number of incorrect answers is significantly lower. As far as I know there are no "experimental" questions in these CATs. I have a real test in 10 days and my confidence is shot by these results.

GMATPrep results were 640(Q47/V31)(before I started Knewton) and 720(Q49/V39)(a few weeks ago)
PowerPrep: 700(Q49/V35) and 670(Q48/V34)

well, from what i know about the CAT, the first 10 Quant. problems decide the level of difficulty, with the increase of the difficulty increases the points you get from these questions.

hope this helps

I've just done a free Knewton CAT and have observed the same anomaly. Although i got 21/37 correct in Quant and 31/41 correct in verbal, they scored me higher in quant than in verbal.

might it have something to do with their 50 point guarantee? I speculate of course, but perhaps the test scores you 50 points below what they think you'll get in the real thing, That way, they're guaranteed to achieve the 50 point jump up in scores.

Hi everyone -- I thought some explanation here might help everyone understand what's going on.

There are three main factors that determine your score on test day: the difficulty of questions answered correctly, the difficulty of questions answered incorrectly, and answers omitted at the end of sections. It's not the number of items you get right or wrong, nor when you get them right or wrong, but the difficulty and discrimination of the items.

This is part of the reason official GMAT score reports only have scores and percentiles and not item performance, to avoid the inevitable questions and comparisons with other test instances. However, since our CATs are learning tools as well, we show response information for each item. The Knewton CAT engine does in fact use experimental items, just as the official GMAT does, for reasons such as security and item calibration.

While I don't know this student's name and thus cannot investigate the psychometrics on these particular CATs, the data doesn't alarm me. If this student got some particularly difficult items correct and missed a bunch of experimental items in the first test, then slipped up on some easy operational items in the next test, these results are in line with what we might expect to see.

Lastly, just to dispel the myth, we don't tweak our CAT scoring to help out our Money-Back Guarantee policy. We constantly work to ensure that our CATs continue to correlate with the official exam, and our data bear that out. For students who have official GMAT scores, which turns out to be a significant percentage, we recognize official scores as the baseline for the minimum 50-point increase. Because we have such a small percentage of students who end up utilizing the guarantee, accurate CAT administration is a more important concern because it affects all of our students.

Hope that helps explain a few things. You can also check out the computer-adaptive section of our GMAT FAQ and also read a bit about the test experts who built our CAT engines and algorithms.

Best,
Chris

awo84 wrote:well, from what i know about the CAT, the first 10 Quant. problems decide the level of difficulty, with the increase of the difficulty increases the points you get from these questions.

hope this helps

Thanks for you response. I am aware of the 10 first questions "rule". The problem is I had the same number of incorrect answers within the first 10 questions.

CliffordP wrote:I've just done a free Knewton CAT and have observed the same anomaly. Although i got 21/37 correct in Quant and 31/41 correct in verbal, they scored me higher in quant than in verbal.

might it have something to do with their 50 point guarantee? I speculate of course, but perhaps the test scores you 50 points below what they think you'll get in the real thing, That way, they're guaranteed to achieve the 50 point jump up in scores.

I highly doubt that Knewton would "tweak" anything. They are honest and I value that. To prove that my baseline score was 660 and one of the CATs for 610 so there is no "cheating" involved for sure. I know in the past some students would get all the questions correct in quant and still get 49. I never had the privilege to test that myself(not that good to get everything correct), but people reported this "bug" on numerous occasions.

chris@knewton wrote:Hi everyone -- I thought some explanation here might help everyone understand what's going on.

There are three main factors that determine your score on test day: the difficulty of questions answered correctly, the difficulty of questions answered incorrectly, and answers omitted at the end of sections. It's not the number of items you get right or wrong, nor when you get them right or wrong, but the difficulty and discrimination of the items.

This is part of the reason official GMAT score reports only have scores and percentiles and not item performance, to avoid the inevitable questions and comparisons with other test instances. However, since our CATs are learning tools as well, we show response information for each item. The Knewton CAT engine does in fact use experimental items, just as the official GMAT does, for reasons such as security and item calibration.

While I don't know this student's name and thus cannot investigate the psychometrics on these particular CATs, the data doesn't alarm me. If this student got some particularly difficult items correct and missed a bunch of experimental items in the first test, then slipped up on some easy operational items in the next test, these results are in line with what we might expect to see.

Lastly, just to dispel the myth, we don't tweak our CAT scoring to help out our Money-Back Guarantee policy. We constantly work to ensure that our CATs continue to correlate with the official exam, and our data bear that out. For students who have official GMAT scores, which turns out to be a significant percentage, we recognize official scores as the baseline for the minimum 50-point increase. Because we have such a small percentage of students who end up utilizing the guarantee, accurate CAT administration is a more important concern because it affects all of our students.

Hope that helps explain a few things. You can also check out the computer-adaptive section of our GMAT FAQ and also read a bit about the test experts who built our CAT engines and algorithms.

Best,
Chris

Chris,

Thanks for clarifying some of the confusion I had about the Knewton CATs.
I have no doubts that Knewton is working very hard to improve the process, but I still think that the Knewton CATs are not representative of the real score or even the GmatPrep software.
I took the GmatPrep CAT test day after I took the Knewton CAT and scored 700. Question difficulty, number of correct answers and distribution of incorrect answers were similar to those in the Knewton CAT.
Use of experimental questions may help to explain such a huge difference in scores, but there are numerous reports that even with all question answered correctly you still can't get the perfect score(Q:49 with all questions answered correctly). The only explanation I can think of is the lack of difficult questions and therefore scoring algorithm "punishes" you more(even though I did see a few difficult questions on the test).
Once again, I would recommend Knewton course any day to anyone. Their instructors are the best, practice questions are well organized and overall organization of the material helps you improve your score. With all that their CATs need some improvement.

chris@knewton wrote:Hi everyone -- I thought some explanation here might help everyone understand what's going on.

There are three main factors that determine your score on test day: the difficulty of questions answered correctly, the difficulty of questions answered incorrectly, and answers omitted at the end of sections. It's not the number of items you get right or wrong, nor when you get them right or wrong, but the difficulty and discrimination of the items.

This is part of the reason official GMAT score reports only have scores and percentiles and not item performance, to avoid the inevitable questions and comparisons with other test instances. However, since our CATs are learning tools as well, we show response information for each item. The Knewton CAT engine does in fact use experimental items, just as the official GMAT does, for reasons such as security and item calibration.

Thanks for clarifying and fair response re your guarantee. I just found it very off that I got a scaled 27 score in your verbal section with 31 correct answers and 41 scaled score in quant with 21 correct answers. any other CAT that I have done would be different.