Accuracy of Knewton CATs

This topic has expert replies
Junior | Next Rank: 30 Posts
Posts: 16
Joined: Sun Jul 05, 2009 4:52 pm

Accuracy of Knewton CATs

by doggdetroit » Sat Jun 19, 2010 3:23 pm
Hello,

I just completed the 5th of 5 Knewton CATs offered as part of the Knewton course. Here are the five scores:

Diagnostic GMAT 02/10/2010 Q35 V35 570
Practice GMAT #1 05/08/2010 Q40 V47 700
Practice GMAT #2 05/22/2010 Q43 V40 680
Practice GMAT #3 06/05/2010 Q32 V38 580
Practice GMAT #4 06/19/2010 Q38 V35 590

The very first diagnostic CAT was taken prior to my most recent attempt at the GMAT on March 27th, (I ended up scoring a 640, Q42 V35,) and I had to stop and start the CAT multiple times, so I don't feel that it is very reflective of my current abilities. I stopped studying for about a month after my test day, before enrolling in Knewton's course. A month later, after some pretty regimented studying, I took CAT #1.

As you can see, I am pretty alarmed about the downward spiral after CAT #1. Not only have I gotten progressively worse, but I have gone down 110 points from CAT #1 to CAT #4.

After CAT#3, I contacted Knewton support to see if anything was wrong with the scoring. They said that my score was in fact accurate and that most people experienced a drop when taking CAT#3. I still found it hard to believe that my score could drop that much, especially because I had been studying pretty hard in between CATs. I mean look at my quant, a 32 good for the 33rd percentile? Please, I scored a 42, good for the 62nd percentile on the real deal, and the real deal was much harder than any of these CATs. The only thing I can think of is that there was a death in my family a few days before I took CAT#3 so I thought that maybe my mind was elsewhere, resulting in a lower than normal score.

After CAT#4, I knew that something had to be wrong with the scoring. I only got 9 out of 37 wrong on the quant portion and only 6 out of 41 wrong on the verbal portion. I realize that the scoring is adaptive, but surely if I only get 6 wrong on the verbal, it must mean that I am scoring at a high level. (I feel that the quant score is a little more reflective of my abilities, but I still feel it is slightly low.) There is just no way that you can finish the verbal section with only 6 incorrect questions and wind up with a 35 (73rd percentile.) Just the fact that you got 35 questions right should indicate that you are scoring at or near a 700 level, correct? Maybe I'm mistaken, but I feel my verbal score should be in the high 80th / low 90th percentile.

Anyway, I decided to contact Knewton one more time just to make sure that there wasn't any foul play going on with the scoring, but I decided to post my thoughts here as well in case Knewton gives me the company line. Has anyone else heard of the Knewton scoring algorithm being off? Have any Knewton CAT users experienced a drop like this from CAT #1 to #4? Do Knewton's CATs generate lower scores than say ManhattanGMAT or Kaplan CATs? (By the way, I have really enjoyed everything else about Knewton. The classes and instructors are helpful, and I feel that the homework problems are extremely well written and very GMAT like. I also think the questions on the CATs are very realistic, but the scoring just seems off.)

The reason that I am concerned is because I am scheduled to take the GMAT on July 24th, and I would like to be scoring near 700 (my target score) heading into it. I am planning on taking 2 GMATPrep CATs in the final three weeks leading up to my test date, but since I have already taken them before, I probably will see some repeats and my scores may not be indicative of my current scoring level.

Newbie | Next Rank: 10 Posts
Posts: 1
Joined: Mon Jun 21, 2010 7:50 am

by rptiffany » Tue Jun 22, 2010 8:06 pm
i had the same issue. i got 6 wrong on the verbal and it gave me a 30/55%. no way!

howd the real test go?

User avatar
GMAT Instructor
Posts: 85
Joined: Thu Aug 13, 2009 5:43 pm
Location: New York City
Thanked: 18 times
Followed by:1 members

by josh@knewton » Thu Jun 24, 2010 1:06 pm
Hi DoggDetroit,

Thanks for your feedback. We wanted to let you know that we have identified a bug which affected your most recent CATs and have corrected it. (The short explanation: the scoring is correct for the questions you saw but the engine didn't present quite enough sufficiently difficult items to enable you to demonstrate a higher score.) A small number of other students were also affected, and we're notifying them now. We're very sorry for any inconvenience. Please don't hesitate to reach out if anything else comes up: [email protected]. Best of luck on your upcoming GMAT!

Sincerely,
Josh
Josh Anish
Senior Editor, Knewton
https://www.knewton.com

Junior | Next Rank: 30 Posts
Posts: 16
Joined: Sun Jul 05, 2009 4:52 pm

by doggdetroit » Mon Jun 28, 2010 5:23 pm
Thanks for the update Josh.

Senior | Next Rank: 100 Posts
Posts: 61
Joined: Sun Mar 21, 2010 9:51 am
Thanked: 1 times

by prac » Thu Jul 01, 2010 10:03 pm
Hi,

I am facing the same issues in the Knewton Tests. In my CAT4, I have got my 5 questions wrong in math and scored 39.
I dont belive this score at all.I am not sure whether gmat penalize you so much for 5 wrong answers. Do we need to correct all 37 questions to get 50 marks?.

I am taking my exam on 22 July and I am really depressed by seeing this.
This scoring pattern is decreasing my confidence level.


Please advise.

Thanks
Prachi

Newbie | Next Rank: 10 Posts
Posts: 1
Joined: Sun Nov 09, 2008 4:04 pm

by mohan12 » Fri Jul 02, 2010 1:53 pm
Talk about good timing!

I also reached out to Knewton about a similar issue. On CAT #2, I thought I nailed the verbal section (37/41 correct, including the last 17 in a row) and initially didn't get a score (bug in their system?) After reaching out to customer service, the issue was apparently fixed and I was scored a 32 on the section.

This seemed off....I was a bit bummed but then became really confused when I looked back at CAT #1 and realized that I was scored a 34 on verbal, even thought I had gotten 16 wrong (25/41) including skipping the last one due to time expiring.

I'm still not sure how this works out:

Practice Test 1 Verbal: 41 questions, 40 answered, 25 correct, 16 incorrect, Score: 34/68%
Practice Test 2 Verbal: 41 questions, 41 answered, 37 correct, 4 incorrect, Score: 32/63%


Huh?! Knewton CS was pleasant as always and sent me a nice link to a posting on BTG about how adaptive scoring works and had someone else look at my verbal section to confirm. That said- can 4 wrong (early in the section, last 17 questions answered correctly) really bring one down to a 32/63%?

IF that's that's possible, how the heck could 16 incorrect responses scattered throughout the section (including the last question due to time running out) result in a HIGHER score (34/68%) ?

Can someone chime in, b/c while I've moved on to prep for CAT #3, something doesn't seem right here and I can't help feeling a bit unsure about Knewton scoring. It's unfortunate, b/c aside from this, Knewton has been nothing but great.

Any input would be really really appreciated!
rptiffany wrote:i had the same issue. i got 6 wrong on the verbal and it gave me a 30/55%. no way!

howd the real test go?

User avatar
GMAT Instructor
Posts: 613
Joined: Thu Mar 22, 2007 6:17 am
Location: madrid
Thanked: 171 times
Followed by:64 members
GMAT Score:790

by kevincanspain » Fri Jul 02, 2010 3:29 pm
Don't use Knewton CATs to gauge your progress. The same can be said for Kaplan and Manhattan Gmat CATs. Do use their CATs to get a sense of the timing and the question types. Make sure you understand your mistakes and what you can do to avoid making similar ones. In fact, you can't even use GMATPrep to mesaure your progress, as the scores are meant to be orientative, not predictive. Simply learn as much as you can in terms of concepts and tecniques and perfect your pacing!
Kevin Armstrong
GMAT Instructor
Gmatclasses
Madrid

User avatar
Legendary Member
Posts: 758
Joined: Sat Aug 29, 2009 9:32 pm
Location: Bangalore,India
Thanked: 67 times
Followed by:2 members

by sumanr84 » Sun Jul 04, 2010 7:07 am
kevincanspain wrote:Don't use Knewton CATs to gauge your progress. The same can be said for Kaplan and Manhattan Gmat CATs. Do use their CATs to get a sense of the timing and the question types. Make sure you understand your mistakes and what you can do to avoid making similar ones. In fact, you can't even use GMATPrep to mesaure your progress, as the scores are meant to be orientative, not predictive. Simply learn as much as you can in terms of concepts and tecniques and perfect your pacing!
I think this is a perfect advice that anyone could give..thanks Kevin..

User avatar
Newbie | Next Rank: 10 Posts
Posts: 6
Joined: Mon Jun 28, 2010 6:32 am

by vivnitrel » Wed Jul 14, 2010 3:16 am
Hi!
I took the Knewton Free CAT today
This is how i performed:

Verbal:
37 correct, 5 incorrect. score = 43 (96 percentile)

Quant:
34 correct, 3 incorrect. score = 49 (87 percentile) (I marked wrong answers for ques 9, ques 36 and ques 37)

Overall = 750 (99 percentile)

Please clarify the following:

1) In quants, GMAT prep awarded me 51 for 4 incorrect ques, 50 for 7 incorrect ques. How does the scoring method for Knewton works? Do I have to answer all questions correctly to get a 51??

2) Was it because of the 2 wrong answers at the end of quants, that I got 49?? Does finishing a section with a right answer matter so much.

3) What is level of difficulty of Knewton's free CAT compared to the actual GMAT. I need this information in order to decide whether to go for the paid course or not. I have taken 6 MGMAT CATs, which I found to be tough compared to GMAT prep.

User avatar
GMAT Instructor
Posts: 85
Joined: Thu Aug 13, 2009 5:43 pm
Location: New York City
Thanked: 18 times
Followed by:1 members

by josh@knewton » Wed Jul 14, 2010 7:48 am
Hi all,

Thanks for your feedback. To answer your last question first, vivnitrel, we're very confident in the accuracy of our CATs and do everything in our power to match the level of difficulty to that of the actual GMAT. Our CATs were designed by the architects of the original GMAT.

For insight into some of the other questions, take a look at this post on a previous thread by David Kuntz, VP of Research at Knewton. Dave also points to Knewton's GMAT FAQ page, which contains answers to some often-asked questions about our CATS.

Some particularly helpful points from these two sources:
1) Vivnitrel, you asked about how the scoring method for Knewton works. You probably already know some of this from general knowledge and research, but just to clarify (this can be found on the GMAT Course FAQ page): A computer-adaptive test (CAT) doesn't really care as much about how many questions you get right or wrong, but rather which questions you get right and wrong. The CAT algorithm estimates your ability based on a variety of criteria, including the difficulty of a question. After each question, it evaluates your response and updates this estimate. When the test is over, the algorithm converts your quantitative and verbal ability estimates into the quantitative and verbal scaled scores, and then separately combines your quantitative and verbal ability estimates to calculate the overall score.
2) mohan, in response to your questions, I think you might find Dave Kuntz's post helpful. In particular, check out the point about experimental questions (which you might get wrong, but which won't affect your score), as well as the explanation of how questions vary not only by how difficult they are, but also by how well they "sort" test-takers.

Please don't hesitate to reach out if you have any further questions after taking a look at these materials!

Best,
Josh
Josh Anish
Senior Editor, Knewton
https://www.knewton.com

Junior | Next Rank: 30 Posts
Posts: 14
Joined: Fri Jul 16, 2010 10:12 pm

by ssaravanaraj » Fri Jul 16, 2010 10:16 pm
Hello:
I got 620 in my diagnostic free GMAT test. Is this tough when compared to other knewton CATS?
I was expecting a better score in Verbal, for some reasons it scored me very low. I got 14 wrong ans in verbal.

Please help, I am taking my GMAT on 20th.

Test Information (Score / Percentile Rank)
Verbal Quantitative Total
26 / 39% 48 / 83% 620 / 76%

User avatar
Senior | Next Rank: 100 Posts
Posts: 31
Joined: Tue Jun 29, 2010 11:02 pm
Location: Honolulu
Thanked: 1 times

by bmorgan » Tue Aug 24, 2010 5:04 pm
This issue has not been resolved. FYI
I do not have superior intelligence or faultless looks. I do not captivate a room or run a mile under six minutes. I only succeeded because I was still working long after everyone else went to sleep.

-Greg Evans

Veritas CAT: 680
GMATPrep CAT: 680

Legendary Member
Posts: 544
Joined: Thu Oct 08, 2009 9:14 am
Location: Pune, India
Thanked: 31 times
Followed by:2 members

by adi_800 » Sat Aug 28, 2010 11:35 pm
To Knewton

Just gave the free test with log in ID [email protected]

Quant : 8 wrong
Verbal : 8 wrong
Score
Verbal Quantitative Total
31 / 58% 42 / 62% 600 / 70%

Some weird software...
Seems that the BUG has not been resolved...lol
Can't you simply produce a proper tests??

N yes...stop copy pasting the questions from OG...PLS...
Changing the subject verb of a CR or SC does not change the reasoning or grammar being tested in a sentence...
It just fools a test taker.....

User avatar
Senior | Next Rank: 100 Posts
Posts: 31
Joined: Sun Nov 08, 2009 1:15 pm
Location: Indianapolis, IN
Thanked: 8 times
Followed by:2 members
GMAT Score:760

by michaelfaulkner » Sat Feb 26, 2011 8:37 am
The Knewton algorithm is still seriously flawed.

Today, I missed 3 on the quant section and scored a 45. Last week, I missed 5 on CAT#4 quant section and scored a 46.

There's no way that missing only 3 question should drop my score that low.

For the first official GMATPrep CAT I missed 11 problems and scored a 48. For the second CAT I missed 9 and scored a 49.

Newbie | Next Rank: 10 Posts
Posts: 1
Joined: Sun Jun 05, 2011 7:53 am

by IMWATIM » Sat Jun 18, 2011 2:26 am
I gave Knewton's daignostic test today and the scores were lower than what I had expected..

Quant 46
Verbal 34

I took GMAT prep #1 couple of months back and scored 700.. after that I worked hard on my quant & verbal ... but the quant score appeared disastrous. I missed 3 questions and scored 46 .. Amazing!!! Also the difficulty level of the questions was pretty much same throughout the Quant section.. the questions never got tougher despite getting most of them right in a row...

Verbal score appears to be skewed as well. Getting 10 questions wrong.. gave me 34.. I got 15 of them wrong in GMAT prep but managed 35... I dont know how to gauge my current preparation level based on this score...

If someone could provide some insights it would be appreciated...

Tnx!!!