Accuracy of Knewton CATs

I wouldn't want to comment on the integrity of any individual company's scoring algorithm, so I'm just speaking generally about GMAT-style scoring algorithms. If a test is full of hard questions, you can make several mistakes and still get a good score. If, on the other hand, a test is full of easy questions, you'll need to be nearly perfect to get a good score. So it is certainly possible for a prep company test to use the same scoring algorithm as the real GMAT, but for the scores to be very different between the two tests even with a similar number of wrong answers. For example, in the Quant section of GMATPrep, you can have 7-10 wrong answers and still get a 50 (out of 51), provided your wrong answers are only on the absolute hardest questions. This is possible because GMATPrep contains many true 800-level problems. If, on the other hand, a test was drawing questions from a pool that only contained easy and medium level problems, then it would be impossible with 10 mistakes to get anything close to a 50.

So if a test is producing low Quant scores even when you have only 7 mistakes, it is possible the test is still using the correct scoring algorithm; it may just be that the question pool is far too easy, compared to the real GMAT. The more questions a test contains at a certain level, the more accurate scores tend to be around that level, so tests which contain only easy questions won't produce very accurate scores for high-level test takers, and tests which contain only hard questions won't produce very accurate scores for low-level test takers.