chet steadman wrote:

Stacey Koprince wrote:We have done extensive testing of our algorithm (using our students' real scores on the official test) and our current standard deviation on our test is in the low 50s (compared to about 30 for the official test). This doesn't mean that the test is that accurate for everyone, of course, but overall, the algorithm output is fairly close to that of the official test.

SD only tests distance to the mean, though. Do you know whether the results are generally skewed positively?

I'm curious about the answer to this-- what is the mean difference between diagnostic test scores and real GMAT scores? -- and also about the methodology behind the comparison. GMAC reports the GMAT test's standard error (not the standard deviation, of course, which is more than 100 points) to be around 30 points. That is not a measurement of how similar one test taker's results will be from one test to the next (the standard deviation of the score change is much higher in that case); it's a measure of how well the GMAT estimates a test taker's 'true' GMAT ability. I would need to be convinced of the justification for a comparison between this and a correlation measure between diagnostic test scores and real test scores; you're measuring different things.

And that's ignoring questions of using a biased estimator (people improve after doing diagnostic tests, so you would not expect identical performance from diagnostic test to real GMAT) and a biased sample (people taking a prep course aimed at those hoping for an elite test score probably aren't representative of the testing pool in general).

I certainly don't mean to dismiss the correlation data presented above; in fact, I applaud Manhattan GMAT for doing the correlation analysis in the first place- I've yet to find another company or organization that does so much. I'm just unconvinced that the '50 standard deviation' can be compared with the 30 standard error of the real GMAT- it's a comparison of different quantities- and because of that, I'm concerned that the above comparison may be misleading. I'd certainly be curious to learn more about the methodology that was used to arrive at the data presented above, and will reserve judgment about the legitimacy of the comparison until I know more.