News from GMAC's Chief Psychometrician (aka: The Guy who Builds the GMAT!)

by , Oct 25, 2009

Last week, I attended the GMAC Test Prep Summit, a biennial gathering held by GMAC specifically for test prep companies (though it was expanded this year to include groups like Beat The GMAT - Eric, David, and Bea were all there as well!). Most of the things we discussed are probably only of interest to those of us who work in the industry. There were a few things, though, that would be quite useful for students to know.

When Dr. Lawrence Rudner (GMACs Vice President of Research & Development and Chief Psychometrician) speaks, it's definitely in all of our best interests to listen. He's basically responsible for the construction and continued operation of a valid GMAT CAT. All data and quotes in this article courtesy:

Rudner, Lawrence M. (2009). GMAT Psychometrics. Materials presented at the 2009 GMAC Test Preparation Summit, New York City, NY. October 15, 2009.

Penalty for leaving questions blank

The slide that I think was most important began with this headline:

Myth: It is more important to respond correctly than it is to finish.

The slide showed a graph with the integers 0 through 5 and the label # of not-reached items on the x-axis, and the percentiles 50 through 100 (in 10-point increments) on the y-axis. (If it's hard to visualize, go ahead and draw yourself a little representation of that.)

There are 6 points on the graph: the first at 0 and 70^th percentile, the second at 1 and approximately the 66^th percentile, the third at 2 and approximately the 63^rd percentile, the fourth at 3 and the 60^th percentile, the fifth at 4 and approximately the 57^th percentile and the final at 5 and approximately the 55^th percentile.

Essentially, the slide is showing what would happen to the score of a student who might have expected a 70^th percentile score if that person left blank 1, 2, 3, 4, or 5 questions at the end of the section: the score drops in a big way.

The bottom of the slide says:

Fact: There is a severe penalty for not completing the GMAT test.

Larry also told us in his voice over that there are different consequences depending upon the scoring level of the student. For lower-scoring students, running out of time doesnt have as big an impact on the score. For higher-scoring students, it makes a big difference and the higher the student was before running out of time, the more of a difference it makes to the final score. This applies both when the high-scoring student leaves items blank and when the high-scoring student guesses but gets most of the questions wrong.

What can we take away from this? Well, the specific numbers shown in Larrys chart may not be the precise penalty for every student who runs out of time on the test; according to him, it varies based upon the students scoring level. I think we can assume that the numbers are probably fairly accurate, then, for the given scenario (a 70^th percentile test-taker), but not necessarily for testers with much higher or lower scores.

Because Larry also told us that running out of time matters more for higher-level students, I think we can also assume that the penalty would be at least as high, and probably higher, for 80^th or 90^th or 99^th percentile hopefuls. Clearly, we absolutely cannot afford to spend a bunch of extra time earlier in the section that chart showed a 15 percentile-point drop for a 70^th percentile student who left 5 questions blank at the end!

Score Improvements On A 2^nd Official Test

Larry also provided some data about the kinds of score changes that people achieve from their first official test to their second official test. On average, students taking the test for the 2^nd time show a 34-point overall increase; this data has been published by GMAC previously. The standard deviation on the statistic is fairly high: 65 points. (This means that, while +34 is the average score change, theres quite a bit of variability among individual students.)

About 31% of test-takers unfortunately experience a score decrease or stay the same on the 2^nd test. About 28% achieve a score increase of 10 to 40 points and 27% achieve a 50 to 90 point increase. About 14% achieve a 100+ point increase (though only 3% achieve a 150+ point increase).

These data points are all from a 1^st to 2^nd official test. GMAC has found that, on subsequent tests, the average score change is close to zero; in other words, after the 2^nd test, it's not typical to see substantial changes in score.

There are also some trends in terms of who tends to achieve the best score increases from a 1^st to a 2^nd official test. People who did not finish the quant section the first time around record an average 49-point improvement on the 2^nd test, while those who didnt finish the verbal section record an average 48-point improvement. (No data was given on the average improvement for those who did not finish both the quant and verbal sections the first time around.)

People with below-average scores on the first test record an average 44-point improvement on the 2^nd test. Finally, people with what are called high discrepancy scores record an average 42-point improvement on the 2^nd test. A high discrepancy score has to do with a discrepancy between the testers GMAT score and his/her undergrad GPA. (Larry didnt tell us what qualifies as enough of a discrepancy to earn the high discrepancy label.)

The take-away here is two-fold. First, the average change is a 34-point increase, which bodes well for those who want to take the test a second time. Second, if you fall into one of the higher score increase categories noted above, then it may be even more worthwhile for you to take the test a 2^nd time. (Of course, you have to do the work to improve though we dont know for sure how people achieved the increased scores, I highly doubt it happened without some very hard work!)