The answer is 171, not 170, though I can see why the question designer got the wrong answer. It's certainly a very difficult question, and I'm not sure my solution will be clear unless you've solved simpler versions of this type of problem, but in any case, you won't need to worry about the solution too much for the GMAT:
If you draw 21 dots on a page, there would be 21C2 = 21*20/2 = 210 ways to connect pairs of dots to make lines. Now if our 21 dots are the corners of our 21-sided shape, any line connecting two corners is a diagonal *unless* it's an edge. So we don't want to count the 21 edges, and you can draw 210 - 21 = 189 diagonals within a 21-sided shape in total. But we also don't want to count all of the diagonals emanating from one corner, and from one corner you can draw 18 diagonals (you can't connect it to itself or to either of its two neighbours). So there will be 189 - 18 = 171 diagonals that meet the conditions in the question.
Now, the way I imagine the question designer solved the problem is as follows: they probably thought "There are 20 points from which we can draw diagonals (we're leaving one out), and from each point we can connect it to any of 17 other points (we can't connect a point to itself, either of its neighbours, or to the unused point). So we should get 20*17 diagonals, but since we're then counting each diagonal twice, we should get the answer 20*17/2 = 170". But that's wrong, because from the two points that are immediately next to the unused point, we can draw 18 diagonals, not 17 (the unused point is also one of the neighbouring points), which is why this method gives an answer that is ever slightly too small. That's a very subtle point, so I can see why they'd make that mistake.
Where is the question from? It's too difficult for the GMAT -- so difficult they couldn't get the answer right! -- so I'd be pretty wary of using that source for other questions.
For online GMAT math tutoring, or to buy my higher-level Quant books and problem sets, contact me at ianstewartgmat at gmail.com
ianstewartgmat.com
