The answer is 171, not 170, though I can see why the question designer got the wrong answer. It's certainly a very difficult question, and I'm not sure my solution will be clear unless you've solved simpler versions of this type of problem, but in any case, you won't need to worry about the solution too much for the GMAT:

If you draw 21 dots on a page, there would be 21C2 = 21*20/2 = 210 ways to connect pairs of dots to make lines. Now if our 21 dots are the corners of our 21-sided shape, any line connecting two corners is a diagonal *unless* it's an edge. So we don't want to count the 21 edges, and you can draw 210 - 21 = 189 diagonals within a 21-sided shape in total. But we also don't want to count all of the diagonals emanating from one corner, and from one corner you can draw 18 diagonals (you can't connect it to itself or to either of its two neighbours). So there will be 189 - 18 = 171 diagonals that meet the conditions in the question.

Now, the way I imagine the question designer solved the problem is as follows: they probably thought "There are 20 points from which we can draw diagonals (we're leaving one out), and from each point we can connect it to any of 17 other points (we can't connect a point to itself, either of its neighbours, or to the unused point). So we should get 20*17 diagonals, but since we're then counting each diagonal twice, we should get the answer 20*17/2 = 170". But that's wrong, because from the two points that are immediately next to the unused point, we can draw 18 diagonals, not 17 (the unused point is also one of the neighbouring points), which is why this method gives an answer that is ever slightly too small. That's a very subtle point, so I can see why they'd make that mistake.

Where is the question from? It's too difficult for the GMAT -- so difficult they couldn't get the answer right! -- so I'd be pretty wary of using that source for other questions.

mehaksal wrote:I just simply used the logic of n(n-3)/2 diagonals, and n here is 20.
But Ian's explanation seems very valid..thoh 171 is not an option!
So what shud I go by??

Is there some reason you trust the source of this problem? Where did you find it? The answer is definitely 171, and not 170.