As best as I could determine from the GMAC reports on scoring, the penalty for unanswered questions is proportional - that is, for each unanswered question your score estimate is reduced by a factor of 1/37. However, their reports aren't completely clear on just how this proportional penalty works, so I'm not entirely sure my interpretation is correct. In any case, because the penalty is taken as a proportion of your score, the penalty is more severe the higher your score.
According to GMAC's own research, there is no clear-cut answer as to whether it helps to guess at the end or to leave questions blank. This is in part because they can't predict just what questions you'll be guessing at - if those questions are low-level, guessing incorrectly will probably hurt you more than would leaving the question unattempted. Still, on balance, it seems that most of the time completing the test is better than leaving the test unfinished, even if that means you need to guess. This is particularly true for high level test takers.
As for whether your last questions might be experimental, I don't agree with the theory in the posts above. There's an extensive GMAC research report which describes precisely how experimental questions are used on your test - when each test taker sits down to start his or her test, before answering any questions, the test determines purely at random which question numbers will be experimental questions (so before you start your Quant section, the test will have decided that, for example, questions 5, 9, 14, 16, 17, 23, 25, 31, and 36 will be experimental, and nothing you do during the test will change that). The report makes no mention that the last question in a section is ineligible to be experimental, and if that were the case, then test takers informed of that technicality would have an advantage over other test takers, so I very much doubt the theory to be true; it makes much more sense, in order for the test to be fair, for the selection of experimental questions to be purely random.
For online GMAT math tutoring, or to buy my higher-level Quant books and problem sets, contact me at ianstewartgmat at gmail.com
ianstewartgmat.com