If the person answers randomly, he has a 50%=1/2 probability of answering correctly.
The probability of answering n questions correctly by random guessing is therefore (1/2)^n:
The probability of answering one question correctly is 1/2
The probability of answering 2 questions correctly is 1/2*1/2=(1/2)^2=1/4
The probability of answering 3 questions correctly is 1/2*1/2*1/2=(1/2)^3=1/8

The question asks what's the minimum nunmber of questions to make the probability less than 1/1000. Move up in powers of 2, until you reach a power that is greater than 1000, making the fraction (1/2)^n smaller than 1/1000
n^5=32
n^6=64
n^7=128
n^8=256
n^9=512
n^10=1012

So the prob of getting 10 questions right is (1/2)^10 = 1/1012 - which is smaller than 1/1000, so B is the answer.

thanks Geva@MasterGMAT