The wording of the question doesn't make sense. It asks for a probability, but if we're finding a probability, we have to be selecting something from a clearly defined set. The question doesn't even mention selecting anything, let alone mention how we're making the selection. I think it should read: "If point P is a randomly selected pointed on line segment AB, what is the probability the length of PD is less than √45?"
This is then a 1-dimensional geometric probability question. In general, if we choose a point at random from a line of length L, and we want to find the probability that point will be in some smaller line segment of length d, the probability will just be d/L. We often do this kind of calculation specifically on the number line -- if, say you pick x at random so that 0 < x < 5, and you want to know the probability that 0 < x < 2, the answer is just 2/5 -- but the same principle applies to a random selection from any line.
Now, returning to the question, if PD < √45, then by the Pythagorean Theorem, 6^2 + PA^2 < (√45)^2, so PA^2 < 45 - 36, and PA < 3. So the length of the line PA from which we can choose P is 3, and since the total length of the line from which we can choose P is 8, the answer is 3/8.
For online GMAT math tutoring, or to buy my higher-level Quant books and problem sets, contact me at ianstewartgmat at gmail.com
ianstewartgmat.com