In the xy-coordinate system, rectangle ABCD is inscribed

If a circle has the equation x^2 + y^2 = r^2, then it is a circle centered at the origin, with radius r. So our circle here is a circle of radius 5, centered at (0, 0). So, if a diagonal of the inscribed rectangle lies on the x-axis, the coordinates of its endpoints must be (-5, 0) and (5, 0).

We know the line y = 3x + 15 intersects the circle in two points. If we solve that line's equation along with the circle's equation, we'll find where those two points are. Substituting y = 3x + 15 into the circle's equation, we have

x^2 + (3x + 15)^2 = 25
x^2 + 9x^2 + 90x + 225 = 25
10x^2 + 90x + 200 = 0
x^2 + 9x + 20 = 0
(x + 4)(x + 5) = 0

and the two intersection points are at x = -5 (which we knew already) and x = -4. The y-coordinate of this second point is 3 (which you can find by plugging x = -4 into either the line's or the circle's equation, or you can notice you'll make a 3-4-5 triangle since the radius is 5).

Finally, the diagonal chops the rectangle in half. So the triangle consisting of the points (-5, 0), (5, 0) and (-4, 3) is half of the rectangle. Taking the horizontal edge, along the x-axis, as our base, the length of the base is 10. Then the height is just 3, the y-coordinate of the point above the x-axis. So the area of this triangle is 3*10/2, and that's half the rectangle's area, so the rectangle's area is 30.