1. Why use that?

Well, you can usually omit "that" after certain verb of attribution..

Ex: He told (that) he was not coming to play cricket.

But, if the word after that could be mistaken as an object of the verb than you shouldn't omit "that"

Ex: He told his new bat was from MRF. (incorrect)

In the above sentence you cannot omit "that" because otherwise it sounds like he told his bat which sounds non-sensical.

Correct: He told that his new bat was from MRF.

Now let's take the sentence in question:
Many nursery schools in New York City now interview two year olds and report that applications outnumber openings by as much as 5 to 1.

We cannot omit "that" here because it will sound "Many nursery schools in NY report applications outnumber openings... and so on.. giving us the meaning that schools report applications. This is not correct. Schools are reporting that applications are greater in number than openings.

So a "that" is required here.

2. what's the usage of up to and by as much as?

Usage of up to: We use up to when we set a higher limit.

Ex: Please fill water in the washing machine up to the high level.

Essentially meaning that don't go beyond the high level and just fill it to that level.

As you can see, when we are comparing two objects using "up to" doesn't help. Yes, we can say that "Please fill water in the washing machine A up to the same level present in washing machine B."

But we can't compare two things using up to.

Usage of as much as or as many as: This is mostly used in comparison.

Ex: Washing machine A holds as many clothes as washing machine B does.

Ex: I exercised as much as John did.

This clearly shows comparison.

Now let's take the original question. We are trying to compare two things here: applications and openings. The meaning that the sentence is trying to imply is that the ratio of applications to openings was as much as 5 to 1.
Lets say there were 20 openings and so there were close to 100 applications and that's the reason while comparing we need to use as much as.

That's the reason the answer is A

Hope that helps.