I think it's less dangerous to follow subject-verb agreement and modifier rules here, but I also think a lot of people might find it easier to understand the explanation as given above. The one danger to the above explanation is that it could lead to mistakes on harder questions where the logic isn't so clear cut - so, if possible, try to learn the actual grammatical rules, but don't stress too much if you find the actual grammar too confusing!

In the first example, we've got:
The swarm of flies was very dense
<subject> <prep phrase> <verb> <other stuff>

Here, we've got a question of standard subject-verb agreement in an independent clause. The prepositional phrase contains a noun and subjects are nouns, yes, but prepositional phrases do not contain subjects. So "flies" cannot be the subject that matches the verb. Instead, we use the noun / subject "swarm."

In the second example, we've got:
He is one of the students who believe math is boring.
To start, this sentence isn't completely grammatically correct. You'd either have to say:
He is one of the students, who believe math is boring. (Implication: he is one of the students out of some larger group that includes both students and non-students - and, by the way, the students all happen to think that math is boring, but this is extra info and not the main point of the sentence)
or
He is one of the students that believe math is boring. (Implication: he is part of a student-only group in which some of the students believe math is boring and some of the students believe math is not boring - in his case, he belongs to that subset that believes math is boring.)

In either case, we've got a modifier set-up in the relevant part of the sentence.
He is one of the students that believe math is boring.
<subject> <verb> <noun> <prep phrase> <modifying clause beginning with "that">
So "He is" already addresses the subject-verb agreement issue (in both sentences, actually). What we've got now is either "who" or "that" introducing a clause - both of these are modifiers and are supposed to modify the main noun preceding that modifier. So: what is that main noun?

Generally, when you have a "one of" or "some of" or "two of" kind of set-up for a modifying clause, then you use the noun that is in the prepositional phrase as your main noun. In the above case, that would be students.

You can have exceptions in which you have a "noun + short prep phrase" followed by a modifier word in which the first noun is actually the main noun. eg "He hated the computer on his colleague's desk, which beeped annoyingly at random times." The computer is beeping, not his colleague's desk. You can't reasonably re-write that as "He hated his colleague's desk's computer, which beeped annoyingly..." so you just have to live with the short prepositional phrase separating "computer" and "which."

But if you've got the "one of (some group)" modifier set-up, this doesn't meet the above exception - you just use the "some group" part to decide how to conjugate the verb sitting in the modifying clause.

The third example:
One of the students believes that math is boring.
<Subject> <prep phrase> <verb> that <subj, verb, etc>
So we're back to the subj-verb thing here, not a modifying situation. Because "believes" needs to match with a subject, and prep phrases can't contain subjects, "one" is the only noun left to qualify as the subject.

Note: some people get confused because this one contains a "that" too. Notice that in the second example, we had <noun> that <verb> - that's a modifier structure. In the third example, we have <noun> <verb> that <noun> <verb> - that's a dual subj-verb agreement structure, with one noun-verb pair before and one after the "that."