The DEMONSTRATIVE PRONOUNS "that" and "those" stand in for a new version of the noun they replace. They always have to agree in number with the noun they replace, though.

My cat is fatter than that of my sister.

Her puppies are fluffier than those of her friend.

In your example sentence, if we use "that of," it has to replace a SINGULAR noun. The only noun it can replace is "composition." If we substitute that back in for the pronoun, it reads:

As Charles Darwin observed, natural selection operates whenever individuals of one genetic composition are better at reproducing than the genetic composition of others.

This doesn't make much sense. The much more logical sentence would read:

As Charles Darwin observed, natural selection operates whenever individuals of one genetic composition are better at reproducing than individuals of others.

In order for us to compare "individuals" to "individuals," we need the plural pronoun "those of."

You may have noticed, though, that there is also an agreement problem with "others." It should be referring to "genetic composition," but "others" is plural.

As Charles Darwin observed, natural selection operates whenever individuals of one genetic composition are better at reproducing than those of others.

We need to change "others" to "another" to correctly replace "genetic composition." The answer is D.