OA B?

X A) which can be roaring by causing its hyoid bones to vibrate, domestic cats - wordy, can be roaring by causing is too long and unnecessary, its must be their since we refer to plural cats and tigers.

B) which can roar by causing their hyoid bones to vibrate, domestic cats
X C) who can roar by causing their hyoid bones to vibrate, domestic cats differently - unnecessary use of differently. Unlike already conveys the difference.
X D) who can roar by causing its hyoid bones to vibrate, domestic cats - its must be their.
X E) of which the hyoid bones vibrate to cause a roar, domestic cats - of which is horribly wrong.

Your are correct, the OA is B.