Correct, (3/80) has to be less than (5/X)
And 133 maximizes the number of students while keeping the relationship between the two ratios true. Compare the value of the two fractions by cross-multiplying:
(3/80)(5/133) gives the value of the first fraction as 399 and the value of the second fraction as 400. Thus, (5/133) > (3/80). Also notice that when comparing the fractions they're only one number apart, illustrating that if the number of students in the class is 134 the relationship between the ratios would not hold true and if the number of students in the class is 132 there is still room for more students to be in the class while keeping the relationship between the ratios true.
(3/80)(5/132) gives the value of the first fraction as 396 and the value of the second fraction as 400. 396 is less than 399, thus, 132 does not maximize the number of students.
Also, I came across this problem recently while taking a gmatprep test. The OA is 133.