This question is awesome. Best way to approach it: first, draw one 3-D box to give yourself an idea of what you're looking at and label the edges according to length. Write the formula for the volume of the cylinder: V = Length * PI * radius^2
From the formula, you'll see that increasing the radius is the best way to increase the cylinder's volume.
Trouble is, and this is the catch: you can't have "the biggest radius" of 5 in this particular case because the biggest face is? 10x8, not 10x10. You still want the biggest radius, so you go with 10x8 and your radius is 4. The volume is L*PI*(4)^2. Lenth is six, so volume is 96Pi. You could draw another box where the length of the cylinder is 10, but your radius would drop to three (face is 8x6) and you'd be stuck at 90Pi for the volume.
Comments: agemroy, we can't use a diameter of 10 but not for the reason you've specified. The cylinder literally won't fit in the box. olpre4, it actually doesn't matter which dimension we assign height to.