Vipul, good question - it's actually a bit simpler than that, but it relies on a bit of set theory that the GMAT doesn't often test ... which, along with the four answer choices, makes me suspect that this is an unofficial question someone wrote to help students prepare for India's CAT.
Let's take the 'literature books' to start. For each of the three books, I have one of two choices: the book is in my collection, or the book is not in my collection. So I have a total of 2*2*2 = 8 ways of making a set of those books. To illustrate, let's call the books A, B, and C:
[no A, no B, no C]
[A, no B, no C]
[no A, B, no C]
[no A, no B, C]
[no A, B, C]
[A, no B, C]
[A, B, no C]
[A, B, C]
But this question stipulates that I have at least one 'literature book' (maybe
English As She Is Spoke?
) in my collection, so I really only have SEVEN options - [no A, no B, no C] is out.
From there, we can extend the principle to the science and math books, for which we'll have 2*2*2*2*2-1 = 31 and 2*2*2*2-1 = 15 possible libraries, respectively. This gives us 31 science libraries, 15 math libraries, and 7 literature libraries, and since we must have one library from each group, we have a total of 31 * 15 * 7 = 3255 possible libraries consisting of at least one book from each group. (The trap answer 4095 is cute: this is what you get if you multiply 32 * 16 * 8, then subtract 1, i.e. if you read the question as requiring at least one BOOK, not one of each type.)
The second part of the question is a little easier, but I'll leave that for students to discuss. (It's no fun to just give everything away, right?)
