Wow, this question is no joke - quite challenging for the GMAT.

Here's my reasoning:

There are (10 choose 3) = 120 ways of choosing three books, with no restrictions.

It's easier to find ways to VIOLATE the conditions than to observe them, so let's work backwards.

Our ILLEGAL groups of three fall into two types:
1) All three books are adjacent (a row of three)
2) Two of the books are adjacent and the third is not adjacent to either of these two

There are 8 ways to choose three books, all of which are adjacent. (Essentially the middle book of the three adjacent books can be any of the non-red books below.)

B B B B B B B B B B

So that gives us 8 illegal groups of three.

Now let's find the number of ways to pick two adjacent books, with the third book a book that ISN'T adjacent to either of the first two.

If I pick the two adjacent books at the end, I could have either

B B B B B B B B B B

or

B B B B B B B B B B

In either case, as long as I pick one of the 7 non-red books for my third book, I have a two-adjacent book selection. This gives 2 * 7 = 14 more INVALID arrangements.

If I pick two adjacent books, neither of which is a book on the end, there are 7 such pairs. Here is one of them:

B B B B B B B B B B

As long as my third book is one of the 6 non-red books, I have a two-adjacent selection. This gives 7 * 6 = 42 more INVALID arrangements.

My total number of invalid arrangements is thus 8 + 14 + 42 = 64.

Total - Invalid = Valid, so 120 - 64 = 56

A word of caution, Sana: you're asking a lot of combinatorics questions, many of which are quite hard by GMAT standards. Combinatorics questions are rare on the GMAT, especially involved ones like this, so I'm not sure this is a productive way to study. (But of course it's up to you )

sana.noor wrote:Ten books are arranged in a row on a shelf.In how many ways can 3
books be selected simultaneously from the self such that no two
adjacent books from the shelf are chosen?

1) 36

2) 48

3) 42

4) 56

5) 84

OA is 4

The number of ways to REMOVE 3 books when there are 7 other books is equal to the number of ways to ADD 3 books when there are 7 other books.
Thus, the problem above is no different from the following:

A shelf that can hold a maximum of 10 books currently holds 7 blue books. How many ways can 3 identical red books be ADDED to the shelf, if no red book may be placed adjacent to another red book?

Here are the 7 blue books currently on the shelf:
_B_B_B_B_B_B_B_

For the 3 red books to be kept separated, they must occupy 3 of the 8 slots above.
Number of ways to choose 3 of 8 options = 8C3 = (8*7*6)/(3*2*1) = 56.

Since there are 56 ways to ADD 3 books when there are 7 other books, there are 56 ways to REMOVE 3 books when there are 7 other books.

The correct answer is D.