Yup, made a typo; meant 1/2, not 1/4. Thanks, iamtensai!
And dxgamez, I didn't 'get' my typo'd solution. In fact, with statement one alone, the answer is impossible to 'get'!
Rather, when Picking Numbers (a helpful strategy that Kaplan recommends), I try to get a Yes and a No. As soon as I can produce two solutions--ANY two solutions--that follow the statement rules but give contradictory answers, I know that contradictory answers are possible and therefore that the statement is insufficient.
So per the Kaplan method, we begin by analyzing the question stem. Again, it's helpful to understand GMAT terminology; this question is asking a specific Yes/No question, so Always Yes and Always No are BOTH sufficient, and 'Maybe' is the insufficient answer. There's not much simplification to do here, but we make sure we understand the terminology in the question: operation does not mean '+ - * /', but rather refers to any and all functions.
Next, we evaluate the statements one at a time. Here it makes the most sense to start with (2); it's a concrete definition of @. Sure enough, as ajith said, we have an operation with addition and multiplication which are both reversible. ab = ba, and a^2 + b^2 = b^2 + a^2. The value for this operation will always be the same for two numbers, regardless of what order the terms are listed in. Our answer is Always Yes. We eliminate choices (B), (C), and (E), leaving (A) and (D) as our only options.
Now we move on to statement (1). This tell us that when a and b are equal and are run through operation @, the solution is always 1/2. We have no other information; we suspect this is insufficient, but we will test it by Picking Numbers--or in this case, Picking Operations.
The first operation we pick should be the one in statement (2). Remember, we don't consider the information in (2) true when considering statement (1) alone. But statement (2) is possible, because the statements on the GMAT never contradict. So, we can choose it as one of our examples. If a@b = ab/(a^2+b^2), then M@M = 1/2, so that operation follows rule (1). We're allowed to pick it. When we pick it, we get an answer of Yes (as we already know from evaluating statement (2))
So now, we've picked a value for statement (1) that gives us a Yes--can we get a No? That's where the previous equation comes in. As I showed, a - b + 1/2 will always produce a value of 1/2 when a = b. However, you cannot flip around a and b in that equation; subtraction is not reversible! So for this value of @, we have an answer of No.
Thus, the information in statement (1) allows both Yes and No answers. Statement (1) is insufficient, and the answer is (B).