Dana's explanation is perfect, but I've never seen a real GMAT question which uses that notation for an interval (ranges of values are always expressed using inequalities, a number line, or words), so you likely don't need to know it. In algebra, square brackets [] and parentheses () have the same meaning (in algebra texts, you'll sometimes see square brackets used as outer brackets when there are brackets within brackets), though I don't think I've ever seen that on the GMAT either - I'm pretty sure the GMAT uses standard parentheses exclusively in algebraic expressions. Set braces {} will only be used in a question about sets, so if you see something like {6}, that's the one element set containing the number 6.
For online GMAT math tutoring, or to buy my higher-level Quant books and problem sets, contact me at ianstewartgmat at gmail.com
ianstewartgmat.com