Generally speaking, "fewer" should refer to countable (or pluralizable) items (fewer items, fewer books), while "less" should refer to things that can't be counted/pluralized (less water, less stuff).

In the examples you've mentioned, there is a slight change in meaning between using "fewer" vs. "less." You're correct in thinking that we could say "it's fewer than 35 miles to San Francisco." Miles can be counted, so we could say "fewer." This would place the emphasis on the numerical value of the miles... and it might imply that the miles would have to be in exact, whole numbers. When we use "less" in this context, it implies that we're focusing less on the numerical count of miles, and more on the idea of a whole distance. A distance would be an uncountable entity: "the distance is less than 30 miles."

Similarly, in the second example "fewer" would be technically correct if we were comparing exact, countable numbers of dollars. Generally when we talk about money, though, we talk about comparing whole amounts: "The amount of money we spent is less than $100."

We're more likely to think of "35 miles" and "$100" as expressions of distance and amount, respectively, than as exact counts of miles or dollars. So you're not wrong... but neither are these examples.

Does that make sense?