There are many types of math, with differing sorts of value, but I can say a little about the sort of math I find moving.
I agree with you. For the most part, applied souls dream up their advances and make them without relying on the mathematical machine. They invent the math they need to describe their ideas. Or perhaps they use a little of the pure mathematician's machine, but quickly develop it in ways that are more important to their work than the previous mathematical meanderings.
I think you underestimate the role of mathematics as the grand expositor. It is the tortoise that trails forever beyond the hare of applied science. It takes the insights of applications, of calculus for example, and digests them. It reworks them, understands them, connects them, rigorizes them.
The work of mathematics is not useful in your mind because a mathematician does not make a truly new applied advance. A mathematician invents and connects notations to ease the traversal, the learning, and most importantly the storage in working memory of past insights.
What is the purpose of a category? An operad? A type theory? A vector bundle? The digit 0? When these languages were introduced, it could always be claimed they were worthless because the old languages could express the same content as these new languages. But somehow the new language makes it easier to conceptualize and think about the old ideas; it increases the working human RAM.
And what of the poor student? He who must learn so many subjects is grateful when it is realized that many of those subjects are in fact the same: http://arxiv.org/abs/0903.0340 . Mathematics digests theories and rewrites them as branches of a common base. It makes it possible to learn more insights quickly and to communicate them to the next generation.
So young applied scientists, perhaps generations later, benefit by more compactly and elegantly understanding the insights of their forebearers. Then, the mathematician dreams, they are freer to envision the next great ideas: http://arxiv.org/abs/1109.0955
So why the mathematician's focus on solving specific problems? Why so much energy to characterize finite groups? It is not that these problems are important. It is that they serve as testbeds for new languages, for new characterizations of old insights. The problems of pure math are invented as challenges to understand an old applied language, not to invent a new one.
For many of us, choosing a career path has a dominant effect on our contribution to the society. For those of us who care what happens to society, this makes it one of the most important decisions we make. Like most decisions, this one is very often made by impulses significantly below the level of conscious recognition, with considerable intellectual effort spent on justifying a conclusion but very little spent on actually reaching one. In the case of smart, altruistic rationalists, this seems like the most tragic failure of rationality; so, whatever the outcome, I advocate much more serious consideration by smart rationalists of how our career choices affect society. For the most part this is a personal thing, but some public discussion may be valuable. I apologize (largely in advance) for anything that seems condescending.
I previously planned to do research in pure math (and more recently in theoretical computer science). I frequently justified my position with carefully constructed arguments which I no longer believe. It still may be the case that doing research is a good idea (and spending the rest of my life doing research is still the easiest possible career path for me), so I am interested in additional arguments, or reasons why anything I am about to write is wrong. Here is a basic list of my justifications, and why I no longer believe them.
Argument 1: Much math is practically important today. The math I am working on is not practically important today, but maybe it will be the math that is practically important tomorrow. How can we predict what will be useful? It seems like pushing math generally forward is the best response to this uncertainty.
Rebuttal: If we really want to evaluate this argument, it is important to understand the conditions under which the important math of today was done. In the case of calculus, differential equations, statistics, functional analysis, linear algebra, group theory, and numerical methods, the important results for modern work were in fact developed after their usefulness could be appreciated by an intelligent observer. There is very little honestly compelling evidence that pushing math for the sake of pushing math is likely to lead to practically important results more effectively than waiting until new math is needed and then developing it. Perhaps the most compelling case is number theory and its unexpected application to cryptography, which is still not nearly compelling enough to justify work on pure math (or even provide significant support).
Argument 2: Math is practically important today. The math I am working on is in a field that is practically important today, and not many people are qualified to work on it, so pushing the state of the art here is an excellent use of my time.
Rebuttal: Consider the actual marginal utility of advances in your field of choice, honestly. In the overwhelming majority of cases, the bulk of research effort is directed grotesquely inefficiently from a social perspective. In particular, a small number of largely artificial applications will typically support research programs which consume an incredible amount of intelligent mathematicians' time, compared to the time required to make fundamental progress on the actual problem that people care about. Here you have to make a different argument for every research program, which I would be happy to do if anyone offers a particular challenge.
Argument 3: Theoretical physics research advances the fundamental limits of understanding, which has led to important advances in the past and will probably continue to lead to important advances.
Rebuttal: What matters are interactions in regimes that humans can engineer---improving understanding in such regimes is responsible for every technological development I am aware of. In particular, improvements in our understanding of high energy physics or cosmology are unlikely to be useful until we can design systems which operate in those regimes. There is fundamental physics research which seems likely to have a high payoff---but if you approach theoretical physics with the honest goal of contributing to technological progress, you end up with a research program which is unrecognizably different from most physicists'.
Argument 4: Pure research is at least a little useful, and its what I am best prepared to do.
Rebuttal: There is a shortage of intelligent, rational people in pretty much every area of human activity. I would go so far as to claim this is the limiting input for most fields. If you don't believe this, at least ask yourself why not. Do you have experience in other fields that suggests you are unable to contribute? Do you have a causal argument?
Argument 5: Society is relatively efficient. The marginal returns for work in every field are roughly comparable, so I should work wherever I have comparative advantage.
Rebuttal: Why should society be remotely efficient? I believed this for a long time, but eventually realized it was just a hold-over from a point in my life where I had more faith in other people. If you are typical LW readers, you probably believe at least half a dozen strong counterexamples to this claim already.
Argument 6: Pure research has fundamental value as an intellectual pursuit.
Rebuttal: For whom? If you are concerned exclusively with the intellectual richness of mathematicians' lives, then I can't well disagree and this argument may be completely convincing. Otherwise, if you believe that the increasing richness of human mathematics is a fundamental good which non-mathematicians can enjoy, consider the inferential distances separating modern advances from even the most intelligent layperson. If your ultimate goal is the production of mathematics, or in fact any temporally altruistic objective, then consider alternatives which may increase the future's capacity to do mathematics and which may be orders of magnitude more effective.
Argument 7: What else would I do to make a living? Research provides at least some benefit to society; alternatives seem even worse.
Rebuttal: My past self, at least, was guilty of motivated stopping. See argument 4.