you addressed the classic example of non-Euclidean geometry paving the way for general relativity by saying that Einstein could have just invented non-Euclidean geometry on his own (!) when he needed it.
My claim was that Einstein explicitly considered gravity as curvature of space before being aware of any formal developments in non-Euclidean geometry. I inferred that, if Einstein had lived before any formal developments in non-Euclidean geometry occurred, then non-Euclidean geometry would have become an object of study not because it was interesting but because it was important. If you really want to have this argument, then you should describe where you think things would have broken down if non-Euclidean geometry had not already been developed. Would general relativity never have occurred to Einstein? Would the community have scoffed at the idea more than they did? Would the theory have stagnated before the math could catch up?
But lets count this as a point for the home team. Lets suppose that general relativity had been set back a hundred years, and that we had never considered the possibility of gravitational time dilation until we discovered the empirical formula for the time dilation experienced by satellites. I agree that this is worse than what really happened, but its not bad enough to convince me that I should support work on interesting problems. I would be interested in additional examples, even if they were much less compelling.
You've characterized my position as motivated cognition in defense of the status quo, but in my view that is both wrong and unfair
I don't really care whether your position is motivated cognition. I realized my own beliefs were motivated cognition, and now I would like to replace them with beliefs that better reflect reality.
you think you have a good way of predicting the long-term impact of mathematical work,
We are both trying to predict the long-term impact of human activities. You don't get to abstain from having beliefs. My claim is that concretely motivated problems are a significantly better use of the smartest researchers' time than interesting abstract problems; your claim is that they aren't. I came to my belief by observing the historical record, noting that important advances have at least been proximately due to smart people working on concrete problems, and suspecting that the causal connections drawn to more abstract problems are highly tenuous. If you think this reasoning is the explanation for the rate of human progress prior to the 17th century, then you should say so and I can describe at length why I believe you are almost certainly wrong.
For the past few days I've been pondering the question of how best to respond to paulfchristiano's recent posts and comments questioning the value of mathematical research. I don't think I can do it concisely, in a single post; bridging the inferential distance may require something more like a sequence of posts. I may end up writing such a sequence eventually, since it would involve ideas I've actually been wanting to write up for some time, and which are actually relevant to more than just the specific questions at issue here (whether society should sponsor mathematics, and given that it does, whether paulfchristiano or anyone else in the LW readership should pursue it).
However, as the preceding parenthetical hints at, I'm actually somewhat conflicted about whether I should even bother. Although I believe that mathematical research should be conducted by somebody, it's not at all clear to me that the discipline needs more people beyond those who already "get" its importance, and are out there doing it rather than writing skeptical posts like paulfchristiano's. It seems perfectly plausible to me that those who feel as paulfchristiano does should just leave the profession and do something else that feels more "important" to them. This is surely the best practical solution on an individual level for those who think they have a better idea than existing institutions of where the most promising research directions lie, at least until Hansonian prediction markets are (ever) implemented.
Nevertheless, for those interested in the society-level question of whether mathematics (as such) may be justifiably pursued by anyone, or any community of people, as a professional occupation (which is quite distinct from the question of whether e.g. paulfchristiano should personally pursue it), I recommend, at least as a start, grappling with the arguments put forward by the best mathematicians in their own words. I think this essay by Timothy Gowers (a Fields Medalist), titled "The Importance of Mathematics", is a good place to begin. I would particularly draw the attention of those like paulfchristiano, who think they have a good idea of which branches of mathematics are useful and which aren't, to the following passage, from pp.8-9 (unfortunately the illustrations are missing, but the point being made is pretty clear nonetheless):