It's fine to include some low-probability catastrophe risk management in your overall planning. But are you considering all the possible catastrophes, or just one particular route to unfriendly AI (one unlocked by your marginal recruitment of mathematically capable tinkerers)?
Wouldn't furthering our mathematical and technological prowess as soon as possible mitigate many catastrophes? See the movie Armageddon, for instance :)
Maybe general AI is inevitable even at current computing power, so long as a small, persistent cult keeps at it for a few hundred years. If so, I think having more mathematical facility gives a better chance of managing the result.
Real, all-of-the-sudden, self-optimizing with increasing speed, with limits way above human, general AI is 99.999% not implemented in the next 10 years, at least. The only reason I consider it so likely (and don't feel comfortable predicting, say, 50 years forward) is the possibility of apparent limits in computing hardware being demolished by some unforeseen breakthrough.
When I read this paper, the risks seem to be on balance increased rather than decreased by greater human intelligence.
The median LWer's guesses on when the singularity will occur is 2067.
Improving math education is a problem I'd really like to work on but it seems likely to be harmful unless I can include an effective anti-existential-risk disclaimer. Even if I'm guaranteed to be relatively unsuccessful, I don't want a big part of my life's work to be devoted to marginally increasing the probability that something really bad will happen.
Here's our place to discuss Less Wrong topics that have not appeared in recent posts. Have fun building smaller brains inside of your brains (or not, as you please).