What I meant is that if I have a choice either to do something that has a very tiny chance of having a very large good effect (e.g., working on friendly AI and possibly preventing a hostile takeover of the world by nasty AI) or to do something with a high chance of having a small good effect (e.g., teaching math to university students), I may take the latter option where others may take the former. Neither need be irrational.
Problem is that the expected utility of an outcome often grows faster than its probability shrinks. If the utility you assign to a galactic civilization does not outweigh the low probability of success you can just take into the account all beings that could be alive until the end of time in the case of a positive Singularity. Whatever you care about now, there will be so much more of it after a positive Singularity that it does always outweigh the tiny probability of it happening.
You raise this issue a lot, so now I'm curious how you cash it out in actual numbers:
For the sake of concreteness, call Va the value to you of 10% of your total assets at this moment. (In other words, going from your current net worth to 90% of your net worth involves a loss of Va.)
What's your estimate of the value of a positive Singularity in terms of Va?
What's your estimate of the probability (P%) of a positive Singularity at this moment?
What's your estimate of how much P% would increment by if you invested an additional 10% of your total assets at this moment in the most efficient available mechanism for incrementing P%?
John Baez's This Week's Finds (Week 311) [Part 1; added for convenience following Nancy Lebovitz's comment]
John Baez's This Week's Finds (Week 312)
John Baez's This Week's Finds (Week 313)
I really like Eliezer's response to John Baez's last question in Week 313 about environmentalism vs. AI risks. I think it satisfactorily deflects much of the concern that I had when I wrote The Importance of Self-Doubt.
Eliezer says
This is true as stated but ignores an important issue which is there is feedback between more mundane current events and the eventual potential extinction of the humane race. For example, the United States' involvement in Libya has a (small) influence on existential risk (I don't have an opinion as to what sort). Any impact on human society impact due to global warming has some influence on existential risk.
Eliezer's points about comparative advantage and of existential risk in principle dominating all other considerations are valid, important, and well-made, but passing from principle to practice is very murky in the complex human world that we live in.
Note also the points that I make in Friendly AI Research and Taskification.