I believe that people like me feel that to fully accept the importance of friendly AI research would deprive us of the things we value and need.
The idea of moral demandingness is really a separate issue. The ability to save thousands of lives through donation to 3rd world public health poses a similar problem for many people. Risks like nuclear war (plus nuclear winter/irrecoverable social collapse), engineered pandemics, and other non-AI existential risks give many orders of magnitude increased stakes (within views that weight future people additively). AI may give some additional orders of magnitude, but in terms of demandingness it's not different in kind.
Most people who care a lot about 3rd world public health, e.g. Giving What We Can members, are not cutting out their other projects, and come to stable accommodations between their various concerns, including helping others, family, reading books, various forms of self-expression, and so forth.
If you have desires pulling you towards X and Y, just cut a deal between the components of your psychology and do both X and Y well, rather than feeling conflicted and doing neither well. See this Bostrom post, or this Yudkowsky one on the virtues of divvying up specialized effort into separate buckets.
The idea of moral demandingness is really a separate issue.
I don't see what difference it makes if you are selfish or not regarding friendly AI research. I believe that being altruistic is largely instrumental in maximizing egoistic satisfactions.
Thanks for the post by Nick Bostrom. But it only adds to the general confusion.
There seem to be dramatic problems with both probability-utility calculations and moral decision making. Taking those problems into account makes it feel like one could as well throw a coin to decide.
Michael Anissimov recently wrote...
Link: johncarlosbaez.wordpress.com/2011/04/24/what-to-do/
His answer, as far as I can tell, seems to be that his Azimuth Project does trump the possibility of working directly on friendly AI or to support it indirectly by making and contributing money.
It seems that he and other people who understand all the arguments in favor of friendly AI and yet decide to ignore it, or disregard it as unfeasible, are rationalizing.
I myself took a different route, I was rather trying to prove to myself that the whole idea of AI going FOOM is somehow flawed rather than trying to come up with justifications for why it would be better to work on something else.
I still have some doubts though. Is it really enough to observe that the arguments in favor of AI going FOOM are logically valid? When should one disregard tiny probabilities of vast utilities and wait for empirical evidence? Yet I think that compared to the alternatives the arguments in favor of friendly AI are water-tight.
The problem why I and other people seem to be reluctant to accept that it is rational to support friendly AI research is that the consequences are unbearable. Robin Hanson recently described the problem:
I believe that people like me feel that to fully accept the importance of friendly AI research would deprive us of the things we value and need.
I feel that I wouldn't be able to justify what I value on the grounds of needing such things. It feels like that I could and should overcome everything that isn't either directly contributing to FAI research or that helps me to earn more money that I could contribute.
Some of us value and need things that consume a lot of time...that's the problem.