The idea of moral demandingness is really a separate issue.
I don't see what difference it makes if you are selfish or not regarding friendly AI research. I believe that being altruistic is largely instrumental in maximizing egoistic satisfactions.
Thanks for the post by Nick Bostrom. But it only adds to the general confusion.
There seem to be dramatic problems with both probability-utility calculations and moral decision making. Taking those problems into account makes it feel like one could as well throw a coin to decide.
Michael Anissimov recently wrote:
For instance, you must have made decisions for your children that were more in alignment with what they would want if they were smarter. If you made judgments in alignment with their actual preferences (like wanting to eat candy all day — I don’t know your kids but I know a lot of kids would do this), they would suffer for it in the longer term.
This sounds good but seems to lead to dramatic problems. In the end it is merely an appeal to intuition without any substance.
If you don't try to satisfy your actual preferences, what else?
In the example, stated by Anissimov, what actually happens is that parents try to satisfy their own preferences by not allowing their children to die of candy-intoxication.
If we were going to disregard our current preferences and postpone having fun in favor of gathering more knowledge then we would eventually end up as perfectly rational agents in static game theoretic equilibria.
The problem with the whole utility-maximization heuristic is that it eventually does deprive us of human nature by reducing our complex values to mere game theoretic models.
Part of human nature, what we value, is the way we like to decide. It won't work to just point at hyperbolic discounting and say it is time-inconsistent and therefore irrational.
Human preferences are always actual, we do not naturally divide decisions according to instrumental and terminal goals.
I don't want a paperclip maximizer to burn the cosmic commons. I don't want to devote most of my life to mitigate that risk. This is not a binary decision, that is not how human nature seems to work.
If you try to force people into a binary decision between their actual preferences and some idealistic far mode then you cause them to act according to academic considerations rather than their complex human values they are supposed protect.
Suppose you want to eat candies all day and are told that you can eat a lot more candies after the Singularity, if only you work hard enough right now. The problem is that there is always another Singularity that promises more candies. At what point are you going to actually eat candies? But that is a rather academic problem. There is a more important problem concerning human nature, as demonstrated by extreme sport. Humans care much more about living their life's according to their urges than about maximizing utility.
What does it even mean to "maximize utility"? Many sportsmen and sportswomen are aware of the risks associated with their favorite activity. Yet they take the risk.
It seems that humans are able to assign infinite utility to pursuing a certain near-mode activity.
Deliberately risking your life doesn't seem to be maximizing experience utility, as you would be able to experience a lot more of the same or similar experience differently. And how can one "maximize" terminal decision utility?
When applying your objections to my own perspective, I find that I see my actions that aren't focused on reducing involuntary death (eating candies, playing video games, sleeping) as necessary for the actual pursuit of my larger goals.
I am a vastly inefficient engine. My productive power goes to the future, but much of it bleeds away - not as heat and friction, but as sleep and candy-eating. Those things are necessary for the engine to run, but they aren't necessary evils. I need to do them to be happy, because a happy engine is an efficient one.
I recogniz...
Link: johncarlosbaez.wordpress.com/2011/04/24/what-to-do/
His answer, as far as I can tell, seems to be that his Azimuth Project does trump the possibility of working directly on friendly AI or to support it indirectly by making and contributing money.
It seems that he and other people who understand all the arguments in favor of friendly AI and yet decide to ignore it, or disregard it as unfeasible, are rationalizing.
I myself took a different route, I was rather trying to prove to myself that the whole idea of AI going FOOM is somehow flawed rather than trying to come up with justifications for why it would be better to work on something else.
I still have some doubts though. Is it really enough to observe that the arguments in favor of AI going FOOM are logically valid? When should one disregard tiny probabilities of vast utilities and wait for empirical evidence? Yet I think that compared to the alternatives the arguments in favor of friendly AI are water-tight.
The problem why I and other people seem to be reluctant to accept that it is rational to support friendly AI research is that the consequences are unbearable. Robin Hanson recently described the problem:
I believe that people like me feel that to fully accept the importance of friendly AI research would deprive us of the things we value and need.
I feel that I wouldn't be able to justify what I value on the grounds of needing such things. It feels like that I could and should overcome everything that isn't either directly contributing to FAI research or that helps me to earn more money that I could contribute.
Some of us value and need things that consume a lot of time...that's the problem.