By the way, I think nihilism often gets short changed around here. Given that we do not actually have at hand a solution to ontological crises in general or to the specific crisis that we face, what's wrong with saying that the solution set may just be null? Given that evolution doesn't constitute a particularly benevolent and farsighted designer, perhaps we may not be able to do much better than that poor spare-change collecting robot? If Eliezer is worried that actual AIs facing actual ontological crises could do worse than just crash, should we be very sanguine that for humans everything must "add up to moral normality"?

I really like the robot metaphor, and I fully agree with the kind of nihilism you describe here. Let me note, though, that nihilism is a technically precise but potentially misleading name for this world view. I am an old-fashioned secular humanist when it comes to 2012 humans. I am a moral nihilist only when I have to consider the plethora of paradoxes that come with the crazy singularity stuff we like to discuss here (most significantly, substrate-independence). Carbon-based 2012 humans already face some uncomfortable edge cases (e.g. euthanasia, abortion, animal rights), but with some introspection and bargaining we can and should agree on some ground rules. I am a big fan of such ground rules, that's why I call myself an old-fashioned humanist. On the other hand, I think our morality simply does not survive the collision with your "ontological crisis". After the ontological crisis forces itself on us, it is a brand new world, and it becomes meaningless to ask what we ought to do in this new world. I am aware that this is an aesthetically deeply unsatisfying philosophical position, so I wouldn't accept it if I had some more promising alternatives available.