Eliezer_Yudkowsky comments on Moral Error and Moral Disagreement - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (125)
Heck, what about babies? What do they want, and would they be complicated enough to want anything different if they knew more and thought faster?
You would not believe exactly the same thing. If you held one of the others, you would believe that your new system was frooter than any of the others, where "frooter" is not at all the same thing as "better". And you would be correct.
If you make matters that complicated to begin with, i.e., we're not discussing metaethics for human usage anymore, then you should construe entailment / extrapolation / unfolding in more robust ways than "anything a superintelligence can convince you of". E.g. CEV describes a form of entailment.
As for what a person's morality is, surely you extrapolate it at least a little beyond their instantaneous beliefs. Would you agree that many people would morally disapprove of being shot by you, even if the actual thought has never crossed their mind and they don't know you exist?