Qiaochu_Yuan comments on By Which It May Be Judged - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (934)
A Type 2 FAI gets its notion of what morality is based on properties of the physical universe, namely properties of humans in the physical universe. But even if counterfactually there were no humans in the physical universe, or even if counterfactually Omega modified the contents of all human brains in the physical universe so that they optimize for paperclips, that wouldn't change what actual-me means when actual-me says "I want an FAI to behave morally" even if it might change what counterfactual-me means when counterfactual-me says that.