Qiaochu_Yuan comments on By Which It May Be Judged - Less Wrong

35 Post author: Eliezer_Yudkowsky 10 December 2012 04:26AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (934)

You are viewing a single comment's thread. Show more comments above.

Comment author: Qiaochu_Yuan 22 December 2012 10:31:48AM *  3 points [-]

A Type 2 FAI gets its notion of what morality is based on properties of the physical universe, namely properties of humans in the physical universe. But even if counterfactually there were no humans in the physical universe, or even if counterfactually Omega modified the contents of all human brains in the physical universe so that they optimize for paperclips, that wouldn't change what actual-me means when actual-me says "I want an FAI to behave morally" even if it might change what counterfactual-me means when counterfactual-me says that.