You're looking at Less Wrong's discussion board. This includes all posts, including those that haven't been promoted to the front page yet. For more information, see About Less Wrong.

wedrifid comments on Stupid Questions Open Thread - Less Wrong Discussion

42 Post author: Costanza 29 December 2011 11:23PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (265)

You are viewing a single comment's thread. Show more comments above.

Comment author: wedrifid 31 December 2011 07:04:36AM 11 points [-]

I understand CEV. What I don't understand is why the programmers would ask the AI for humanity's CEV, rather than just their own CEV.

The only (sane) reason is for signalling - it's hard to create FAI<self> without someone else stopping you. Given a choice, however, CEV<self> is strictly superior. If you actually do want to have FAI<humanity> then FAI<self> will be equivalent to it. But if you just think you want FAI<humanity> but it turns out that, for example, FAI<humanity> gets dominated by jerks in a way you didn't expect then FAI<self> will end up better than FAI<humanity>... even from a purely altruistic perspective.