wedrifid comments on Stupid Questions Open Thread - Less Wrong

42 Post author: Costanza 29 December 2011 11:23PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (265)

You are viewing a single comment's thread. Show more comments above.

Comment author: wedrifid 31 December 2011 07:04:36AM 11 points [-]

I understand CEV. What I don't understand is why the programmers would ask the AI for humanity's CEV, rather than just their own CEV.

The only (sane) reason is for signalling - it's hard to create FAI<self> without someone else stopping you. Given a choice, however, CEV<self> is strictly superior. If you actually do want to have FAI<humanity> then FAI<self> will be equivalent to it. But if you just think you want FAI<humanity> but it turns out that, for example, FAI<humanity> gets dominated by jerks in a way you didn't expect then FAI<self> will end up better than FAI<humanity>... even from a purely altruistic perspective.