Eliezer_Yudkowsky comments on Two questions about CEV that worry me - Less Wrong

29 Post author: cousin_it 23 December 2010 03:58PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (137)

You are viewing a single comment's thread. Show more comments above.

Comment author: Eliezer_Yudkowsky 23 December 2010 09:38:39PM 14 points [-]

The intelligence to calculate the CEV needs to be pre-FOOM.

No, a complete question to which CEV is an answer needs to be pre-FOOM. All an AI needs to know about morality before it is superintelligent is (1) how to arrive at a CEV-answer by looking at things and doing calculations and (2) how to look at things without breaking them and do calculations without breaking everything else.

Comment author: David_Gerard 24 December 2010 10:53:07AM 2 points [-]

Ah, OK. So do we have any leads on how to ask the question?