wedrifid comments on Two questions about CEV that worry me - Less Wrong

29 Post author: cousin_it 23 December 2010 03:58PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Article Navigation

Comments (137)

Comments (137)

You are viewing a single comment's thread. Show more comments above.

Comment author: wedrifid 23 December 2010 05:08:39PM 0 points [-]

I'm still wondering how you'd calculate a CEV. I'm still wondering how you'd calculate one human's volition.

The implication is that you let the FAI do it. (The task in building the FAI so that it will do this has somewhat different challenges.)

Comment author: David_Gerard 23 December 2010 05:43:32PM 4 points [-]

Yes, but saying "we get the FAI to do it" is just moving the hard bit.

The intelligence to calculate the CEV needs to be pre-FOOM. We have general intelligences of pre-FOOM level already.

So: what would a pre-FOOM general intelligence actually do?

Comment author: Eliezer_Yudkowsky 23 December 2010 09:38:39PM 14 points [-]

The intelligence to calculate the CEV needs to be pre-FOOM.

No, a complete question to which CEV is an answer needs to be pre-FOOM. All an AI needs to know about morality before it is superintelligent is (1) how to arrive at a CEV-answer by looking at things and doing calculations and (2) how to look at things without breaking them and do calculations without breaking everything else.

Comment author: David_Gerard 24 December 2010 10:53:07AM 2 points [-]

Ah, OK. So do we have any leads on how to ask the question?

Comment author: Eugine_Nier 23 December 2010 09:17:28PM 2 points [-]

I believe the idea is the have the pre-FOOM AI commit to doing the calculation first thing post-FOOM.

Comment author: wedrifid 24 December 2010 02:31:41AM 0 points [-]

Eliezer beat me to the punch. My answer was approximately the same.

Powered by Reddit Powered by Reddit