Wei_Dai comments on The Urgent Meta-Ethics of Friendly Artificial Intelligence - Less Wrong

45 Post author: lukeprog 01 February 2011 02:15PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (249)

You are viewing a single comment's thread. Show more comments above.

Comment author: TheOtherDave 03 February 2011 04:02:02AM 1 point [-]

Depends on what would satisfy us, I suppose.

I mean, for example, if it turns out that implementing CEV creates a future that everyone living in desires and is made happy and fulfilled and satisfied by and continues to do so indefinitely, and that everyone living now would if informed of the details of also desire and etc., but we are never able to confirm that any of that is right... or worse yet, later philosophical analysis somehow reveals that it isn't right, despite being desirable and fulfilling and satisfying and so forth... well, OK, we can decide at that time whether we want to give up what is desirable and etc. in exchange for what is right, but in the meantime I might well be satisfied by that result. Maybe it's OK to leave future generations some important tasks to implement.

Or, if it turns out that EY's approach is all wrong because nobody agrees on anything important to anyone, so that extrapolating humanity's coherent volition leaves out everything that's important to everyone, so that implementing it doesn't do anything important... in that case, coming up with an alternate plan that has results as above would satisfy me.

Etc.

Comment author: Wei_Dai 03 February 2011 06:20:33PM *  0 points [-]

OK, we can decide at that time whether we want to give up what is desirable and etc. in exchange for what is right, but in the meantime I might well be satisfied by that result

No, once ostensibly-Friendly AI has run CEV and knows what it wants, it won't matter if we eventually realize that CEV was wrong after all. The OFAI will go on to do what CEV says it should do, and we won't have a say in the matter.

Comment author: TheOtherDave 03 February 2011 06:30:04PM 2 points [-]

Agreed: avoiding irreversible steps is desirable.