Mitchell_Porter comments on Hacking the CEV for Fun and Profit - Less Wrong

52 Post author: Wei_Dai 03 June 2010 08:30PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (194)

You are viewing a single comment's thread. Show more comments above.

Comment author: Wei_Dai 09 June 2010 12:14:23AM 9 points [-]

Why would CEV operate on humans that do exist, and not on humans that could exist?

To do the latter, you would need a definition of "human" that can not just distinguish existing humans from existing non-humans, but also pick out all human minds from the space of all possible minds. I don't see how to specify this definition. (Is this problem not obvious to everyone else?)

For example, we might specify a prototypical human mind, and say that "human" is any mind which is less than a certain distance from the prototypical mind in design space. But then the CEV of this "humankind" is entirely dependent on the prototype that we pick. If the FAI designers are allowed to just pick any prototype they want, they can make the CEV of "humanity" come out however they wish, so they might as well have the FAI use the CEV of themselves. If they pick the prototype by taking the average of all existing humans, then that allows the same attack described in my post.

Comment author: Mitchell_Porter 09 June 2010 12:24:29AM 2 points [-]

If the FAI designers are allowed to just pick any prototype they want, they can make the CEV of "humanity" come out however they wish, so they might as well have the FAI use the CEV of themselves. If they pick the prototype by taking the average of all existing humans, then that allows the same attack described in my post.

Who ever said that CEV is about taking the average utility of all existing humans? The method of aggregating personal utilities should be determined by the extrapolation, on the basis of human cognitive architecture, and not by programmer fiat.