Mitchell_Porter comments on Hacking the CEV for Fun and Profit - Less Wrong

52 Post author: Wei_Dai 03 June 2010 08:30PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (194)

You are viewing a single comment's thread. Show more comments above.

Comment author: Mitchell_Porter 09 June 2010 12:24:29AM 2 points [-]

If the FAI designers are allowed to just pick any prototype they want, they can make the CEV of "humanity" come out however they wish, so they might as well have the FAI use the CEV of themselves. If they pick the prototype by taking the average of all existing humans, then that allows the same attack described in my post.

Who ever said that CEV is about taking the average utility of all existing humans? The method of aggregating personal utilities should be determined by the extrapolation, on the basis of human cognitive architecture, and not by programmer fiat.