amcknight comments on Thoughts and problems with Eliezer's measure of optimization power - Less Wrong

17 Post author: Stuart_Armstrong 08 June 2012 09:44AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (23)

You are viewing a single comment's thread.

Comment author: amcknight 08 June 2012 09:03:27PM 0 points [-]

It turns out we can't. Imagine that, without changing anything else, the utility of X7 is suddenly set to ten trillion, rather than 7. The OP of X7 is still 3 - it's still the best option, still with probability 1/8.

I think a problem with this line of attack is that you are mixing preferences and utilities. You could imagine two types of optimization power. A preference-centric one and a utility-centric one, both of which can be useful depending what you're talking about. You can map preferences to utilities and utilities to preferences, but one may be more natural than the other for your purposes.