Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Mass_Driver comments on The Importance of Goodhart's Law - Less Wrong

75 Post author: blogospheroid 13 March 2010 08:19AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (113)

You are viewing a single comment's thread. Show more comments above.

Comment author: Mass_Driver 05 April 2010 02:01:30PM 2 points [-]

If the question is "what should we want?" then CEV is much better than a black box, because it fleshes out some of the intuitions behind the magical category "want."

If the question is "how should we measure what we want?" then CEV is just a black box, because it doesn't solve or suggest a method for solving any of our measurement problems. We know we want coherent, extrapolated, volitional thingies, but we have no idea, for example, how to rigorously define "volitional." We likewise have no idea how far into the future we should be extrapolating things, nor how many facets of a personality or society can reasonably be expected to converge/cohere.