Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

botogol comments on The Importance of Goodhart's Law - Less Wrong

75 Post author: blogospheroid 13 March 2010 08:19AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (113)

You are viewing a single comment's thread.

Comment author: botogol 16 March 2010 04:28:39PM 4 points [-]

At work a large part of my job involves choosing G* , and I can report that Goodhart's Law is very powerful and readily observable.
Further : rational players in the workspace know full-well that management desire G, and the G* is not well-correlated with G, but nonethelss if they are rewarded on G*, then that's what they will focus on.

The best solution - in my experience - is mentioned in the post: the balanced scorecard. Define several measures G1* G2* G3* and G4* that are normally correlated with G. The correlation is then more persistent : if all four measures improve it is likely that G will improve.

G1* G2* G3* G4* may be presented as simulaneous measures, or if setting four measures in one go is too confusing for people trying to prioritise (the frwer the measures the more powerful) they can be sequential. IE If you hope to improve G over 2 years, then measure G1* for two quarters, then switch the measurement to G2* for the next two and so on. (obviously you don't tell people in advance). NB this approach can eb effective, but will make you very unpopular.

Comment author: Christian_Szegedy 27 May 2010 07:05:34PM *  0 points [-]

measure G1* for two quarters, then switch the measurement to G2* for the next two and so on. (obviously you don't tell people in advance).

Why obviously? Are you so afraid that people would do the right thing without immediate incentives?

I think I'd measure G1 first, but would tell in advance that next quarter we will measure that one of G1,G2,G3,... which will be most critical at the beginning of that quarter.