There are no generic solutions to bridging the gap between G and G*, but the body of knowledge of theory of constraints is a very good starting point for formulating better measures for corporates.
A good example from my own history of doing this is when I worked for an ISP and persuaded them to eliminate "cases closed" as a performance measurement for customer service and tech support people, because it was causing email-based cases to be closed without any actual investigation. People would email back and create a new case, and then a rep would get credit for closing that one without investigation either.
The replacement metric was one I derived via the Theory of Constraints, inspired by Goldratt's "throughput-dollar-days" measurement. The replacement metric was "customer-satisfaction-waiting-hours" - a measurement of collective work-in-progress inventory at the team level, and a measurement of priority at the ticket level.
I also made it impossible to truly "close" a case - you could say, "I think this is done", but the customer could still email into it and it would jump right back to its old place in the queue, due to the ac...
I am reminded of one of Dijkstra's sayings:
To this very day we have organizations that measure "programmer productivity" by the "number of lines of code produced per month"; this number can, indeed, be counted, but they are booking it on the wrong side of the ledger, for we should talk about "the number of lines of code spent".
So, in short: incentives can have unintented consequences, as the incentives influence whatever you want to influence with them.
There are a lot of examples of this in e.g. Dan Ariely's book and Freakonomics.
But the best example must be the bizarre 1994 footbal (soccer) match between Barbados and Grenada. Barbados needed to win with a two goal difference.
The special incentive here was that any goal scored in the extra time would count double. Now, shortly before the end of the regular time, it was 2-1 for Barbados. Imagine what happened...
(edit: added the note about the two-goal difference, thanks Hook)
It's an important note for the soccer game that Barbados needed to win by two points in order to advance to the finals. Otherwise, Grenada would go to the finals. Now people have a chance of imagining what happened.
Goodhart's Law starts some other way. It's not quite right to say:
Superiors want an undefined goal G.
Mathematically speaking, the problem can't be that G is undefined. If G were really undefined in any absolute sense, then superiors would be indifferent to all possible outcomes, or would choose their utility function literally at random. That rarely happens.
Instead, the problem could be that G is difficult to articulate. It is "undefined" only in the sense that people have had trouble coming up with an explicit verbal definition for it. i know what I want and how to get it, but I don't know how to communicate that want to you ex ante. For example, maybe I want you (the night shift manager) to page me (the owner) whenever there's a decision to make that could affect whether our business keeps a client, but I've never taken any business classes and don't quite have the vocab to say that, so instead I say to only page me if it's "important." "Important" is vague, but "important' is just a map, and the map is not the territory.
Alternatively, the problem could be that G is difficult to commit to. I can define my goal in words just fine toda...
CEV, until designed and defined properly, is just a black box that everyone universally agrees is 'good', but has little else in term of defining features.
The fact that students who are motivated to get good scores in exams very often get better scores than students who are genuinely interested in the subject is probably also an application of Goodhart's Law?
Partially; but a lot of what is being tested is actually skills correlated with being good in exams - working hard, memorisation, bending youself to the rules, ability to learn skill sets even if you don't love them, gaming the system - rather than interest in the subject.
But those skills don't correlate with doing good science, or with good use of the subject of the exams in general, nearly so well, and they are easy to test in other ways.
Pretty much every trick in organization design or management can be thought of as a partial solution to this problem. Listing "anarchy" or "absolute authority" explicitly on a short list of solutions is therefore a bit misleading.
Goodhart's law seems very applicable to natural selection: the Blind Idiot God wants creatures to have higher fitness (G), and so creates targets that are correlated with fitness in the ancestral habitat (e.g., pleasure-seeking and pain-avoidance (G*)). Once you get creatures that are self-aware (us), they figure out G-star, and start optimizing for that instead of G.
Example:
The Big Mac Index has been used to compare prices across countries, as we have noted before. Argentina currently has very high prices due to a combination of inflation and a strong economy, and this shows up glaringly in the Big Mac Index.
Tyler Cowen reports (translating a Spanish original) that the Argentinian government has persuaded McDonalds to lower the price of the Big Mac (relative to other McDonalds items, and relative to competing hamburgers), so that Brazil’s Big Mac Index becomes more competitive.
http://www.statschat.org.nz/2011/11/16/goodharts-law-and-brazilian-hamburgers/
In other words, the real price of the Big Mac rose nearly twice as much as the official statistics were willing to admit, in Argentina of course. That’s not right, so the government sprang into action. The minister of the commerce department “persuaded” McDonald’s to price the Big Mac at $16, while other sandwiches at the chain are in the $21 to $23 range.
The outlets now keep the Big Mac well-hidden
http://marginalrevolution.com/marginalrevolution/2011/11/sentences-to-ponder-32.html
Getting back to trying to propose practical mitigation strategies for goodhart's law, I propose a fairly simple solution: Choose a G*, evaluate performance based on it, but KEEP IT SECRET. This of course wouldn't really work for national scale, GDP-esque kind of situations, but for corporate management situations it seems like it could work well enough. If only upper management knows what G* is, it becomes impossible to optimize for it, and everyone has to just keep working under the assumption they're being evaluated on G.
Taking it a step further, to hedge against employees eventually figuring out G* and surreptitiously optimizing for it, you could have a bounty on guessing G* - the first employee who figures out what the mystery metric G* really is gets a prize, and as soon as it's claimed, you switch to using G**
The hedge is absolutely necessary, elsewise, a manager will just tell subordinates what G* is in order to look impressive for managing a high-performing group.
At work a large part of my job involves choosing G , and I can report that Goodhart's Law is very powerful and readily observable.
Further : rational players in the workspace know full-well that management desire G, and the G is not well-correlated with G, but nonethelss if they are rewarded on G*, then that's what they will focus on.
The best solution - in my experience - is mentioned in the post: the balanced scorecard. Define several measures G1 G2 G3 and G4 that are normally correlated with G. The correlation is then more persistent : if all four me...
Goodhart's Law is a very nice corollary to the Snafu Principle: Communication is impossible in a hierarchy.
Temple Grandin has written about the importance of finding relevant, measurable standards-- the example she gives is the number of cattle falling down on the way to slaughter. Not falling down means the genes, food, lighting, walking surface etc. are all good enough.
Thing to check: Do measures used as targets for policy always become completely useless, or do they sometimes become increasingly less useful, but not totally useless? Does culture matter...
We laugh at such ridiculous stories, because our societies are generally much better run than Soviet Russia.
That's not even true, when you measure results Soviet Union was ran about as well as any other country. They were just ran differently.
Anyway, it doesn't even seem mathematically obvious to me that optimizing for G* will reduce correlation between G and G*.
Let G=market value of nails, G*=number of nails. Once G* target is introduced everyone switches from making medium nails to making tiny nails, but correlation between market value of nails and n...
blogospheroid: When you pick a metric of success, countries will game it by doing well on that metric, yet not achieving what is really meant by success. A good example would be the Soviet Union, whose leaders constantly made sure they did well by the metrics, yet were actually far from successful.
taw: Not true -- the Soviet Union did about average by those metrics, so they had about average success.
me: *falls out of chair*
Key question: are the metrics optimized by the Soviet Union identical to the metrics suggested to evaluate the success of the Soviet Union?
You misread me. GDP is one of those really-hard-to-game high-correlation-with-everything-meaningful metrics, and Soviet Union did ok with other metrics like access to clean water, electricity etc.; life expectancy, child mortality, and pretty much everything else you can think of.
People's claim that Soviet Union was a disaster as if it was a well established fact, while it was not. South America was a disaster. India was a disaster. Indonesia was a disaster. Africa was a disaster. Soviet Union and other Communist countries were fairly average.
What you're saying is basically "Soviet Union was unsuccessful and I base it on my feelings about it and no metrics of any kind".
The Soviet Union's GDP was approximately half military spending. In other words, at least half of Soviet GDP was an almost complete dead-weight loss to the citizens. GDP is just an aggregation of total spending, if money is spent on pure dross, it still shows up as an improvement in GDP.
The Soviet Union killed tens of millions of its people through work camps, starvation and purges. How is that not a disaster?
And GDP is not hard to game if you're centrally planning the economy. And even that excludes simply lying about your figures.
The Soviet Union killed tens of millions of its people through work camps, starvation and purges. How is that not a disaster?
This is a really good point. It kind of goes to the heart of the emptiness of GDP as a statistic. What was the death toll of a 5% increase in GDP? The Soviet economy also was infamous for overproducing big capital goods but failing to produce consumer goods people actually wanted to buy.
Similarly, Ceauşescu supposedly managed to clear Romania's national debt, but, in doing so, impoverished the people and crushed their spirit. The socio-cultural damage he did to the country cannot be overstated. (btw I'm half-Romanian and lived there during his reign).
When discussing the Soviet Union, and more specifically Russia, you have to also consider the beginning point as well. It should be noted how far behind Russia was compared to the rest of Europe in 1918. Coming out of abject serfdom bordering on generalized slavery, they actually made tremendous progress in both abstract metrics and tangible result in quality of life up until the late 50's or early 60's. Over time that then declined.
In any case, to an extent their G was "production", measurable production, in the sample case: nails. Their G was not the market value of nails, their G was "progress through central planning", but they didn't know how to measure "progress" except through the early-capitalist metrics of "production". Thus: produce more = progress, in their practice.
Our G is GDP. People seem so happy with our GDP, without reflecting on things like income disparity, striation of wealth, etc. If we allow it, we can G* ourselves into a mutant 3rd world nation, with great GDP performance but declining quality of life generally. G is quality of life. Economists, and lay people, generally equate the two, and the correlate generally, but they are not irrevocably entangled.
Most people on this supposedly rationalist site don't even bother looking at the data when it comes to Soviet Union - they get instant emotional reaction.
How exactly have you determined the instant emotional reaction of most of the people on this site in response to the Soviet Union? I haven't seen most people even comment on the subject, much less display obvious evidence emotional involvement.
Did you actually think through your estimates of soviet-emotionalism in the population or is this a case of "the pot calling a non representative sample of kettles black"?
1) Examples of G* should be given a cost-benefit analysis. Yeah, scammers and parasites exist, but societies that use money still seem to better off than societies that try to get rid of it.
2) It's unclear to me why you list CEV as one of the solutions. We use money to allocate limited resources. If magic nano-AI appears and resources become unlimited, why keep score at all? If it doesn't and resources stay limited, how does CEV help you distribute bread, and would you really like it to replace money? (I wouldn't. No caring daddies for me, please.)
In the case of a FAI G would be friendliness and G the friendliness definition. Avoiding a Goodhart's Law effect on G is pretty much the core of the friendliness problem in a nutshell. An example of such a Goodhart's Law effect would be the molecular smiley faces scenario.
I noticed this tendency in British running of hospitals, schools and police forces. The gov got hooked on the idea of targets and not on medicine, education and public order.
I wish there were some examples (other than the Soviet nails) ... if I had some better idea of what G and G* might actually represent, I'd be able to more easily get my head around the rest of the post.
I'm surprised no one has yet brought up (G*) the LW karma system as a proxy for (G) contributing to "refining the art of human rationality".
LW karma is an interesting example because no one has direct access to the karma giving algorithm.
It's a bit like telling the nail factory that you're going to evaluate them on something, but not telling them whether its nail mass or number or something else until the end of the evaluation period.
If the one being evaluated knows nothing about how he's going to be evaluated except that it's going to be a proxy for goodness, then he can't really cheat. However, they might know that it's going to be very simple criteria so they make a very massive nail and many miniature ones.
This reminds me of the way I hear they do state censorship in China. The censoring agencies don't actually give out any specific guidelines on what is allowed and what isn't, instead just clamping down on cases they do consider to be over the line. As a consequence, everyone self-censors more than they might with specific guidelines: with the guidelines, you could always try to twist their letter to violate their spirit. Instead, people are constantly unsure of just exactly what will get you in trouble, so they err on the side of caution.
While I strongly oppose state censorship, I can't help but admire the genius in the system.
In education, this is one of the criticisms of high-stakes testing: you'll just get schools teaching to the test, in ways that aren't correlated to real learning (the test is G*, real knowledge/learning is G). People say the same thing about the SAT and test prep - kids get into better colleges because they paid to learn tricks for answering multiple choice questions. The Wire does a great job of showing the police force's efforts to "juke the stats" (e.g. counting robberies as larcenies) so that crime statistics (G*) look better even while crime (G) is getting worse. Athletes get criticized for playing for their stats (G*), or trying to pad their stats, instead of playing to win, when the stats are supposed to be a measure of how much a player has contributed to his team's chances of winning (G). I'm not sure if it's historically accurate, but I've heard that body count (G*) was used by the US as one of the main metrics of success (G) in the Vietnam war, and as a result we ended up with a bunch of dead bodies but a misguided war.
In general, any time you measure something you care about in order to incentivize people, or to hold people accountable, or to keep track of what's going on, and the thing you measure isn't exactly the same as the thing that you care about, there's a risk of figuring out ways to improve the measurement that don't translate into improvements on the thing that you care about.
I suggest editing a "summary break" into this post to create a "continue reading" link on the frontpage. It's the 6th button from the left atop the editing interface.
Diversity of an ecosystem is a way to reduce the impact of Goodhart's law. If different universities would use very different G* for their hiring decisions it would be harder for young researchers to optimize for any particular G*.
I don't see how hierarchical rule is a solution. Hierarchy requires the people at the top of the hierarchy to give order to people at the bottom to achieve certain outcomes and measure whether those outcomes are achieved.
Often a goal set is not based on a single set of arguments justifying it, but because it is a good compromise point between multiple arguments, motivations or interest groups. For example human rights formulations don't perfectly fulfill any groups desires (utilitarians, egalitarians, deontological groups, religious motivations etc.) but are a point of overlap between their goal sets (both utilitarians and deontologists both think torture and murder are generally bad). Similarly with GDP, economic growth is a shared interest of several groups in society.
So some instances of goodhearts law may be an observation that particular sets of goals are not being perfectly fulfilled.
body of knowledge of theory of constraints is a very good starting point for formulating better measures for corporates
I've had some interest in TOC, could you please expand on how it works to get G* closer to G?
Generally I've found TOC to be some really interesting semi-scientific stuff mixed with a ton of self promotion by goldratt.
One method is to have no G*. Tell people some of the things you'll be looking at, but don't give them specific targets or tell them exactly how you will be judging G.
This is the method currently in use to assess the quality of research in departments of universities in the UK. Every department that wants to be assessed must supply certain very detailed information (e.g. for each member of staff declared as doing research, a list of their recent publications, grants held, awards received, etc.). The actual assessment is carried out behind closed doors. They...
The same issue came up many years ago in the work of H Edwards Deming in quality control.
I like this article a lot. My solution (borrowed from Nassim Taleb) would be skin in the game. Any potential outcome resulting from the actions of the agent should be also affecting the agent.
Interesting: Left Anarchist, Right Libertarian, and Distributist ideals are fundamentally the same. While Right-Libertarians pay a form of lip service to the idea of hierarchical corporate capitalism, scratch them a bit and you find they long for SV startups or farmers on the American Frontier, as presented in books like The Moon Is A Harsh Mistress - family busines...
CEV isn't even an idea yet. It's a collection of phrases stringing together undefined terms. What isn't undefined is self-contradictory or incoherent. I very much doubt that it is possible of being refined into anything coherent and consistent; it has contradictions built into its core.
I think the Lucas Critique is actually more general than Goodhart's Law. Goodhart is a specific form of Lucas!
It's true that GDP is not identical to national welfare. And you can come up with anecdotes where some welfare measure isn't fully captured by GDP (both positive and negative).
But GDP is useful, because it is very hard to game. The examples in your "fetishism" link are very weak. Unlike the nails example, where we can all agree that the factory made the wrong choice for society, it is far from clear that the GDP examples resulted in the wrong policy, even if GDP is only an approximation for welfare.
GDP is not a good example of Goodhart's Law. It's nothing at all like the (broken) correlation between inflation and unemployment, which varies widely depending on policy choices.
I'm not sure whether it's that hard to game GDP, but I am sure that it just measures the money economy. If people need to spend more on repairing damage, or on something which is useless for them, then the GDP goes up just as if they were getting more of what they want.
Example of wheel-spinning: tax law becomes more complex. People need to spend more on help with their taxes, and possibly work longer hours to afford it. More economic activity, bur are their lives better?
This article introduces Goodhart's law, provides a few examples, tries to explain an origin for the law and lists out a few general mitigations.
Goodhart's law states that once a social or economic measure is turned into a target for policy, it will lose any information content that had qualified it to play such a role in the first place. wikipedia The law was named for its developer, Charles Goodhart, a chief economic advisor to the Bank of England.
The much more famous Lucas critique is a relatively specific formulation of the same.
The most famous examples of Goodhart's law should be the soviet factories which when given targets on the basis of numbers of nails produced many tiny useless nails and when given targets on basis of weight produced a few giant nails. Numbers and weight both correlated well in a pre-central plan scenario. After they are made targets (in different times and periods), they lose that value.
We laugh at such ridiculous stories, because our societies are generally much better run than Soviet Russia. But the key with Goodhart's law is that it is applicable at every level. The japanese countryside is apparently full of constructions that are going on because constructions once started in recession era are getting to be almost impossible to stop. Our society centres around money, which is supposed to be a relatively good measure of reified human effort. But many unscruplous institutions have got rich by pursuing money in many ways that people would find extremely difficult to place as value-adding.
Recently GDP Fetishism by David henderson is another good article on how Goodhart's law is affecting societies.
The way I look at Goodhart's law is Guess the teacher's password writ large. People and instituitions try to achieve their explicitly stated targets in the easiest way possible, often obeying the letter of the law.
The way I see Goodhart's law work, or a target's utility break down, is the following.
If you consider the law to be true, solutions to Goodhart's law are an impossibility in a non-singleton scenario. So let's consider mitigations.
Pointing out what most people would have in mind as G and showing that institutions all around are not following G, but their own convoluted G*s. Hansonian cynicism is definitely the second step to mitigation in many many cases (Knowing about Goodhart's law is the first). Most people expect universities to be about education and hospitals to be about health. Pointing out that they aren't doing what they are supposed to be doing creates a huge cognitive dissonance in the thinking person.
Taking multiple factors into consideration, trying to make G* as strong and spoof-proof as possible. The Scorecard approach is mathematically, the simplest solution that strikes a mind when confronted with Goodhart's law.
There are no generic solutions to bridging the gap between G and G*, but the body of knowledge of theory of constraints is a very good starting point for formulating better measures for corporates.
CEV tries to mitigate Goodhart's law in a better way than mechanical measures by trying to create a complete map of human morality. If G is defined fully, there is no need for a G*. CEV tries to do it for all humanity, but as an example, individual extrapolated volition should be enough. The attempt is incomplete as of now, but it is promising.
Human discretion is the one thing that can presently beat Goodhart's law because the constant checking and rechecking that G and G* match. Nobody will attempt to pull off anything as weird as the large nails in such a scenario. However, this is not scalable in a strict sense because of the added testing and quality control requirements.
Left anarchist ideas about small firms and workgroups are based on the fact that hierarchy will inevitably introduce goodhart's law related problems and thus the best groups are small ones doing simple things.
On the other end of the political spectrum, Molbuggian hierarchical rule completely eliminates the mechanical aspects of the law. There is no letter of the law, its all spirit. I am supposed to take total care of my slaves and have total obedience to my master. The scalability is ensured through hierarchy.
Of all proposed solutions to the Goodhart's law problem confronted, I like CEV the most, but that is probably a reflection on me more than anything, wanting a relatively scalable and automated solution. I'm not sure whether the human discretion supporting people are really correct in this matter.
Your comments are invited and other mitigations and solutions to Goodhart's law are also invited.
