Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Comment author: botogol 20 May 2010 09:31:42AM 1 point [-]
Comment author: kodos96 17 March 2010 08:56:17PM *  8 points [-]

Getting back to trying to propose practical mitigation strategies for goodhart's law, I propose a fairly simple solution: Choose a G*, evaluate performance based on it, but KEEP IT SECRET. This of course wouldn't really work for national scale, GDP-esque kind of situations, but for corporate management situations it seems like it could work well enough. If only upper management knows what G* is, it becomes impossible to optimize for it, and everyone has to just keep working under the assumption they're being evaluated on G.

Taking it a step further, to hedge against employees eventually figuring out G* and surreptitiously optimizing for it, you could have a bounty on guessing G* - the first employee who figures out what the mystery metric G* really is gets a prize, and as soon as it's claimed, you switch to using G**

Comment author: botogol 01 April 2010 03:15:07PM 0 points [-]

if management are doing that then are neglecting a powerful tool in their tool-kit, because announcing a G* will surely cause G* to fall, and experience says that to begin with a well-chosen G* and G remain correlated (because many of the things to do to reduce G* also reduce G). It is only over time that G* and G detach.

Comment author: botogol 16 March 2010 04:28:39PM 4 points [-]

At work a large part of my job involves choosing G* , and I can report that Goodhart's Law is very powerful and readily observable.
Further : rational players in the workspace know full-well that management desire G, and the G* is not well-correlated with G, but nonethelss if they are rewarded on G*, then that's what they will focus on.

The best solution - in my experience - is mentioned in the post: the balanced scorecard. Define several measures G1* G2* G3* and G4* that are normally correlated with G. The correlation is then more persistent : if all four measures improve it is likely that G will improve.

G1* G2* G3* G4* may be presented as simulaneous measures, or if setting four measures in one go is too confusing for people trying to prioritise (the frwer the measures the more powerful) they can be sequential. IE If you hope to improve G over 2 years, then measure G1* for two quarters, then switch the measurement to G2* for the next two and so on. (obviously you don't tell people in advance). NB this approach can eb effective, but will make you very unpopular.

Comment author: MatthewB 26 January 2010 09:28:41AM 0 points [-]

Of course, just like with Military Training, the Firefighters may have biases about what they consider to be rational.

For instance, most would probably save the injured baby at the expense of an uninjured adult or child. Yet, the baby has less immediate worth than the adult or small child, as these latter two are conscious and self-aware in a way that the baby is not.

Yet, almost instinctively, humans tend to go for the baby. Of course, genetics has wired us to be that way.

Comment author: botogol 28 January 2010 11:38:57AM 1 point [-]

That's true (that they have biases) although I understand the training is attend to the nature of the injury, and practicalities of the situation - eg danger to the firefighter - rather than the age of the victim.

However what one might expect to see in firefighters would be ethical dilemmas like the trolley problem to trigger the cerebral cortex more, and the amaglydia less than in other people.


Unless of course the training works by manipulating the emotional response. So firefighters are just as emotional, but their emotions have been changed by their training.

This is the sort of problem Kahane was talking about when he said it is very difficult to interpret brain scans.

Comment author: MatthewB 25 January 2010 09:40:10AM 4 points [-]

Are these sorts of ethical dilemmas ever posed to people serving in the Military?

I often wonder about this, because there seems to be no shortage, in times of real crisis, to find a soldier willing to fling himself under the trolley (without needing to hurl a fat man) if need be.

Of course, this is something that is ingrained in the soldier's behavior during training, and it, at times, doesn't take.

Then, there are times when the military becomes filled with people who are willing to throw anyone onto that railroad track, as long as the math turns out correctly (more people saved than killed). It is an officer's job to do just that (and then order a soldier to do the actual throwing).

I guess this just shows that people are capable of having their normal morality either amplified or nullified depending upon the context. In some cases, there may be no morality to speak of in some classes of soldiers.

However, you do bring up an excellent point. Immediacy of the situation/decision. It is relatively easy for people to make dispassionate and rational decisions when they are not in the heat of the moment or more removed from the actual situation. I have found that it is a rare individual who can actually make a rational decision in the heat of the moment without considerable training (and not just mental training. It takes real simulation of the act for most people to learn just how they will react in such situations).

Comment author: botogol 25 January 2010 10:36:36AM 2 points [-]

A person in the audience suggested taking firefighters, who sometimes face dilemmas very like this (Do I try to save life-threatened person A or seriosly injured Baby B), and hooking them up to scans and seeing if their brains work differently - The hypothesis being that they would make decision in dilemmas more 'rationally' and less 'emotionally', as a result of their experience and training. Or the pre-disposition that led to them becoming fire-fighters in the first place.

Comment author: Kevin 25 January 2010 09:13:16AM 1 point [-]

I think the thing that made it seem like a draft is the missing "I went" at the beginning of the article. I also noticed illustrate is misspelled, at a quick glance.

Comment author: botogol 25 January 2010 09:27:30AM 4 points [-]

The opening was deliberate - it's a common way that newspaper Diarists start their entries.... but perhaps it's a common way that British newspaper diarists start their entries, and sounds wrong to american ears. So I have changed it. Nations divided by a common language etc.

Comment author: AngryParsley 23 January 2010 10:33:48PM 9 points [-]

I thought most people chose not to push the fat man because there is no conceivably realistic way that a fat man could stop a train, even one as small as a trolley. Although the thought experiment tells us the fat man will stop the train, our knowledge of trains tells us that nothing stops trains. When I envision this scenario, I can't help but (realistically) imagine the trolly hitting the fat man, then continuing on and running over the five others.

See also: Ends Don't Justify Means (Among Humans)

Comment author: botogol 25 January 2010 09:23:25AM 2 points [-]

Yes. People get bogged down with the practical difficulties. Another common one is whether you have the strength to throw the stranger off the bridge (might he resist your assault and and even throw you off).

I think the problem is the phrasing of the question. People ask 'would you push the fat man', but they should ask 'SHOULD you push the fat man'. A thought experiemnt is like an opinion poll, the phrasing of the question has a large impact on the answers given. Another reason to be suspicious of them.

Comment author: RobinHanson 24 January 2010 08:07:55PM 1 point [-]

You are really going to take a concept worked out in dozens of academic papers and declare it meaningless because you have trouble figuring out how to apply it in one particular context?

Comment author: botogol 25 January 2010 08:53:14AM 3 points [-]

No, I wasn't declaring it meaningless.

My (perhaps trivial) points were that all hypothetical thought experiments are necessarily conducted in Far mode, even when thought experiment is *about * simulating Near modes of thinking. Does that undermine it a little?

And - while all Thought Experiments are Far - Actual Experiements are Near.

I was illustrating that with what I hoped was an amusing anecdote -- the bizarre experience I had last week of having the trolley problem discussed with the fat man actually personified and present in the room, sitting next to me, and how that nudged the thought experiment into something just slightly closer to a real experiment.

It's easy to talk about sacrificing one person's life to save five others, but hurting his feelings by appearing to be rude or unkind, in order to to get to a logical truth was harder. This is somewhat relevant to the subject of the talk - decisions may be made emotionally and then rationalised afterwards.

Look, I wasn't hoping to provoke one of Eliezer's 'clicks', just to raise a weekend smile and to discuss scenario where lesswrong readers had no cached thought to fall back on.

Comment author: Eliezer_Yudkowsky 23 January 2010 10:29:58PM 0 points [-]

Was this intended to be published? It seems unedited. Save in "Drafts" instead of "Less Wrong" to unpublish.

Comment author: botogol 25 January 2010 08:32:06AM 4 points [-]

:-( no, not a draft! It was just supposed to be light-hearted - fun even - and to make a small point along the way.... it's shame if lesswrong article must be earnest and deep.

Far & Near / Runaway Trolleys / The Proximity Of (Fat) Strangers

9 botogol 23 January 2010 10:13PM

I went to the Royal Institute last week to hear the laconic and dismissive Dr Guy Kahane on whether we are 'Biologically Moral

[His message: Neurological evidence suggests - somewhat alarmingly - that our moral and ethical decisions may be no more than post-hoc rationalisations of purely emotional, instinctive reactions.  However, we should not panic because this is early days in neuroscience, and the correct interpretation of brain-scans is uncertain: scientist find the pattern, and the explanation, they expect to find]

continue reading »

View more: Next