Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

The Importance of Saying "Oops"

52 Post author: Eliezer_Yudkowsky 05 August 2007 03:17AM

I just finished reading a history of Enron's downfall, The Smartest Guys in the Room, which hereby wins my award for "Least Appropriate Book Title".

An unsurprising feature of Enron's slow rot and abrupt collapse was that the executive players never admitted to having made a large mistake.  When catastrophe #247 grew to such an extent that it required an actual policy change, they would say "Too bad that didn't work out—it was such a good idea—how are we going to hide the problem on our balance sheet?"  As opposed to, "It now seems obvious in retrospect that it was a mistake from the beginning."  As opposed to, "I've been stupid."  There was never a watershed moment, a moment of humbling realization, of acknowledging a fundamental problem.  After the bankruptcy, Jeff Skilling, the former COO and brief CEO of Enron, declined his own lawyers' advice to take the Fifth Amendment; he testified before Congress that Enron had been a great company.

Not every change is an improvement, but every improvement is necessarily a change.  If we only admit small local errors, we will only make small local changes.  The motivation for a big change comes from acknowledging a big mistake.

As a child I was raised on equal parts science and science fiction, and from Heinlein to Feynman I learned the tropes of Traditional Rationality:  Theories must be bold and expose themselves to falsification; be willing to commit the heroic sacrifice of giving up your own ideas when confronted with contrary evidence; play nice in your arguments; try not to deceive yourself; and other fuzzy verbalisms. 

A traditional rationalist upbringing tries to produce arguers who will concede to contrary evidence eventually— there should be some mountain of evidence sufficient to move you.  This is not trivial; it distinguishes science from religion.  But there is less focus on speed, on giving up the fight as quickly as possible, integrating evidence efficiently so that it only takes a minimum of contrary evidence to destroy your cherished belief.

I was raised in Traditional Rationality, and thought myself quite the rationalist.  I switched to Bayescraft (Laplace/Jaynes/Tversky/Kahneman) in the aftermath of... well, it's a long story.  Roughly, I switched because I realized that Traditional Rationality's fuzzy verbal tropes had been insufficient to prevent me from making a large mistake. 

After I had finally and fully admitted my mistake, I looked back upon the path that had led me to my Awful Realization.  And I saw that I had made a series of small concessions, minimal concessions, grudgingly conceding each millimeter of ground, realizing as little as possible of my mistake on each occasion, admitting failure only in small tolerable nibbles.  I could have moved so much faster, I realized, if I had simply screamed "OOPS!" 

And I thought:  I must raise the level of my game.

There is a powerful advantage to admitting you have made a large mistake.  It's painful.  It can also change your whole life.

It is important to have the watershed moment, the moment of humbling realization.  To acknowledge a fundamental problem, not divide it into palatable bite-size mistakes.

Do not indulge in drama and become proud of admitting errors.  It is surely superior to get it right the first time.  But if you do make an error, better by far to see it all at once.  Even hedonically, it is better to take one large loss than many small ones.  The alternative is stretching out the battle with yourself over years.  The alternative is Enron.

Since then I have watched others making their own series of minimal concessions, grudgingly conceding each millimeter of ground; never confessing a global mistake where a local one will do; always learning as little as possible from each error.  What they could fix in one fell swoop voluntarily, they transform into tiny local patches they must be argued into.  Never do they say, after confessing one mistake, I've been a fool.  They do their best to minimize their embarrassment by saying I was right in principle, or It could have worked, or I still want to embrace the true essence of whatever-I'm-attached-to.  Defending their pride in this passing moment, they ensure they will again make the same mistake, and again need to defend their pride.

Better to swallow the entire bitter pill in one terrible gulp.

 

Part of the Letting Go subsequence of How To Actually Change Your Mind

Next post: "The Crackpot Offer"

Previous post: "Feeling Rational"

Comments (25)

Sort By: Old
Comment author: BillK 05 August 2007 10:00:08AM 15 points [-]

It really is the hardest thing in life for people to decide when to cut their losses.

When you have time and effort invested, it is *so* difficult to finally decide that 'enough is enough' and stop following that path.

Take simple queing at the bank or checkout or waiting for a bus. After you know you should quit, you feel that joining a new queue means 'losing' all the time already invested in the slower queue.

From financial investments to marriage it is all the same problem. Do I carry on or write it off and try something else? It is easy to say, but really, really difficult to do.

Comment author: Hopefully_Anonymous2 05 August 2007 11:50:29AM 0 points [-]

Great post.

Comment author: Robin_Hanson2 05 August 2007 12:25:59PM 21 points [-]

It would be very useful if we could have some idea of how often people make big mistakes. Then it would be suspicious if we had gone on much longer than this time thinking we had not made such a mistake.

Comment author: Doug_S. 05 August 2007 03:02:21PM 5 points [-]

I think the title of the book was supposed to be ironic: they thought of themselves as the "smartest guys in the room" but their carefully constructed house of cards eventually collapsed.

I know of at least one decision I made that turned out poorly; my choice of college major. I do not know if it was the best decision to have made based on the information I had at the time.

My experience was this:

I was considering which of two majors would be better. I was advised to choose one of them. I chose that one.

In my second year of college, I began taking courses specific to that major. I did not like them. I was informed that those courses are, in fact, awful, everyone hates them, but they cover necessary material for the better, more advanced courses, and that once I started to take the upper level courses, I would like the major more than I currently did. I accepted this advice, and did not change majors.

I then began to take higher level courses. As it turned out, they were about as bad as the lower level courses. It took me a while to realize this. By the time I thought it was likely that I was in the wrong major, however, I had taken many courses that could not be easily transferred to any other field. I decided that, as the courses I had already taken were a sunk cost, it would be better to put up with a few more awful courses and then graduate than to transfer to another major and start my college degree from scratch.

In the end, it took me six years to graduate. I graduated in May 2006, worked during the summer of 2006 in a temporary position I did not like, and have been happily job-free ever since (much to the dismay of my parents).

In hindsight, I realize that I would be more satisfied if I had chosen a different major, but I do not know at what point I ought to have known that I should have chosen a different major.

Comment author: TobyBartels 22 August 2010 02:16:25AM *  4 points [-]

I do not know at what point I ought to have known that I should have chosen a different major.

I suggest, by your second year. I expect that you hated those classes more than most of the people who continue in that major, and by checking with third-year students, you could have discovered this fact.

In my second year, I took classes that everybody traditionally hated and grumbled about. And I dutifully grumbled about them along with the rest. But secretly, I liked them. (There was much to complain about, but in the end, they were interesting.) And when second-year students asked me about them in my third year, I believe that I admitted this. (Ironically, I changed majors in my third year anyway, but for different reasons, and with little waste.)

Well, this is just a hypothesis, but I wonder (if you even notice this comment three years late) if it seems reasonable to you.

Comment author: Byrne 05 August 2007 04:13:33PM 3 points [-]

One fascinating thing about Enron is that they found a way to corrupt their own standards. They were huge fans of marking assets to market -- which could have averted both savings and loan crises, kept Executive Life from collapsing, averted the Japanese banking crisis, etc. On top of that, they loved incentive-based compensation.

This all fell apart when some Enron traders became the market: if you get paid based on prices, and you set prices, the rest of the Enron story is inevitable.

Comment author: Eliezer_Yudkowsky 05 August 2007 05:33:18PM 1 point [-]
Comment author: Emmanuel_Okyere 05 August 2007 08:48:15PM 0 points [-]

brilliant post; BillK's comment, even better :)

cheers, eokyere

Comment author: anon2 06 August 2007 03:11:17PM 2 points [-]

Why don't you have to take into account the prior probability of the large mistake that occurred? Of course, you might be biased and believe it to be smaller than it truly is, in which case there should be a whoops moment (your mistake was overconfidence), but clearly there must also be cases in which there was a small prior probability of a big mistake. Shouldn't we not judge these cases by only examining the outcome?

Comment author: Bernard_Guerrero 07 August 2007 05:54:58PM 1 point [-]

What's the heuristic supposed to be, here? Taking your medicine early and going for the big change sounds better, in principle, but I think it amounts to more fuzziness. The first small bit of evidence that you made a mistake may or may not actually relate to an error. There are false positives as well as false negatives. At some point the evidence overwhelms your own risk tolerance (as it relates to future costs, the historical ones are sunk) and you change your mind....or not. It isn't clear to me that you minimize your costs by jumping to the conclusion that you were wrong any more than you do so by clinging to the idea that you remain correct.

Comment author: Eliezer_Yudkowsky 07 August 2007 06:03:56PM 2 points [-]

It would be possible to recognize large mistakes too early, rather than too late. Can anyone here think of any case where they've ever seen a human being do that?

Comment author: ABranco 20 October 2009 04:36:15AM 4 points [-]

I admit I don't remember any case of people publicly affirming, out of the blue: "I am damn wrong, big time!".

No homeopath waking up in a regular Tuesday and declaring "Gee!, I'm sorry, I've read an article and realized I've been medicating people with this placebo 'Styrofoam' little balls for the last five years and now I see this was insane. I am publicly apologizing for my patients. I quit. But I am happy that I can see it now, and can still become a specialist in something else."

Confirmation bias seems to grow stronger the more the time passes, and the more public their opinion gets.

(I know this is an ooooold post, but since I thought about it, why not reply?)

Comment author: Hopefully_Anonymous2 07 August 2007 06:52:27PM 3 points [-]

Eliezer, not bothering to go after a goal may fall into that category. For example, it's reasonable to choose to live an average life, because one is probably mistaken if one thinks one is likely to have strongly positively deviant outcomes in life, such as becoming a billionaire, or procreating with a 1 in a million beauty, or winning a nobel prize for one's academic contributions, or becoming an A list celebrity. So one may choose never to invest in going after these goals, and devote the balance of one's time and energy to optimizing one's odds of maintaining a median existence, in terms of achievements and experiences. I could name people who seem to be doing that, but you've never heard of them.

Comment author: ABranco 20 October 2009 04:28:22AM 0 points [-]

This is a case of availability heuristic, if I understand what you're saying. That people who screw up and admit only in the end gets into the news; those who admit big mistakes early do not.

Sounds reasonable. But this example is not really about mistakes, just an adjustment of ambitions and expectations, which might even be subconscious. It's not really about right or wrong, as I see it.

Comment author: Bernard_Guerrero 07 August 2007 06:57:41PM 2 points [-]

Eliezer,

Not so much recognizing mistakes too early. Rather, mistakenly seeing a mistake where there isn't one. False positives abound.

Comment author: John 22 September 2007 05:02:29AM 23 points [-]

BillK said:

"It really is the hardest thing in life for people to decide when to cut their losses."

No it's not. All you have to do is to periodically pretend that you were magically teleported into your current situation. Anything else is the sunk cost fallacy.

Comment author: Eliezer_Yudkowsky 22 September 2007 06:45:44AM 1 point [-]

Excellent heuristic, John!

Comment author: Krishna 22 September 2007 01:05:49PM 0 points [-]

I don't think the "oops" situation will show up early under all circumstances. If it's a situation where we've been before, tasting success and failure, we could sense its symptoms and diagnose early. But if it's a new venture and we are passionate about it, we'll give it a longer rope hoping for the best.

That said, Enron collapse had many dimensions. Not just the non-admission of a mistake or not having the gall to capitulate. The perpetrators walked into it with eyes wide open, perhaps relying on the theory of "greater fool" - that goes there will always be a greater fool to whom you can palm off your bad bets. Had they been lucky, they would've even gotten away with it all - perhaps selling off to a Private Equity fund just like Sam Zell did (with Equity Office Trust) to Blackstone, sensing the madness of subprime early on, and Schwarzman going public and cashing out...

Wouldn't you agree, Eleizer...?

Comment author: BillK 22 September 2007 02:13:35PM 3 points [-]

John wrote: All you have to do is to periodically pretend that you were magically teleported into your current situation. Anything else is the sunk cost fallacy.

Yup! I agree. That's *all* you have to do.

Try explaining that to your wife, when you decide that you didn't really want to have a mortgage, two kids to bring up, a job with no prospects, etc. and that the sunk cost fallacy means that you are going off to California with the blonde cheerleader down the road.

You'll probably find it real easy. ;)

Comment author: Eliezer_Yudkowsky 22 September 2007 05:55:03PM 2 points [-]

BillK, the external consequences of that are the same whether you were just teleported into the situation or got into it yourself. Though I agree that the internal consequences may be different, if you truly did, in the past, give your word.

Krishna, of course the Enron collapse was complicated, but - at least according to the book I read - they were drinking their own Kool-Aid.

Comment author: BillK 22 September 2007 06:27:12PM 5 points [-]

Eliezer, I agree completely.

That is exactly the point of my original remark. Knowing that you should cut your losses due to a previous mistaken decision is not the difficult part. Deciding to actually do it and face the consequences is probably one of the hardest things people have to face in their life. It affects people personally because they have to admit that they have spent possibly many years on the wrong path. In some cases it will also have serious consequences on their friends and family.

Some people find it so hard to do that they will commit suicide in preference.

Comment author: Krishna 23 September 2007 03:20:51AM 0 points [-]

Well said, Bilk. I agree with you in that not many are endowed with the faculties to spot the rot early on and the eventual courage to walk down the road with the blonde cheerleader. Deeper we are mired, harder it gets to come out. I could tangent it to a space travel metaphor. A slew of peripheral factors get built around the `wrong' that becomes the core, to which our existence (including that of our family) gravitates and extrication would call for application of an intense escape velocity, impossible at a time when we are nearly out of gas and don't even have cruising speed. Even if we manage that, somehow, somehow, we'll find ourselves propelled into a space, a new environment that offers neither gravity nor any kind of support to which we’ve grown so much used to, where we just have to stay suspended not knowing where to go... It's this vision of uncertainty that holds us back from abject confession, tempting us to evaluate the cost v. benefit that in the near term will mostly be reclining more towards *cost* ( that of shame induced fear, the looming prospect of sudden loss of self esteem) than *benefit* (lightness of conscience, a clean slate, scope for restart) . There is also a grail of hope that there'll-soon-be-a-way-out that tempts us to let our folly will lie buried deep within till we choose to turn the spigot on, at an hour when we have the advantage and to an audience that will laud our triumph than laugh at our misjudgment.

Hope we are one here.

Comment author: Once_in_a_civilization_mistake 27 September 2008 11:00:20PM 3 points [-]

A few years ago I realized I'd been totally wrong for the past two decades - my entire professional life. It felt like my life had been wasted, because, in fact, it had been; none of the skills, none of the knowledge I possessed applied to my new situation. The solid steel building I thought I'd built was turned to sand and blown away in one terrible instant.

Now after years of previously unimaginable success following that epiphany, it all seems like a bad dream, like that part of my life really never happened. The current me is unable to see into the mind that made those mistakes. It was someone else. That guy could never have achieved what I have now.

Comment author: MrHen 17 December 2009 07:24:28PM 7 points [-]

Not every change is an improvement, but every improvement is necessarily a change. If we only admit small local errors, we will only make small local changes. The motivation for a big change comes from acknowledging a big mistake.

I do not fully understand this last sentence. Specifically, why does a big mistake imply that a big change needs to be made? Perhaps I am not understanding the difference between a small mistake and a big mistake.

In other words, is a mistake's "bigness" dependent on how much change is required to undo/fix/prevent the mistake? Or is is a mistake's bigness dependent on the effects of the mistake?

A simple mathematical error can cause planes to drop out of the sky for lack of fuel. I instinctively say this is a big mistake with a small change required to fix it.

I think the minor differences between admitting mistakes that require big changes and admitting mistakes that have large consequences trigger different defense mechanisms. Namely, the former is laziness and the latter is shame. Knowing where the resistance comes from is useful when overcoming the resistance. It is also possible to notice the resistance before you realize that a mistake has been made. Consciously acknowledging a mistake can only happen after you begin to realize that a mistake may have been made.

The big examples you use seem to point toward large, shameful confessions, which makes perfect sense. They seem to qualify for both big effects and big changes required to fix them. These probably trigger whatever resistances would have applied to one category or the other. Which is probably why a post like this is such a useful shot in the arm to make one sit down and self-reflect for a few minutes.

All of which is to say, "Yeah, I agree," with paragraphs instead of three words.

That being said, I don't understand why the motivation for a big change comes from acknowledging a big mistake. Wouldn't it be the other way around? If I want to make better, bigger changes, I could start by acknowledging big mistakes. The motivation for the acknowledgement comes from the desire to improve. If acknowledging errors provides motivation to change than I would argue that the change is based on guilt or dissatisfaction. The goal should be positive change, not alleviating pressure on your ego. If acknowledging mistakes is in the way of the higher goal than throw your ego in the fire and improve.

Comment author: Ebthgidr 01 January 2014 06:20:08PM 0 points [-]

Is this a possible explanation or corollary to the sunk-costs fallacy of economics?