You are viewing a version of this post published on the . This link will always display the most recent version of the post.
This reminds me of the "extreme programming" planning technique: you estimate work by chopping it up into tasks and figuring out how long they would take in "ideal days" (that is if everything goes perfectly) . Then you see how many "ideal days" worth of tasks you managed to deliver last month (for instance) and multiply your estimate by the ratio actual days/ideal days.
This method works reasonably well but people can be quite resistant to facing up to the consequences of how long things actually take. It strikes me that this is a pretty good template for de-biasing in general: use frequent feedback of reality to adjust your priors.
Don't forget that this bias is reinforced by incentives, as managers tend to prefer overconfident planning forecasters to accurate ones.
Very interesting idea, the inside-outside contrast. Is this connected in any way to the fact that outside consultants are often better a spotting problems than those inside the company?
Both groups finished an average of 3 days before Christmas.
Is this true in general? Is there really no difference between time taken on an optimistic schedule and a realistic one? I've found that projects with long dealines tend to "slip" a bit, no matter how long the deadline is.
In Boston we have heard many reporters claim that the delays in the Big Dig (the highways tunneled under the city and the harbor) were increased by contractors stretching out the work to increase their incomes. This suggests that there are strong incentives, particularly in projects paid for through government, to over promise (underbid) and under deliver (negotiate higher pay when work is under way).
Not so much a planning bias as a pocket book bias.
John
Very very very late reply.
But there's been research on this in recent years. See Bent Flyvbjerg's papers on "strategic misrepresentation", where he outlines how perverse incentives can lead people to intentionally make overconfident predictions in government work projects.
However, Flyvbjerg also points out that there is probably a combination of psychological factors involved, too, as we continue to see this kind of overconfidence/optimism in areas like student predictions or trading (where actively trading often does worse).
I really like your posts, but with most of them I'm already a believer. But estimating how long it takes me to finish papers is something I'm really bad at, by a factor of about 10, yet I never learn. So this post could be a big help.
> Both groups finished an average of 3 days before Christmas.
Is this true in general? Is there really no difference between time taken on an optimistic schedule and a realistic one? I've found that projects with long dealines tend to "slip" a bit, no matter how long the deadline is.
My impression from reading the literature so far is that project estimates are highly informative, that is, students which estimated earlier completion times did tend to finish earlier. However, manipulations which resulted in earlier estimates, such as asking students to visualize their plans in detail, did not affect completion time; nor did manipulations which led to more realistic estimates.
That's my impression from reading so far, but I also have a dim recollection of having heard different results from elsewhere (a case where manipulating estimates did affect completion times).
There may also be a difference between individual and bureaucratic projects, or manipulating an external deadline versus manipulating the estimate.
I notice in the 1994 study, the students were directly asked for their forecasts. Do any of the studies try to get students to write down their forecasts on an envelope to be opened after they have finished their project, to try to avoid any possible social pressure?
"In a different paper, Buehler and colleagues suggest an explanation in terms of the self-serving bias in how people interpret their past performance. By taking credit for tasks that went well but blaming delays on outside influences, people can discount past evidence of how long a task should take.[1] One experiment found that when people made their predictions anonymously, they do not show the optimistic bias. This suggests that the people make optimistic estimates so as to create a favorable impression with others,[8] which is similar to the concepts outlined in impression management theory." -- Wikipedia (https://en.wikipedia.org/wiki/Planning_fallacy)
The article fails to take into account actual time spent on the project. Why is it that I can write just as good a paper with a one week or 10 week deadline? Is the problem underestimation of time or lack of motivation to finish before the predicted deadline? I don't think student reports are a very good model for this kind of cognitive bias.
Interesting. That corresponds remarkably well with the best advice I've ever gotten for estimating software schedules: take your best guess as to when you'll be done, then double it.
Having worked on several large software projects I have experienced first hand overly optimistic goals. Just because you think you have a clue about what it is you're supposed to do doesn't mean that when you actually sit down and start working that you know even 1/10th what you need to know. For me personally that is the reason why the projects I've worked took longer than I (and everyone else involved) thought at first. I simply didn't know what I didn't know. It wasn't long before half my work days were spent simply reading and learning about what it was I was trying to do before I actually got around to doing it.
I was reading this inside view / outside view and before I got to the explanation of what it means, i had a completely different meaning forming in my head. So... here's my idea.
How about estimating based on how others perform? What I mean is, instead of thinking how long you'd take to finish that paper (or whatever), try to thinking how long would it take your classmate, your friend, someone else.
Do you think this could give better estimates?
Estimates get derailed by "unexpected delays or unforeseen catastrophes". These are nothing but risks. If one can manage the project by managing the risks in the project and keeping it in control I think most tasks can be accomplished within a reasonable estimate.
I have written more on this on my blog http://sudeepdsouza.blogspot.com/2008/03/managing-risk-for-better-estimates.html.
Good stuff. Here are three relevant tidbits that serve to illustrate an expanded view of the point.
In "The Logic of Failure", the point was made that humans are horrible at recognizing that an exponential process is in play.
There is a saw to the effect (I'm paraphrasing) of "Human's grossly overestimate what will happen in 5 years but grossly underestimate what will happen in 10." This pretty much is a result of No. 1.
I believe Thomas Jefferson (an outsider in the context of this post) was said to predict that it would take "1000 years to settle the West." He was off by roughly a factor of 20 ... but not in the direction predicted by this post. This pretty much is an actual instance of the effect posited by 2.
I can rationalize the completion of the Sydney Opera House being a manifestation of this phenomena. It's more of a reach, but I can probably rationalize that the prediction of the completion time of a paper is also explained.
Joel Spolsky has an interesting take on this kind of cognitive bias WRT software planning, where his company uses software to attempt to quantify the amount of individual bias by tracking task estimates vs. actual times and developing an individualized factor for padding subsequent planning. See e.g. http://www.joelonsoftware.com/items/2007/10/26.html
Interestingly enough, this approach emphasizes tracking more granular inside view-type tasks over the large-scale outside view perspective mentioned in the original post. I'm curious whether generating estimate velocities in an outside view manner (i.e. for larger-scale tasks) would further improve the realism of this estimation method.
Glyn, I did something similar, but with mine after the granular tasks are estimated, a random delay is added to each according to a pareto distribution. The more subtasks, the more certain that a few of them will be very much behind schedule.
I chose a pareto distribution because it had the minimal number of parameters to estimate and it had a fat tail. Also I had a maximum entropy justification. Say you use an exponential distribution, you're assuming a constant chance for completion at any time that it's incomplete. But other things equal, the more you get behind schedule the less likely that the chance for cmpletion at any time will stay constant. It should go down. If you estimated 3 hours to completion to start with, and it's already been 6 hours, is it more likely that the correct estimate now is 3 hours, or something larger? And when it's been 9 hours and still incomplete, should you predict 3 hours then? The more you're already behind deadline, the more reasonable it is to suppose that you'll get even farther behind.
I always find it strange that, every year, the US Congress passes a budget that assumes that nothing will go wrong over the next year. Every long-range budget plan also assumes that nothing will go wrong. (On the flip side, they also assume that nothing will go right: Planning for health care assumes that investment in health research will have no effect.)
The estimate you would like to have for a project is the investment needed to complete it in the average case. But humans don't think in terms of averages; they think in terms of typicality. They are drawn to the mode of a distribution rather than to its mean.
When distributions are symmetric, this isn't a problem. But in planning, the distribution of time or cost to completion is bounded below by zero, and hence not symmetric. The average value will be much larger than the modal value.
What a wonderful blog, I just discovered it. This is an old post so I am not sure if anyone is still following it. While I think the article raises some excellent points, I think it may be missing the forest for the trees. Perhaps due to bias :-).
For instance, the article states:
The conclusion then seems to be that everyone did a poor job of estimating. Maybe, maybe not. Why not instead question if their were other cognitive/behavioral factors at play? For example:
These are just a few thoughts. I submit the opposite of the articles conclusion (without invalidating it). Most projects take longer than they have to because of cognitive/behavior issues. And here I will quote the blog mission statement: "If we know the common patterns of error or self-deception, maybe we can work around them ourselves, or build social structures for smarter groups."
And that is also the key to achieving faster and on/time projects, not just accepting that our planning is faulty and looking at past projects - many of those past projects likely took longer than they needed to because of cognitive bias.
As an architect and sometime builder, as an excellent procrastinator, I heartily concur with this comment.
The range of biases, psychological and 'structural' factors at work is wide. Here are a few:
'tactical optimism' : David Bohm's term for the way in which humans overcome the (so far) inescapable assessment that; 'in the long run, we're all dead'. Specifically, within the building industry, rife with non-optimal ingrained conditions, you wouldn't come to work if you weren't an optimist. Builders who cease to have an optimistic outlook go and find other things to do.
maintaining flexibility has benefits: non-trivial projects have hidden detail. It often happens that spending longer working around the project - at the expense of straight-ahead progress - can lead to higher quality at the end, as delayed completion has allowed a more elegant/efficient response to inherent, but unforeseen problems.
self-application of pressure: as someone tending to procrastinate, I know that I sometimes use ambitious deadlines in order to attempt to manage myself - especially if I can advertise that deadline - as in the study
deadline/sanction fatigue: if the loss incurred for missing deadlines is small, or alternatively if it is purely psychological, then the 'weight' of time pressure is diminished with each failure.
I'm going to stop now, before I lose the will to live.
I did publish an article about overconfidence as a strong bias in estimation (here's the link: forecasting and estimating biases in projects ).
Personally, I've seen it, even when the person (especially when the person is a programmer) thinks s/he's pessimistic in estimating time, estimates are always missed. A recent example I've had was a task that the programmer promised he was going to finish it first in 2 days, I told him to be overly pessimistic, he said 2 weeks. The task was scheduled for 4. Now, after 5 weeks, the programmer says he's 2 days away from finishing.
1) Procrastinating until the last moment to actually do the work (you have never heard of students doing that, have you?) :-). This is a common reason that no matter how long people are given to complete a task, they do not complete it on time, or do so at the last minute.
David, I think you're kind of missing the point here. The question is whether students could predict their projects' actual completion time; they're not trying to predict project completion time given a hypothetical version of themselves which didn't procrastinate.
If they aren't self-aware enough to know they procrastinate and to take that into account - their predictions are still bad, no matter why they're bad. (And someone on the outside who is told that in the past the students had finished -1 days before the due date will just shrug and say: 'regardless of whether they took so long because of procrastination, or because of Parkinson's law, or because of some other 3rd reason, I have no reason to believe they'll finish early this time.' And they'd be absolutely correct.)
It's like a fellow who predicts he won't fall off a cliff, but falls off anyway. 'If only that cliff hadn't been there, I wouldn't've fallen!' Well, duh. But you still fell.
This research should be read in conjunction with Bent Flyvbjerg's work on Megaprojects and Risk . Flyvbjerg has recently moved to Oxford, I believe.
'... a cross-cultural study, found that Japanese students expected to finish their essays 10 days before deadline. They actually finished 1 day before deadline. Asked when they had previously completed similar tasks, they responded, "1 day before deadline."'
'This is the power of the outside view over the inside view.'
Is it? This assumes that the "outside" '1 day before deadline' represents an actual forecast of the time needed to complete a task. But what does it mean to "complete" an essay? The deadline itself FORCES completion to take place, ready or not. If say three days before the deadline the faculty announced a one week extension, I think you would find that most of the essay durations would extend another week, so as to still be "1 day before deadline." So the previous outside estimates become little better than the inside ones.
In Boston we have heard many reporters claim that the delays in the Big Dig (the highways tunneled under the city and the harbor) were increased by contractors stretching out the work to increase their incomes.
Not so much a planning bias as a pocket book bias.
I presume that (a) this is not the first time that contractors have stretched out a job; (b) the Big Dig planners FAILED TO EXPECT contractors to behave exactly as they ALWAYS DO.
Interesting. This reminds me of the recursive Hofstadter's law:
"It always takes longer than you expect, even when you take into account Hofstadter's Law."
If you're lucky, it will converge to a finite amount of time!
interesting...
this planning fallacy could also help explain gross underestimates of project costs beyond the effects of personal incentives.
I'm still reading through the archives of LW, catching up with where the conversation has been, figuring out what I can expect to learn, what I can expect to contribute. And this comment is as much a note to myself as a it is a tiny part of that conversation. But...
Extended discussion of "agile" or "extreme programming" techniques seems to be too specialized to be relevant to LW. (Although it seems like the general topic of how to write reliable and maintainable code should be a priority for the subset of the LW community who's interested in working on AIs.)
However, a systematic look at normative, descriptive and prescriptive models of thinking as it applies to planning and executing projects, including biases, heuristics and shortcomings would be very interesting, not just to me but probably to a fair number of LW readers. I mean "projects" here in the broadest possible meaning; "predicting the future by inventing it" to paraphrase Alan Kay. This includes not just the various flavors of "project management" within businesses, but the more general topic of how people formulate projects for themselves, enlist others in their projects, and succeed or fail with those projects.
This post barely scratches the surface of the issues relating to planning, for instance, inasmuch as estimation is not just an individual but also a social phenomenon. Extrapolating from what students to in a task estimation study to the underlying causes of large delays in big projects such as DIA is likely oversimplifying.
The topic map on the Wiki doesn't suggest that there are many more articles like this one, but I might be missing something. If I am, can someone give me pointers ?
Specifically, the researchers asked for estimated times by which the students thought it was 50%, 75%, and 99% probable their personal projects would be done.
99%? How am I supposed to answer if I assign less than 99% probability to me actually completing the task? That isn't even particularly pessimistic given reasonable priorities.
ETA after surprising vote: Really an actual answer would be appreciated. How am I supposed to understand that question? Or is it somehow shame-worthy to ask even when I am so entirely distanced from those individuals expected to give such an estimate to an actual superior?
99%? How am I supposed to answer if I assign less than 99% probability to me actually completing the task?
Not with a finite number, obviously. "I'm not 0.99 sure I'll ever get this done", "your question makes an incorrect assumption", "Mu" all seem like reasonable replies. "∞" will make some people cringe, but should also get the point across.
Thankyou. I wasn't familiar with the 'Mu' word. English or not I shall put it to good use.
"∞" will make some people cringe, but should also get the point across.
Agree on both counts. Especially the cringing part.
Something I wrote a while back... A consideration of a relationship between self-serving bias and self-handicapping i.e. self-sabotage. We tend to perceive ourselves favourably and seek out rationales and behaviour that confirms our opinion of ourselves - this is called the self-serving bias.
Some times we self-handicap to create excuses for our possible failure, this shores up our self-serving bias, but self-sabotage also makes our failure more likely, potentially creating a self-fulfilling prophecy. One way we self-handicap is to not try, or not try hard enough.
Procrastination for study is one example of this relationship, which when combined with the tendency to underestimate the time needed to complete a task - i.e. the Planning Fallacy, it's no wonder why so many students struggle to achieve. We leave it to the last minute AND it takes longer to do than we thought it would!
Place these above concepts in the context of the negative aspects of both perfectionism and correction based ideologies and the power of not trying as means to protect ones self stands clearly revealed... to those seeking mastery.
I worked for over a decade as an elected official trying to bring some rationality to the planning of Denver's massive fixed rail transit system. One would have thought watching the issues with DIA would have created a collective consciousness so as to not repeat the same errors with "Guide the Ride" and "FastTracks". The project went ahead with similar results to DIA. The experiential lesson is that even within a population that has experienced the results of a planning fallacy, they will do it again.
A quick point: "et" is a complete Latin word meaning "and". Just like "and", it doesn't need a period after it, as it is not an abbreviation. Thus,
Buehler et. al. (2002)
should be changed to
Buehler et al. (2002)
Another factor into the planning fallacy is for affirmation - for oneself and others - that the 'plan' is being given high priority.
I work in the mortgage industry and see this all the time, from my clients and from coworkers. They assure that a file should be able to close by the week's end, when several key items haven't been received, or that a file will be reviewed by such and such date, and an underwriter hasn't been assigned to review it.
Instead of saying something will happen 'soon,' giving an (albeit unrealistic) deadline can be more comforting to all parties.
I believe it is silly to even try to assign a numerical probability to any event unless you can rigorously derive that number from antecedent circumstances or events (for instance, it can make sense if you are talking about scenarios involving the results of dice rolls). Thus I find the questions in LW's annual survey which demand such numbers annoying and pointless.
As for the errors in predictions of the time or money it will take to build some promised project, there's no mystery; the individuals making those predictions stand to gain substantial money or prestige if the predictions are believed, so they lie (or at least make the rosiest predictions they expect to get away with making). This especially goes for politicians, who have all the more incentive to lie because the law gives them absolute immunity (from, for example, being sued for fraud) for anything they say during legislative debate.
The way to get reliable data about these things is to create incentives that make it in someone's best interest to gather and share that reliable data. For most projects, the simplest and easiest way to do this is to have those who want the project built commission it using their own money, rather than do it through the political system.
13% of subjects finished their project by the time they had assigned a 50% probability level; 19% finished by the time assigned a 75% probability level; and only 45% (less than half!) finished by the time of their 99% probability level. As Buehler et. al. (2002) wrote, "The results for the 99% probability level are especially striking: Even when asked to make a highly conservative forecast, a prediction that they felt virtually certain that they would fulfill, students' confidence in their time estimates far exceeded their accomplishments."
I do agree that it is good to have a pessimistic view on projects over an optimistic view because then you will finishi sooner but if you have in mind that at best I will finish two weeks before the deadline and at worst the deadline day then you might be happier working on the project because you think to yourself that the project could be almost done. What is most important is to be honest about when you can be done because if you say that you might be done at a time that you for sure won't be done then you work on the project thinking inside your head that I will be finished later than I said. Optimistic vs. Pessimistic view is a balance between being happier and finishing sooner.
But experiment has shown that the more detailed subjects' visualization, the more optimistic (and less accurate) they become.
I'm working in software engineering, and I have often seen the opposite. You ask a guy, hey how long do you think you'll spend on this task? And they say, 150 hours. Now, you say, let's break it down into specific actions, and estimate them. And often it happens that the result is twice as large as original rough estimate.
This reminds me of Bostrom and others who ask experts about timelines for the development of Human level AI. Do you think one should be more conservative than these results indicated? Scted's response would seem to suggest we ought to be less conservative.
The Denver International Airport opened 16 months late, at a cost overrun of $2 billion (I've also seen $3.1 billion asserted). The Eurofighter Typhoon, a joint defense project of several European countries, was delivered 54 months late at a cost of £19 billion instead of £7 billion. The Sydney Opera House may be the most legendary construction overrun of all time, originally estimated to be completed in 1963 for $7 million, and finally completed in 1973 for $102 million.
Are these isolated disasters brought to our attention by selective availability? Are they symptoms of bureaucracy or government incentive failures? Yes, very probably. But there's also a corresponding cognitive bias, replicated in experiments with individual planners.
Buehler et. al. (1995) asked their students for estimates of when they (the students) thought they would complete their personal academic projects. Specifically, the researchers asked for estimated times by which the students thought it was 50%, 75%, and 99% probable their personal projects would be done. Would you care to guess how many students finished on or before their estimated 50%, 75%, and 99% probability levels?
As Buehler et. al. (2002) wrote, "The results for the 99% probability level are especially striking: Even when asked to make a highly conservative forecast, a prediction that they felt virtually certain that they would fulfill, students' confidence in their time estimates far exceeded their accomplishments."
More generally, this phenomenon is known as the "planning fallacy". The planning fallacy is that people think they can plan, ha ha.
A clue to the underlying problem with the planning algorithm was uncovered by Newby-Clark et. al. (2000), who found that:
...produced indistinguishable results.
When people are asked for a "realistic" scenario, they envision everything going exactly as planned, with no unexpected delays or unforeseen catastrophes - the same vision as their "best case".
Reality, it turns out, usually delivers results somewhat worse than the "worst case".
Unlike most cognitive biases, we know a good debiasing heuristic for the planning fallacy. It won't work for messes on the scale of the Denver International Airport, but it'll work for a lot of personal planning, and even some small-scale organizational stuff. Just use an "outside view" instead of an "inside view".
People tend to generate their predictions by thinking about the particular, unique features of the task at hand, and constructing a scenario for how they intend to complete the task - which is just what we usually think of as planning. When you want to get something done, you have to plan out where, when, how; figure out how much time and how much resource is required; visualize the steps from beginning to successful conclusion. All this is the "inside view", and it doesn't take into account unexpected delays and unforeseen catastrophes. As we saw before, asking people to visualize the "worst case" still isn't enough to counteract their optimism - they don't visualize enough Murphyness.
The outside view is when you deliberately avoid thinking about the special, unique features of this project, and just ask how long it took to finish broadly similar projects in the past. This is counterintuitive, since the inside view has so much more detail - there's a temptation to think that a carefully tailored prediction, taking into account all available data, will give better results.
But experiment has shown that the more detailed subjects' visualization, the more optimistic (and less accurate) they become. Buehler et. al. (2002) asked an experimental group of subjects to describe highly specific plans for their Christmas shopping - where, when, and how. On average, this group expected to finish shopping more than a week before Christmas. Another group was simply asked when they expected to finish their Christmas shopping, with an average response of 4 days. Both groups finished an average of 3 days before Christmas.
Likewise, Buehler et. al. (2002), reporting on a cross-cultural study, found that Japanese students expected to finish their essays 10 days before deadline. They actually finished 1 day before deadline. Asked when they had previously completed similar tasks, they responded, "1 day before deadline." This is the power of the outside view over the inside view.
A similar finding is that experienced outsiders, who know less of the details, but who have relevant memory to draw upon, are often much less optimistic and much more accurate than the actual planners and implementers.
So there is a fairly reliable way to fix the planning fallacy, if you're doing something broadly similar to a reference class of previous projects. Just ask how long similar projects have taken in the past, without considering any of the special properties of this project. Better yet, ask an experienced outsider how long similar projects have taken.
You'll get back an answer that sounds hideously long, and clearly reflects no understanding of the special reasons why this particular task will take less time. This answer is true. Deal with it.
Buehler, R., Griffin, D. and Ross, M. 1994. Exploring the "planning fallacy": Why people underestimate their task completion times. Journal of Personality and Social Psychology, 67: 366-381.
Buehler, R., Griffin, D. and Ross, M. 1995. It's about time: Optimistic predictions in work and love. Pp. 1-32 in European Review of Social Psychology, Volume 6, eds. W. Stroebe and M. Hewstone. Chichester: John Wiley & Sons.
Buehler, R., Griffin, D. and Ross, M. 2002. Inside the planning fallacy: The causes and consequences of optimistic time predictions. Pp. 250-270 in Gilovich, T., Griffin, D. and Kahneman, D. (eds.) Heuristics and Biases: The Psychology of Intuitive Judgment. Cambridge, U.K.: Cambridge University Press.
Newby-Clark, I. R., Ross, M., Buehler, R., Koehler, D. J. and Griffin, D. 2000. People focus on optimistic and disregard pessimistic scenarios while predicting their task completion times. Journal of Experimental Psychology: Applied, 6: 171-182.