Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Bias in capital project decision making

40 jsalvatier 26 May 2011 06:06PM

This is a story about an odd fact about capital project decision making in engineering I noticed and how it might be related to cognitive biases

Background

Although I don't work in the field, I was trained as a chemical engineer. A chemical engineer's job is a little different than you might imagine. A chemical engineers primary job isn't to design chemical processes, they actually do relatively little chemistry, but to build, optimize and maintain industrial plants that produce chemicals (petrol products, cleaners, paint etc.) and materials that are produced similarly to chemicals (wood pulp, composite materials etc.). Questions similar to 'how fast should we mix the fluid in this reactor to make it most efficient?' or 'how can we reuse the waste heat from this process?' are much more common than questions similar to 'how can we create compound A from compound B?'.  

Chemical engineers often have to make decisions about what capital improvement projects the firm will undertake, so they must answer questions such as 'install cheap pumps that wear out quickly or the expensive ones that don't?',  'what ethanol producing bacteria is most efficient for producing ethanol?' and 'is it worth it to install a heat exchanger to recover the waste head from this process or not?'. The standard technical way of judging the profitability of an option or project is to calculate the Net Present Value (NPV) of the expected cash flows to and from the firm for each different option (installing pump type A or B, using bacteria A, B or C, installing or not installing a heat exchanger). The option with the highest NPV is the most profitable. Calculating the NPV discounts future expected cash flows for the fact that they occur in the future and you have other productive things you could do with money, such as earning interest with it. 

Oddly high discount rates

When I was in school, I noticed an odd thing: the interest rates that people used to evaluate projects on this basis, called the Minimum Acceptable Rate of Return (MARR), were often rather high, 15-50%/year. I saw this in textbook discussions and had it confirmed by several working engineers and engineering managers. My engineering economics teacher mentioned that firms often require two year "pay back periods" for projects; that annualizes to a 50% interest rate! I was very confused about this because a bank will loan a small business at ~8% interest (source) and mortgage rates are around 5% (source). This implied that many many industrial projects would be profitable if only outsiders could fund them, because investors should jump at the chance to get 15% returns. I know I would! The profit opportunity seemed so great that I started to work out business models around alternative sources of investment for industrial projects. 

Overestimating benefits, underestimating costs

To understand why MARRs are so high in chemical engineering, I tried to find research on the question and talked to experienced engineers and managers. Unfortunately, I never did find research on this topic (let me know if you know of relevant research). I talked to several managers and engineers and the most common answer I got was that investors are short sighted and primarily interested in short run profits. I didn't find this answer very plausible. Later, I met an engineer in charge of reviewing project evaluations made by other engineers in order to decide which projects would be approved. His explanation was that engineers usually overestimate the benefits of a project under consideration and underestimate the costs and that they gave engineers high MARRs in order to counterbalance this. I asked him why they didn't just apply a scaling factor to the costs and benefits and he explained that they did this a little bit, but engineers respond to this by inflating benefits and deflating costs even more! I later met another engineer who talked about doing exactly that; adjusting estimated costs down and estimated benefits up because the process evaluating projects did the reverse. 

One thing to note is that if engineers overestimate benefits, underestimate costs uniformly over time, then a high MARR will make projects which pay off in the short term artificially attractive (which is why I asked about using a scaling factor instead of a large interest rate). On the other hand, if engineers make more biased predictions about costs and benefits the further out they are in time (for example, if they tend to overestimate the productive life of equipment), then a high MARR is a more appropriate remedy.

Possible explanations

There are a couple of reasons why engineers might end to overestimate the benefits and underestimate the costs of projects. Any number of these may contribute. I suspect cognitive bias is a significant contributor. 

  1. Confirmation bias suggests engineers will tend to overestimate the benefits and underestimate the costs of projects they initially think are good ideas. The head project engineer I spoke with described a common mindset thus, 
    'And this is why we tend to focus on the goodness and diminish the badness of projects.  We know they are good so all we need to do is prove it to get the approvers bought in. Then the project approvers over time notice that these projects returns are lower than expected so they say, “Let’s raise the bar.”  But guess what?  The bar never rises.  Why?  Because we still "know" what the good projects are and all we need to do is prove they are good.'
  2. The planning fallacy suggests engineers will underestimate completion times and costs. 
  3. Overconfidence suggests engineers will underestimate costs even when explicitly accounting for uncertainty.
  4. Bad incentives: engineers may often be rewarded for spearheading projects and not punished commensurately if the project is not beneficial so that they often expect to be rewarded for spearheading a project even if they don't expect it to be a success.
Addendum: The same head project engineer suggests that one way to get better predictions, at least with respect to project duration, is to have non-technical observers make the predictions (related to taking an outside view).

Anyway I started asking our cost coordinator about predicted schedule and she is by far more accurate than the engineers with how long it takes to do a project. That has led me to think that an independent review would be a good step in project returns. Unfortunately, I have not noticed her to be any better on predicting project performance than the engineers. 

On adjusting predictions based on a track record

The problem with predicting a project will take longer than expected based on experience does not help because managers (usually engineers) want to know "why" so the can "fix it."