So... how do you measure programmer productivity? ;)

That's a good rationalist success story. You remind me of my own story with the tooth fairy: I will not relate it in detail here, as it is similar to yours, just less dramatic. At a certain point, I doubted the existence of the tooth fairy, so the next time a tooth fell out I put it under my pillow without telling anyone, and it was still there the next day. I confronted my parents, and they readily admitted the non-existence of the tooth fairy.

In fact, it went off as a perfect experiment, which kind of ruins its value as a story, at least when compared with yours. I did an experiment, got a result, and that was that. The one thing I'm still kind of bitter about is my parents' first reaction to my confrontation of them: Rather than praising me on my discovery and correct use of the scientific method, their reaction was along the lines of "If you suspected, why didn't you just tell us? We would have just admitted it. There was no need for that test to find proof to confront us with."

"But this example also helps show the limits of VoI: VoI is best suited to situations where you've done the background research and are now considering further experiments."

Do you mean this in the sense that there is usually some low-hanging fruit (e.g. the background research itself) where the VOI is obviously so high that you there's no need to calculate it -- you obviously should obtain the information?

I think Douglas Hubbard, author of How to Measure Anything, makes a good case for making VOI calculations the default in important decisions. When he acts as a consultant, he first spends a couple of days training the decision makers in calibrating their probability assessments, and then they do a VOI calculation for all the important unknowns, based on those subjective probabilities. It's often precisely those questions for which they can't get much from background research and haven't even considered measuring -- because they don't know how to -- that have the highest VOI.

Maybe these cases are atypical, as they are biased towards difficult decisions that warrant hiring a consultant. But difficult decisions are the raison d'etre for the field of decision analysis.

I read the chapter on diet. Adams claims that "Science has demonstrated that humans have a limited supply of willpower." This idea is important for at least this chapter. However as Robert Kurzban has noted it is a weak theory that cannot be falsified. I would prefer Adams to use the concept of willpower as another self-delusion to help optimize one's systems.

Thanks, I liked this post.

However, I was initially a bit confused by the section on EVPI. I think it is important, but it could be a lot clearer.

The expected opportunity loss (EOL) for a choice is the probability of the choice being “wrong” times the cost of it being wrong. So for example the EOL if the campaign is approved is $5M × 40% = $2M, and the EOL if the campaign is rejected is $40M × 60% = $24M.

The difference between EOL before and after a measurement is called the “expected value of information” (EVI).

It seems quite unclear what's meant by "the difference between EOL before and after a measurement" (EOL of which option? is this in expectation?).

I think what must be intended is: your definition is for the EOL of an option. Now the EOL of a choice is the EOL of the option we choose given current beliefs. Then EVI is the expected reduction in EOL upon measurement.

Even this is more confusing than it often needs to be. At heart it's the expected amount better you'll do with the information. Sometimes you can factor out the EOL calculation entirely. For example say you're betting $10 at even odds on a biased coin. You currently think there's a 70% chance of it landing heads; more precisely you know it was either from a batch which lands heads 60% of the time, or from a batch which lands heads 80% of the time, but these are equiprobable. You could take a measurement to find out which batch it was from. Then you are certain that this measurement will change the EOL, but if you do it carefully the expected gain is equal to the expected loss, so there is no EVI. We could spot this directly because we know that whatever the answer is, we'll bet on heads.

I think it might be useful to complete your simple example for EVPI (as in, this would have helped me to understand it faster, so may help others too): Currently you'll run the campaign, with EOL of $2M. With perfect information, you always choose the right option, so you expect the EOL to go down to 0. Hence the EVPI is $2M (this comes from the 40% of the time that the information stops you running the campaign and saving you $5M).

Then in the section on the more advanced model:

In this case, the EVPI turns out to be about $337,000. This means that we shouldn’t spend more than $337,000 to reduce our uncertainty about how many units will be sold as a result of the campaign.

Does this figure come from the book? It doesn't come from the spreadsheet you linked to. By the way, there's a mistake in the spreadsheet: when it assumes a uniform distribution it uses different bounds for two different parts of the calculation.

Tip: frame your estimates in terms of intervals with confidence levels, i.e. "90% probability that the answer is within <low end> and <high end>". Try to work out both a 90% and a 50% interval.

I've found interval estimates to be much more useful than point estimates, and they combine very well with Fermi techniques if you keep track of how much rounding you've introduced overall.

In addition, you can compute a Brier score when/if you find out the correct answer, which gives you a target for improvement.

Write down your own Fermi estimation attempts here. One Fermi estimate per comment, please!