In some recent work (particularly this article) I built models for estimating the cost effectiveness of work on problems when we don’t know how hard those problems are. The estimates they produce aren’t perfect, but they can get us started where it’s otherwise hard to make comparisons.

Now I want to know: what can we use this technique on? I have a couple of applications I am working on, but I’m keen to see what estimates other people produce.

There are complicated versions of the model which account for more factors, but we can start with a simple version. This is a tool for initial Fermi calculations: it’s relatively easy to use but should get us around the right order of magnitude. That can be very useful, and we can build more detailed models for the most promising opportunities.

The model is given by:

This expresses the expected benefit of adding another unit of resources to solving the problem. You can denominate the resources in dollars, researcher-years, or another convenient unit. To use this formula we need to estimate four variables:

• R(0) denotes the current resources going towards the problem each year. Whatever units you measure R(0) in, those are the units we’ll get an estimate for the benefit of. So if R(0) is measured in researcher-years, the formula will tell us the expected benefit of adding a researcher year.

  • You want to count all of the resources going towards the problem. That includes the labour of those who work on it in their spare time, and some weighting for the talent of the people working in the area (if you doubled the budget going to an area, you couldn’t get twice as many people who are just as good; ideally we’d use an elasticity here).

  • Some resources may be aimed at something other than your problem, but be tangentially useful. We should count some fraction of those, according to how much resources devoted entirely to the problem they seem equivalent to.

• B is the annual benefit that we’d get from a solution to the problem. You can measure this in its own units, but whatever you use here will be the units of value that come out in the cost-effectiveness estimate.

• p and y/z are parameters that we will estimate together. p is the probability of getting a solution by the time y resources have been dedicated to the problem, if z resources have been dedicated so far. Note that we only need the ratio y/z, so we can estimate this directly.

  • Although y/z is hard to estimate, we will take a (natural) logarithm of it, so don’t worry too much about making this term precise.

  • I think it will often be best to use middling values of p, perhaps between 0.2 and 0.8.

And that’s it.

Example: How valuable is extra research into nuclear fusion? Assume:

• R(0) = $5 billion (after a quick google turns up $1.5B for current spending, and adjusting upwards to account for non-financial inputs);

• B = $1000 billion (guesswork, a bit over 1% of the world economy; a fraction of the current energy sector);

• There’s a 50% chance of success (p = 0.5) by the time we’ve spent 100 times as many resources as today (log(y/z) = log(100) = 4.6).

Putting these together would give an expected societal benefit of (0.5*$1000B)/(5B*4.6) = $22 for every dollar spent. This is high enough to suggest that we may be significantly under-investing in fusion, and that a more careful calculation (with better-researched numbers!) might be justified.

Caveats

To get the simple formula, the model made a number of assumptions. Since we’re just using it to get rough numbers, it’s okay if we don’t fit these assumptions exactly, but if they’re totally off then the model may be inappropriate. One restriction in particular I’d want to bear in mind:

• It should be plausible that we could solve the problem in the next decade or two.

It’s okay if this is unlikely, but I’d want to change the model if I were estimating the value of e.g. trying to colonise the stars.

Request for applications

So -- what would you like to apply this method to? What answers do you get?

To help structure the comment thread, I suggest attempting only one problem in each  comment. Include the value of p, and the units of R(0) and units of B that you’d like to use. Then you can give your estimates for R(0), B, and y/z as a comment reply, and so can anyone else who wants to give estimates for the same thing.

I’ve also set up a google spreadsheet where we can enter estimates for the questions people propose. For the time being anyone can edit this.

Have fun!

Developing rational patterns of thought in children is very important and I'm glad Gunnar brought that issue up.

I wanted to share with you some thoughts I have regarding estimation games.

From an early age I've been constantly calculating various kinds of estimates - e.g. "how many people live in this building", "how long will it take to cross the US on foot", "what's the height of that tower", "how many BMWs are manufactured annually" and so on.

I believe that practising this technique is not only fun but also helpful. Sometimes one has no way or time to acquire accurate information regarding something and even a rough estimate can be very valuable. 

People are often surprised when they see me do it whereas for me it is completely natural. I think the reason is that I do it from a very early age.

I think it's easy and natural for children to grasp if this method is introduced through everyday experiences. By making this into a game children can gain intuitive understanding of quantitative techniques. I suspect many children can enjoy this kind of games.

I'd like to hear your thoughts on the subject.

Do you remember yourself doing something like this? From what age? Do you practice anything similar with your children?

Fermi problem

In science, particularly in physics or engineering education, a Fermi problem, Fermi question, or Fermi estimate is an estimation problem designed to teach dimensional analysis, approximation, and the importance of clearly identifying one's assumptions. Named after physicist Enrico Fermi, such problems typically involve making justified guesses about quantities that seem impossible to compute given limited available information.

Fermi was known for his ability to make good approximate calculations with little or no actual data, hence the name. One example is his estimate of the strength of the atomic bomb detonated at the Trinity test, based on the distance travelled by pieces of paper dropped from his hand during the blast. Fermi's estimate of 10 kilotons of TNT was remarkably close to the now-accepted value of around 20 kilotons, a difference of less than one order of magnitude.

[...]

Scientists often look for Fermi estimates of the answer to a problem before turning to more sophisticated methods to calculate a precise answer. This provides a useful check on the results: where the complexity of a precise calculation might obscure a large error, the simplicity of Fermi calculations makes them far less susceptible to such mistakes. (Performing the Fermi calculation first is preferable because the intermediate estimates might otherwise be biased by knowledge of the calculated answer.)

Fermi estimates are also useful in approaching problems where the optimal choice of calculation method depends on the expected size of the answer. For instance, a Fermi estimate might indicate whether the internal stresses of a structure are low enough that it can be accurately described by linear elasticity; or if the estimate already bears significant relationship in scale relative to some other value, for example, if a structure will be over-engineered to withstand loads several times greater than the estimate.

Although Fermi calculations are often not accurate, as there may be many problems with their assumptions, this sort of analysis does tell us what to look for to get a better answer.

Link: en.wikipedia.org/wiki/Fermi_problem

Fermi Problem: Power developed at the eruption of the Puyehue-Cordón Caulle volcanic system in June 2011

Enrico Fermi was renowned for his ability to make reliable estimates. But how well can you do on a modern estimation problem?

[...]

Hernan Asory and Arturo Lopez Davalos at the Comision Nacional De Energia Atomica in Argentina, have set themselves (and their students) a similar estimation task. The problem is to estimate the energy release as well as the volume and mass of sand ejected during the eruption of the Puyehue-Cordon Caulle volcano in Chile on 4 July.

You can look up the calculations and the assumption they make in the paper. You might want to try the estimate yourself.

Link: technologyreview.com/blog/arxiv/27140/

If you want to get better at doing rough mental calulcations, the following books might provide some valuable heuristics:

Street-Fighting Mathematics: The Art of Educated Guessing and Opportunistic Problem Solving

Time for some quick arithmetic: Is 3600 x 4.4 x 10⁴ x 32 larger or smaller than 3 x 10⁹?

Finding the right answer, says Sanjoy Mahajan, associate director for teaching initiatives at MIT’s Teaching and Learning Laboratory, does not require crafting a long, tedious calculation. Instead, the key to solving this problem — and many others — lies in having informal tools on hand that let us attack the problem. Though the result may not be perfectly precise, he believes, intuitive mathematical reasoning is often sufficient for our needs.

“That’s not to say exact answers aren’t useful,” says Mahajan, “but if looking for them is your only approach, you may never get any answer at all. Sometimes it’s better to start with something rough.”

[...]

Mahajan believes we should learn practical math tools and understand why they work.

[...]

Mahajan’s unconventional teaching practices stem from his focus, as a physicist, on finding quick, practical answers. Then again, perhaps rolling up one’s sleeves and hacking through problems is how everyone works. “There is a culture in pure mathematics that emphasizes rigor and careful proofs,” says Strogatz. “Yet all practicing mathematicians know we also use our intuitions, then we clean our answers up.”

[...]

So let’s get back to the initial question (the numbers relate to the storage capacity of a data CD-ROM). The key to solving it, says Mahajan, is to recognize that the components of the first, messy-looking number can be broken into powers of 10. Then we can temporarily set aside these powers of 10 — Mahajan calls this “taking out the big part,” one of his tenets of problem-solving — while handling the smaller, simpler multiplication problem.

Okay: Picture the number as (3.6 x 10³) x (4.4 x 10⁴) x (3.2 x 10¹). To multiply powers of 10 in practice, we add them, here producing 10⁸. Leave that aside momentarily and multiply 3.6 x 4.4 x 3.2. The answer is about 50, or 5.0 x 10¹. Combine that with 10⁸, and we have our answer: Roughly 5.0 x 10⁹, which is bigger than 3 x 10⁹. Street-fighting math, and we barely got a scratch. 

Link: web.mit.edu/newsoffice/2010/street-fight-0329.html

Secrets of Mental Math: The Mathemagician's Guide to Lightning Calculation and Amazing Math Tricks

Yes, even you can learn to do seemingly complex equations in your head; all you need to learn are a few tricks. You’ll be able to quickly multiply and divide triple digits, compute with fractions, and determine squares, cubes, and roots without blinking an eye. No matter what your age or current math ability, Secrets of Mental Math will allow you to perform fantastic feats of the mind effortlessly. This is the math they never taught you in school.

So you think its important to be able to estimate how well you are estimating something?  Here is a fun test that has been given to plenty of other people.  

I highly recommend you take the test before reading any more.  

http://www.codinghorror.com/blog/2006/06/how-good-an-estimator-are-you.html

The discussion of this test at the blog it is quoted in is quite interesting, but I recommend you read it after taking the test.  Similarly, one might anticipate there will be interesting discussion here on the test and whether it means what we want it to mean and so on.  

My great apologies if this has been posted before.  I did my bast with google trying to find any trace of this test, but if this has already been done, please let me know and ideally, let me know how I can remove my own duplicate post.

PS: The Southern California meetup 19 Dec 2010 was fantastic, thanks so much JenniferRM for setting it up.  This post on my part is an indirect result of what we discussed and a fun game we played while we were there.