It could be interesting to look at self-driving cars. This could be defined as any car that doesn't depend on human input for things like speed, driving accuracy, etc. I estimate that a self-driving car would reduce car crashes by something between 50% and 80%.
Since there are already pretty successful self-driving cars, I expect that the goal above would reached within 10 years if research were 10 times greater than today.
Try to give an estimate with:
Seems interesting. Paul Christiano looked into some of these figures, and without deeper investigation I'm happy to use figures from that.
It suggests R(0) = $0.5b, and B = $200B. Putting that into the spreadsheet gets a benefit:cost ratio of about 140 to 1.
A research area with a great deal of uncertainty but potentially high payoff is anti-ageing medicine. But how good is it to put more resources into?
To be concrete, let's look at the problem of being able to stop a majority of the ageing processes in cells. Let's:
So the estimate for y/z should be how many times historical efforts to solve the problem we'll need before there's a 20% total chance of success.
I think this is a particularly uncertain problem in various ways: our error bars on estimates are likely to be large, and the model is not a perfect fit. But it's also a good example of how we might begin with really no idea about how cost-effective we should think it is, and so produce a first number which can be helpful.
My estimates.
R(0): The SENS Foundation has an annual budget of around $4m, plus extra resources in the form of labour. Stem cell research has a global annual budget probably in the low billions, although it's not all directly relevant. Some basic science may be of relevance, but this is likely to be fairly tangential. Overall I will estimate $1b here, although this could be out by an order of magnitude in either direction.
B: Around 100m people die every year. It's unclear exactly what the effects of success would be on this figure, but providing a quarter of them with an extra 10 years of life seems conservative but not extremely so. So I estimate 250m QALYs/year
y/z: Real head-scratching time. I think 10 times historical resources wouldn't get us up to a 20% chance of success, but 10,000 times historical resources would be more than enough. I'm going to split the difference and say 300.
It would be simple to put it back in by taking the time derivative. By not forcing time in to begin with you can just consider total effort and probability per effort rather than having to mix in considerations of time.
So my initial model did essentially that. But you can't reasonably estimate cost-effectiveness without considering time, because the counterfactual when you don't do the work is that it gets done later -- you need an idea of how much later to know how valuable it is.
No, it's supposed to be annual spend. However it's worth noting that this is a simplified model which assumes a particular relationship between annual spend and historical spend (namely it assumes that spending has grown and will grow on an exponential).
I'm not sure I'm doing this right, but it might influence my PhD... I have a feeling that somewhere along the way I should factor in the 'new data should make your estimates more confident' thing, but in a Fermi estimate we don't take this into account, right?
A Fermi estimate of new data on Ophioglossum vulgatum bringing progress in true population estimates. (Problem: a plant with branching rhizome might give several stalks year - or might not give any, if a year is unfavourable - so you have to estimate the number of actual specimens from the number of stalks you see. Exposing and tracing the rhizome is (unethical - possibly lethal to the plant, but I doubt it if precautions are taken) the only way to know for certain. I think that if I dig up several patches of rhizomes, we can extrapolate on how many plants we have, and so have more accurate population censuses. It seems more useful if it does manage to lessen the gap between different researchers - another source of uncertainty - counting stalks.)
Let B, Annual benefit – more accurate estimates of population size (estimated number of clones/true n.o.c., measured in %) Which would be useful to estimate the probability of population going extinct within 10 years of undisturbed succession (= final benefit).
Let B be about 20%, for a start. R(0) be current resources per year, around 6 man-days. Y/Z be around 4, if we take 4 years and count time already spent on research as an annual increment.
expected benefit ≈ 0.7 20% / (6man-days 1.4) = 1.6 %/man-day, so if I actually spend this season about 30 man-days on surveying populations, it would give me 50% closer-to-truth estimates of population sizes given censuses?
For 30 man-days, I need 30 populations to survey, and I only know about around 10. Suppose I want to estimate how closeness of stalks might reveal clonal structure below, and instead of digging plants up immediately, I count the stalks, see if there are any definable patches, and then try making three guess-models of clonal structures based on that observed patchiness.
How should it influence my Fermi estimate? If the expected utility of such severe actions is too low, I'll just stick to counting stalks, and be content with less precize data. Thank you.
There's a continuum between Fermi estimates and more detailed models. At some resolution you'd definitely want to take into account the fact that new data will affect your confidence, but it may not be worth modelling at that resolution unless you think that this is one of the major routes to value.
With the scenario you outline, I think B is under-specified. You just say "more accurate estimates of population size" -- in order to get this model to work you need some way of expressing how big a change in accuracy you're looking at.
I'd also be wary of assuming that the only work that has occurred is the stuff you can directly count. Other scientists working on related problems might have produced a generalisable solution which would work here. To the extent that they haven't, we should be more pessimistic about your chances of success than we would otherwise.
Thank you. By B=20% I mean that I will be 20% more certain of my estimate of the true number of single plants when I find a new population, count the stalks and roughly check how clustered they are, compared to 'how confident I would be without this research'. I will certainly look into works on other plants. I think people just don't bother. We don't need to know exactly how many specimens are in a spot if we can say that mowing makes the environment more favourable to stalk production. We cannot really say much about genetic diversity and long-term conservation strategies, but considering that nobody is going to implement those strategies... It is, however, of some interest as to how such an ancient plant 'games the system' of the world we have - it is largely inbred, always must live with a fungus of some specificity, always 'on the move' (shrubbery incursion makes it die off, so it must produce spores before its window of opportunity is closed), glaciations have nudged it into retreats... and it still survives. It's just an awesome little thing. *end of rant:)
It's been argued that open borders could give a massive boost to the world economy. Let's see what happens if we try to apply this cost-effectiveness model to the problem of successfully lobbying for open borders. (Lack of inevitability means the model is not ideally suited compared to with research problems, but it still seems like a reasonable position.)
We'll look at the problem of getting to a political situation which permits emigration levels of 5% of the population of poor countries. This might produce a 20% increase in the population of rich countries.
Let's:
My estimates:
R(0): I'm very unsure. It seems like it's at least in the hundreds of millions of dollars, and not higher than tens of billions of dollars. So I will guess $5 billion. I've put little research into this and this number could easily update a lot.
B: Based on some of the estimates in this paper, emigration of 5% might add in the region of $2.5 trillion to the world economy.
y/z: If $1,000 of resources were dedicated annually to this for each of the ~2 billion people living in the rich world, I'd be happy that there was a significant chance of success. So I'll estimate that y/z = 2000/5 = 400.
In some recent work (particularly this article) I built models for estimating the cost effectiveness of work on problems when we don’t know how hard those problems are. The estimates they produce aren’t perfect, but they can get us started where it’s otherwise hard to make comparisons.
Now I want to know: what can we use this technique on? I have a couple of applications I am working on, but I’m keen to see what estimates other people produce.
There are complicated versions of the model which account for more factors, but we can start with a simple version. This is a tool for initial Fermi calculations: it’s relatively easy to use but should get us around the right order of magnitude. That can be very useful, and we can build more detailed models for the most promising opportunities.
The model is given by:
This expresses the expected benefit of adding another unit of resources to solving the problem. You can denominate the resources in dollars, researcher-years, or another convenient unit. To use this formula we need to estimate four variables:
R(0) denotes the current resources going towards the problem each year. Whatever units you measure R(0) in, those are the units we’ll get an estimate for the benefit of. So if R(0) is measured in researcher-years, the formula will tell us the expected benefit of adding a researcher year.
You want to count all of the resources going towards the problem. That includes the labour of those who work on it in their spare time, and some weighting for the talent of the people working in the area (if you doubled the budget going to an area, you couldn’t get twice as many people who are just as good; ideally we’d use an elasticity here).
Some resources may be aimed at something other than your problem, but be tangentially useful. We should count some fraction of those, according to how much resources devoted entirely to the problem they seem equivalent to.
B is the annual benefit that we’d get from a solution to the problem. You can measure this in its own units, but whatever you use here will be the units of value that come out in the cost-effectiveness estimate.
p and y/z are parameters that we will estimate together. p is the probability of getting a solution by the time y resources have been dedicated to the problem, if z resources have been dedicated so far. Note that we only need the ratio y/z, so we can estimate this directly.
Although y/z is hard to estimate, we will take a (natural) logarithm of it, so don’t worry too much about making this term precise.
I think it will often be best to use middling values of p, perhaps between 0.2 and 0.8.
And that’s it.
Example: How valuable is extra research into nuclear fusion? Assume:
R(0) = $5 billion (after a quick google turns up $1.5B for current spending, and adjusting upwards to account for non-financial inputs);
B = $1000 billion (guesswork, a bit over 1% of the world economy; a fraction of the current energy sector);
There’s a 50% chance of success (p = 0.5) by the time we’ve spent 100 times as many resources as today (log(y/z) = log(100) = 4.6).
Putting these together would give an expected societal benefit of (0.5*$1000B)/(5B*4.6) = $22 for every dollar spent. This is high enough to suggest that we may be significantly under-investing in fusion, and that a more careful calculation (with better-researched numbers!) might be justified.
Caveats
To get the simple formula, the model made a number of assumptions. Since we’re just using it to get rough numbers, it’s okay if we don’t fit these assumptions exactly, but if they’re totally off then the model may be inappropriate. One restriction in particular I’d want to bear in mind:
It should be plausible that we could solve the problem in the next decade or two.
It’s okay if this is unlikely, but I’d want to change the model if I were estimating the value of e.g. trying to colonise the stars.
Request for applications
So -- what would you like to apply this method to? What answers do you get?
To help structure the comment thread, I suggest attempting only one problem in each comment. Include the value of p, and the units of R(0) and units of B that you’d like to use. Then you can give your estimates for R(0), B, and y/z as a comment reply, and so can anyone else who wants to give estimates for the same thing.
I’ve also set up a google spreadsheet where we can enter estimates for the questions people propose. For the time being anyone can edit this.
Have fun!