Prioritisation is mostly about working out how to trade different resources off against one another. Prioritisation problems come at different scales: for individuals, for companies or organisations, for the world at large. At the Global Priorities Project we’re mostly interested in the large-scale questions. But we sometimes have something to say about smaller scale problems, too.

I’ve just tidied and released old research notes (mostly from 2013) on the personal prioritisation problem of how to value time spent on different activities. This is primarily of use for individuals making decisions about how to spend their time, money, and mental energy.

Abstract: We get lots of opportunities to convert between time and money, and it’s hard to know which ones to take, since they use up other mental resources. I introduce the neutral hour as a tool for thinking about how to make these comparisons. A neutral hour is an hour spent where your mental energy is the same level at the start and the end. I work through some examples of how to use this tool, look at implications for some common scenarios, and explore the theory behind them.

There may be benefits for broader prioritisation questions. Since societies are comprised of individuals, it could help to know how to value time savings or costs to individuals when performing cost-benefit analysis on larger projects. And there may be techniques for comparing between different resources that we could usefully apply in wider contexts. However we think these benefits are secondary. We’re releasing this work now to let others take advantage of it: either for personal benefit; or to build on it and release easier-to-use guidance or tools.

You can find the full document here. I'm happy to answer questions and I'd love to know if people have thoughts on this material.

I've just released a Future of Humanity Institute technical report, written as part of the Global Priorities Project.

Abstract:

This article is about priority-setting for work aiming to reduce existential risk. Its chief claim is that all else being equal we should prefer work earlier and prefer to work on risks that might come early. This is because we are uncertain about when we will have to face different risks, because we expect diminishing returns of extra work, and because we expect that more people will work on these risks in the future.

I explore this claim both qualitatively and with explicit models. I consider its implications for two questions: first, “When is it best to do different kinds of work?”; second, “Which risks should we focus on?”.

As a major application, I look at the case of risk from artificial intelligence. The best strategies for reducing this risk depend on when the risk is coming. I argue that we may be underinvesting in scenarios where AI comes soon even though these scenarios are relatively unlikely, because we will not have time later to address them.

You can read the full report here: Allocating risk mitigation across time.

Summary: We can split the cost-effectiveness of an intervention into how good the cause is, and how good the intervention is relative to the cause. This perspective could help our efforts in prioritisation by letting us bring appropriate tools to bear on the different parts.

Cost-effectiveness comparisons

When we choose between giving time or money to different interventions, we’re making a comparison. It’s nice to know what these comparisons come down to. There are a lot of sources of evidence, and different ones will be more appropriate in different contexts. For this post I'll assume that we are seeking the most cost-effective interventions.

Say we are comparing between intervention x in cause area X, and intervention y in cause area Y. How they compare depends on things like how well thought-out x and y are, how competent the people and organisations implementing them are, as well as how valuable X is as a whole compared to Y.

These are all important factors in telling us how x and y ultimately compare, but they’re quite different from one another. So it shouldn’t be a surprise if it’s best to use different methods to compare the different factors. I think this is the case.

Consider the equation:

Cost-effectiveness of intervention = (cost-effectiveness of area) * (leverage ratio of intervention)

The left-hand side of this equation expresses how much good is achieved per unit of resources invested in the intervention. For the intervention x we’ll denote this G(x). The right-hand side breaks this up as C(X), how much good is achieved per unit of resources invested in X as a whole, and a ‘leverage ratio’ L(x) which expresses the ratio of how effective x is compared to X as a whole [1].

Now to compare between x and y we’re interested in the ratio G(x)/G(y). We can use the above equation to expand this:

G(x)/G(y) = (C(X)L(x))/(C(Y)L(y)).

This rearranges to:

G(x)/G(y) = C(X)/C(Y) * L(x)/L(y).

Here we’ve split the comparison into two parts, each of which is comparing like with like. This is a good general strategy: making comparisons between dissimilar things is hard, and our intuitions are sometimes terrible at it, so it’s helpful to break it into more comparable chunks [2].

Comparing cause effectiveness

Comparing the cost-effectiveness of different cause areas is quite far removed from everyday experience, and it doesn’t have a good feedback mechanism as it can be hard to tell how much something helped the world even after the fact. Moreover it’s just the right setting for scope insensitivity to cause problems for intuitive judgements. This means that relative to most areas of experience, we should be particularly cautious about putting too much weight on intuitive judgements. This in turn means that it’s an area where explicit models are particularly valuable.

That doesn’t mean that explicit models always trump intuitive judgements in this domain; in particular, simple models often omit important factors that are incorporated into our intuitions. Nor does it mean that we should put all our trust into a single model. But it does mean that it’s particularly valuable to build, critique, and refine models for the cost-effectiveness of different causes. It also means we should put more weight on the outputs of such models than we do in most domains -- not because the models are more trustworthy, but because the alternatives are worse than usual. This is why I think developing such models is a high value activity, and why I’ve been spending time on it.

Comparing leverage ratios

The leverage ratios are determined largely by things like: whether the intervention is a sensible way of progressing on the cause; the quality of the team involved; how functional the implementing organisation is. In contrast to the overall effectiveness of a cause, these are the much closer to regular experience, so we should be less keen to use explicit models. On the other hand, methods and experience from valuing shares of companies (which have good feedback mechanisms) should be relevant in this context.

There are several reasons why leverage ratios may vary within a cause area. Many of these will be common across cause areas. Because of this, we might expect similar distributions of leverage ratios in different cause areas (but probably some areas have more variance than others, just as some jobs have more variance in the productivity of employees than others). It could be valuable to have an idea of how much leverage ratios do vary in practice. This is an empirical question we might be able to get data for.

Implications for prioritisation work

To choose between interventions, we need to compare cost-effectiveness. I’ve claimed that this is best done by comparing the cost-effectiveness of cause areas, and comparing the leverage ratios of the interventions. If this is right, what’s more valuable to work on evaluating?

Of course they are complementary to each other. The better we are able to identify the best cause areas, the more valuable it is to have good estimates for the leverage ratios of interventions in those areas. And the better we are able to identify interventions with very high leverage ratios, the more valuable it is to be able to say which of those are in the most effective causes. So the answer depends in part on how much work each is already receiving.

It also depends on your beliefs about which component has more variance. If you think that most of the variation in intervention effectiveness comes from leverage ratios, while cost-effectiveness of causes doesn’t vary that much, then it’s more important to evaluate the leverage ratios of interventions. If on the other hand you think more variation comes between causes, then it’s more important to evaluate cause effectiveness. I currently think there is likely to be more variation in cause effectiveness even after you filter to the ones which could plausibly be high value; however I am quite uncertain about this.

There is also an asymmetry which pushes us towards doing more cause assessment first: it’s much easier to cut down the work of evaluating leverage ratios by restricting to a few causes than it is to cut down the work of evaluating cause effectiveness by first identifying opportunities with high leverage ratios. Similarly, if we identify a cause area which is valuable but see no good interventions available to fund, we can advertise this and hopefully create good interventions in the area.

Of course to support giving decisions today we need to compare leverage ratios as well as cause effectiveness. And in some cases studying the interventions may help us to evaluate the cause effectiveness. But I think it will usually be right to investigate leverage ratios only within cause areas that we think have, or might have, high effectiveness, and only after we’ve made an effort to assess that.

Acknowledgements: thanks to Toby Ord and Nick Beckstead for helpful conversations.

Crossposted from the Global Priorities Project.

[1] The leverage ratio is really a function of x together with X. AMF may have one leverage ratio with respect to the area of global health, and another with respect to malaria treatment.

[2] An extra advantage of breaking the comparisons into like-with-like is that it’s easier to track uncertainty so that it doesn’t blow up unnecessarily. I might be very uncertain about how good X is, so I think C(X) lies somewhere in (1, 100). I might also be very uncertain about how good Y is, so that I think C(Y) lies in (1, 100). But it doesn’t follow that C(X)/C(Y) could lie anywhere in (1/100, 100). If my uncertainty about X is related to my uncertainty about Y (say X is reducing carbon emissions and Y is helping communities adapt to climate change), then I might have a better idea of the ratio C(X)/C(Y) than I do about either individually. Of course this just means that my estimates for C(X) and C(Y) are strongly correlated. But I think it’s helpful to have an idea of practical ways to break up the calculation which help to keep the uncertainty under control. For more thoughts on tracking uncertainty through estimates, see here.

At a societal level, how much money should we put into medical research, or into fusion research? For individual donors seeking out the best opportunities, how can we compare the expected cost-effectiveness of research projects with more direct interventions?

Over the past few months I've been researching this area for the Global Priorities Project. We've written a variety of articles which focus on different parts of the question. Estimating the cost-effectiveness of research is the central example here, but a lot of the methodology is also applicable to other one-off projects with unknown difficulty (perhaps including political lobbying). I don't think it's all solved, but I do think we've made substantial progress.

I think people here might be interested, so I wanted to share our work. To help you navigate and find the most appropriate pieces, here I collect them, summarise what's contained in each, and explain how they fit together.

• I gave an overview of my thinking at the Good Done Right conference, held in Oxford in July 2014. The slides and audio of my talk are available; I have developed more sophisticated models for some parts of the area since then.
• How to treat problems of unknown difficulty introduces the problem: we need to make decisions about when to work more on problems such as research into fusion where we don't know how difficult it will be. It builds some models which allow principled reasoning about how we should act. These models are quite crude but easy to work with: they are intended to lower the bar for Fermi estimates and similar, and provide a starting point for building more sophisticated models.
• Estimating cost-effectiveness for problems of unknown difficulty picks up from the models in the above post, and asks what they mean for the expected cost-effectiveness of work on the problems. This involves building a model of the counterfactual impact, as solvable research problems are likely to be solved eventually, so the main effect is to move the solution forwards. This post includes several explicit formulae that you can use to produce estimates; it also explains analogies between the explicit model we derive and the qualitative 'three factor' model that GiveWell and 80,000 Hours have used for cause selection.
• Estimating the cost-effectiveness of research into neglected diseases is an investigation by Max Dalton, which uses the techniques for estimating cost-effectiveness to provide ballpark figures for how valuable we should expect research into vaccines or treatments for neglected diseases to be. The estimates suggest that, if carefully targeted, such research could be more cost-effective than the best direct health interventions currently available for funding.
• The law of logarithmic returns discusses the question of returns to resources into a field rather than on a single question. With some examples, it suggests that as a first approximation it is often reasonable to assume that diminishing marginal returns take a logarithmic form.
• Theory behind logarithmic returns explains how some simple generating mechanisms can produce roughly logarithmic returns. This is a complement to the above article: we think having both empirical and theoretical justification for the rule helps us to have higher confidence in it, and to better understand when it's appropriate to generalise to new contexts. In this piece I also highlight areas for further research on the theoretical side, into when the approximation will break down, and what we might want to use instead in these cases.
• How valuable is medical research? written with Giving What We Can, applies the logarithmic returns model together with counterfactual reasoning to produce an estimate for the cost-effectiveness of medical research as a whole.