If you’re deciding what (research) project, organization or intervention to go for, analyzing your options through prioritization research can be invaluable. I used it to settle on founding Catalyze, an AI Safety field-building non-profit. In this post, I will share my blueprint and learnings from this process. Please note that you don’t need to be a researcher to benefit from conducting prioritization research.
Why prioritization research
No need to wait for inspiration to strike
If you're at a crossroads trying to come up with a great idea, I have good news: you don't have to invent something new. There's a world of ideas waiting to be discovered and executed. So don't wait for that idea to come to you; instead, you can go to the ideas.
You can probably find a better intervention than the first one(s) you stumbled upon
Suppose you have a handful of ideas for a project or an organization. That’s a fantastic start! But consider the possibility that by conducting a broad and deliberate search, you could stumble upon something even more impactful. More likely than not, if you consider a wide range of ideas, your initial favorite won’t come out on top. Don’t go for the first thing that came to mind, but rather take some time to scope the field and possibly find an even better option.
The difference in impact between a ‘good’ and ‘great’ intervention is huge - especially within a high-impact cause area
One of the original points EA focused on was the huge difference in impact between different interventions within global health.
To me, it seems likely that the big difference in impact between interventions is similar in different cause areas.
Therefore, apart from wisely choosing what cause to work on, I think it is still very crucial to wisely pick what you work on within that cause area. One of the ‘best’ interventions in this area is likely many times more impactful than the median.
If you work within a cause area that is especially high-impact, then finding one of the very best things to do could be especially consequential. Within such a cause area, the median intervention might already make a very big difference. Don’t let this be encouragement to settle - instead let this be encouragement to find an outlier which is 10x as impactful as this already-great median intervention.
In other words, don’t fall into the “it’s-in-an-EA-cause-area-so-it’s-high-impact-anyways” fallacy (I’ll have to work on the naming). Although at this point it might all just feel ‘very impactful’, don’t forget that the Scope Insensitivity Bias might be clouding your judgement.
Get better feedback and advice by reasoning explicitly & strengthen your skills
The skills you can learn through prioritization research can for example be useful if you’d like to move into a grantmaker role or research role. I’ve personally also found it useful in my impact-focused entrepreneurship endeavours.
My blueprint for prioritization research
Below I’ll describe the steps I took to decide roughly what intervention I wanted to build an organization around. In deciding what these steps looked like, I largely took inspiration from Charity Entrepreneurship’s (CE) materials (their book and information I could find on their website about their approach).
I by no means want to claim that the method I put together below is the best way to do prioritization research. Instead, I’d like to give readers some place to start from more quickly by sharing the specific steps and templates I put together. I think that finding materials like these would have helped me to get going more quickly and improve my methods. I encourage others to iterate and adjust the approach for themselves, and to share in the comments how they would change it. I’ll also recommend some things you can change about the process below depending on your situation (e.g. how much time you have available).
As you proceed, your list of ideas becomes more concise, like a funnel. In the beginning, your approach should be broad, collecting a wide array of ideas. As you progress, the number of interventions narrows, and the depth of your research increases. This funnel process ensures that, by the end, you are left with a few good and extensively researched options.
Step 1: Consider and collect lots of ideas on a longlist
You can collect the ideas you think are somewhat promising on a longlist. The ideas you consider can come from various sources, including posts online, that ‘ideas’ note in your phone, and brainstorming sessions.
You can also try to source ideas from people who are experienced in your field(s) of choice and you trust. They might have a better sense of the important gaps in the field than someone from the outside. Referring to experts’ intuitions can be especially useful if there is little evidence to lean on or if you have little time to investigate the field yourself.
At this stage, some advice I have is: try to go for speed, not perfection. Your bar for including an idea shouldn’t be too high; separate the process of ideation and selection. You probably don’t have time to research the ideas before putting them on your longlist. You will do this later. Just go by your intuition: does this intervention sound like something I might want to do? And does this sound like it could have a lot of impact?
Personally, I aimed to gather around 100 ideas on my longlist and landed at around 85.
If your time or scope is limited, consider instead collecting 20, 40, or 60 ideas, just see what is realistic and useful. Then, instead of going from top 100 -> top 10 -> top 3, you can also go from top 20 -> top 6- > top 2 or any other variation you find sensible.
Your longlist (here’s a template so you can set up your longlist in a way that makes step 2 easier).
Step 2: Reduce to a top 10
Now, you can transition to a more analytical phase using a Weighted Factor Model (WFM). In short, a WFM lets you rank a list of options by the set of criteria you determine (e.g. impact, motivation, personal fit). You give the criteria different ‘weights’ which depend on how important you think the criteria are.
An example from Charity Entrepreneurship’s article on Weighted Factor Models
In this stage it’s about finding a balance – if you have too many criteria, ranking items in your longlist will take a long time. So focus on what truly matters to you, and keep in mind that this is just initial filtering. You’ll consider more criteria once you have a shorter list of options.
I’d also encourage you to pre-determine how much time you’re willing to spend on each idea at this stage. I would roughly propose: take around 1-4 mins per idea if you have 80-100 interventions to go through. You could take a bit longer if you have less ideas to go through. You could then do some more quick research to take away major uncertainties.
To check my reasoning, I asked 4 people who know me (so they can judge my personal fit), who know the domains of the ideas, and who I think have good judgement to look at the top 30 in my longlist document. I asked them: 1) which ratings would you change, and 2) are there any interventions in the top 30 that you feel should be in the top 10?
Alternatively, you could also ask people to individually rate the ideas on different dimensions. This way you can make sure they don’t get influenced by your or others’ ratings.
If your interventions are within a specific domain, try asking experts in that domain to help you assess the ideas.
Resources:
Yourtrusty Weighted Factor Model - this a template (the same as I linked in step 1).
CE’s explanation of how to make a Weighted Factor Model.
Step 3: Reduce to a top 3
At this stage, I’d recommend investing more serious amounts of time into each of your top 10 ideas. Personally, I spent 4-6 hrs on researching each of the top 10 ideas. You can dive into each option with more in-depth analysis reports. The main things I tried to focus on in these reports is the evidence and quality of evidence for intervention x and the most likely reasons intervention x might not be feasible or impactful.
At this point, you could also create a more refined WFM which considers additional or more granular criteria. The reports you write can inform how you rate different facets of the intervention. For instance, as you learn more about the intervention you might learn that it seems much harder than you expected, and you might give it a low rating on your ‘feasibility’ criterion. Personally, I added some criteria into my WFM that I did not actually weigh because I was too uncertain about them. For example, I didn’t go into detail regarding impact in the WFM because I didn’t feel I could make a numerically accurate estimate. Instead, that was the focus of the reports I wrote on the interventions.
Something that was useful for me was spending 1-2 hrs discussing the options in your top 10 with a few experts in the field. They can give you feedback on your ideas and tell you why they think certain ones are more or less promising. You can also ask a few other people to attempt to independently fill out part of your WFM. You can then take their intuitions into account as you refine your own.
Taking into consideration the WFM and the reports, you can now pick which few ideas you think are most promising, which brings you down to your top 3.
Resources:
A list of questions to guide your more in-depth research (this is a template you can use for this).
For this, you can conduct additional research into your top 3. I spent around 15-20 additional hrs on each, building on the initial report I had written for them. My focus here was on hashing out uncertainties. At this stage, talking to experts regarding those interventions is very useful.
After this, I picked the intervention I found most promising. This is only a starting point though, from here many more hours go into refining choices within that and iterating on your plans.
Some tips to keep in mind
The trade-off between depth and breadth
Be mindful of the trade-offs involved in your approach. Starting with a narrow focus, such as specifically prioritizing between anti-factory farming interventions, can yield more precise outcomes. However, this also means you might overlook other potentially impactful areas. Consider how you want to strike this balance between depth and breadth.
Decide what time commitment makes sense and hold on for dear life to time-capping
From my own experience I can tell you that it is easy to spend way too much time on this. You will discover a million uncertainties you need to look into to feel like you’re making an informed choice. This makes it easy to get stuck in ‘analysis paralysis’. Try to prevent this by pre-committing to how much time you want to invest into this research phase - and then sticking to it (a.k.a. time-capping).
You’ll have to trust in the 80/20 principle: 80% of results come from 20% of the efforts. This involves making choices even when you feel like you need more information to do so.
How much time you put into this should probably vary depending on how big the future commitment is - are you starting a new org or just a 3-month research project?
It can also vary depending on how important it is that you identify a very impactful next step. For instance, if you want to just learn a lot in your upcoming project, the impact of the project might matter less to you.
Seek feedback
Throughout every stage of this process, I encourage you to actively seek out feedback from trusted individuals. Their insights can help you identify blind spots in your reasoning and approach.
Find someone to do this with
I would recommend going through this process together with a small group of people. I didn’t but would if I were to do it again. This can help you stay on track, stick to the time-capping, and stay motivated. If you don’t know where to find your prioritization research partner, you can start by just posting a message about it in the EA Anywhere Slack channel.
Many thanks to Marie Buhl for the useful additions and input, and thank you to Gábor Szórád, Veerle Korsten and Bryce Robertson for providing feedback on drafts of this post.
If you’re deciding what (research) project, organization or intervention to go for, analyzing your options through prioritization research can be invaluable. I used it to settle on founding Catalyze, an AI Safety field-building non-profit. In this post, I will share my blueprint and learnings from this process. Please note that you don’t need to be a researcher to benefit from conducting prioritization research.
Why prioritization research
No need to wait for inspiration to strike
If you're at a crossroads trying to come up with a great idea, I have good news: you don't have to invent something new. There's a world of ideas waiting to be discovered and executed. So don't wait for that idea to come to you; instead, you can go to the ideas.
You can probably find a better intervention than the first one(s) you stumbled upon
Suppose you have a handful of ideas for a project or an organization. That’s a fantastic start! But consider the possibility that by conducting a broad and deliberate search, you could stumble upon something even more impactful. More likely than not, if you consider a wide range of ideas, your initial favorite won’t come out on top. Don’t go for the first thing that came to mind, but rather take some time to scope the field and possibly find an even better option.
The difference in impact between a ‘good’ and ‘great’ intervention is huge - especially within a high-impact cause area
One of the original points EA focused on was the huge difference in impact between different interventions within global health.
Illustrative Graph from this 80.000 Hours article
To me, it seems likely that the big difference in impact between interventions is similar in different cause areas.
Therefore, apart from wisely choosing what cause to work on, I think it is still very crucial to wisely pick what you work on within that cause area. One of the ‘best’ interventions in this area is likely many times more impactful than the median.
If you work within a cause area that is especially high-impact, then finding one of the very best things to do could be especially consequential. Within such a cause area, the median intervention might already make a very big difference. Don’t let this be encouragement to settle - instead let this be encouragement to find an outlier which is 10x as impactful as this already-great median intervention.
In other words, don’t fall into the “it’s-in-an-EA-cause-area-so-it’s-high-impact-anyways” fallacy (I’ll have to work on the naming). Although at this point it might all just feel ‘very impactful’, don’t forget that the Scope Insensitivity Bias might be clouding your judgement.
Get better feedback and advice by reasoning explicitly & strengthen your skills
The skills you can learn through prioritization research can for example be useful if you’d like to move into a grantmaker role or research role. I’ve personally also found it useful in my impact-focused entrepreneurship endeavours.
My blueprint for prioritization research
Below I’ll describe the steps I took to decide roughly what intervention I wanted to build an organization around. In deciding what these steps looked like, I largely took inspiration from Charity Entrepreneurship’s (CE) materials (their book and information I could find on their website about their approach).
I by no means want to claim that the method I put together below is the best way to do prioritization research. Instead, I’d like to give readers some place to start from more quickly by sharing the specific steps and templates I put together. I think that finding materials like these would have helped me to get going more quickly and improve my methods. I encourage others to iterate and adjust the approach for themselves, and to share in the comments how they would change it. I’ll also recommend some things you can change about the process below depending on your situation (e.g. how much time you have available).
As you proceed, your list of ideas becomes more concise, like a funnel. In the beginning, your approach should be broad, collecting a wide array of ideas. As you progress, the number of interventions narrows, and the depth of your research increases. This funnel process ensures that, by the end, you are left with a few good and extensively researched options.
Step 1: Consider and collect lots of ideas on a longlist
You can collect the ideas you think are somewhat promising on a longlist. The ideas you consider can come from various sources, including posts online, that ‘ideas’ note in your phone, and brainstorming sessions.
You can also try to source ideas from people who are experienced in your field(s) of choice and you trust. They might have a better sense of the important gaps in the field than someone from the outside. Referring to experts’ intuitions can be especially useful if there is little evidence to lean on or if you have little time to investigate the field yourself.
At this stage, some advice I have is: try to go for speed, not perfection. Your bar for including an idea shouldn’t be too high; separate the process of ideation and selection. You probably don’t have time to research the ideas before putting them on your longlist. You will do this later. Just go by your intuition: does this intervention sound like something I might want to do? And does this sound like it could have a lot of impact?
Personally, I aimed to gather around 100 ideas on my longlist and landed at around 85.
If your time or scope is limited, consider instead collecting 20, 40, or 60 ideas, just see what is realistic and useful. Then, instead of going from top 100 -> top 10 -> top 3, you can also go from top 20 -> top 6- > top 2 or any other variation you find sensible.
Resources:
Step 2: Reduce to a top 10
Now, you can transition to a more analytical phase using a Weighted Factor Model (WFM). In short, a WFM lets you rank a list of options by the set of criteria you determine (e.g. impact, motivation, personal fit). You give the criteria different ‘weights’ which depend on how important you think the criteria are.
In this stage it’s about finding a balance – if you have too many criteria, ranking items in your longlist will take a long time. So focus on what truly matters to you, and keep in mind that this is just initial filtering. You’ll consider more criteria once you have a shorter list of options.
I’d also encourage you to pre-determine how much time you’re willing to spend on each idea at this stage. I would roughly propose: take around 1-4 mins per idea if you have 80-100 interventions to go through. You could take a bit longer if you have less ideas to go through. You could then do some more quick research to take away major uncertainties.
To check my reasoning, I asked 4 people who know me (so they can judge my personal fit), who know the domains of the ideas, and who I think have good judgement to look at the top 30 in my longlist document. I asked them: 1) which ratings would you change, and 2) are there any interventions in the top 30 that you feel should be in the top 10?
Alternatively, you could also ask people to individually rate the ideas on different dimensions. This way you can make sure they don’t get influenced by your or others’ ratings.
If your interventions are within a specific domain, try asking experts in that domain to help you assess the ideas.
Resources:
Step 3: Reduce to a top 3
At this stage, I’d recommend investing more serious amounts of time into each of your top 10 ideas. Personally, I spent 4-6 hrs on researching each of the top 10 ideas. You can dive into each option with more in-depth analysis reports. The main things I tried to focus on in these reports is the evidence and quality of evidence for intervention x and the most likely reasons intervention x might not be feasible or impactful.
At this point, you could also create a more refined WFM which considers additional or more granular criteria. The reports you write can inform how you rate different facets of the intervention. For instance, as you learn more about the intervention you might learn that it seems much harder than you expected, and you might give it a low rating on your ‘feasibility’ criterion. Personally, I added some criteria into my WFM that I did not actually weigh because I was too uncertain about them. For example, I didn’t go into detail regarding impact in the WFM because I didn’t feel I could make a numerically accurate estimate. Instead, that was the focus of the reports I wrote on the interventions.
Something that was useful for me was spending 1-2 hrs discussing the options in your top 10 with a few experts in the field. They can give you feedback on your ideas and tell you why they think certain ones are more or less promising. You can also ask a few other people to attempt to independently fill out part of your WFM. You can then take their intuitions into account as you refine your own.
Taking into consideration the WFM and the reports, you can now pick which few ideas you think are most promising, which brings you down to your top 3.
Resources:
Step 4: Pick one
For this, you can conduct additional research into your top 3. I spent around 15-20 additional hrs on each, building on the initial report I had written for them. My focus here was on hashing out uncertainties. At this stage, talking to experts regarding those interventions is very useful.
After this, I picked the intervention I found most promising. This is only a starting point though, from here many more hours go into refining choices within that and iterating on your plans.
Some tips to keep in mind
The trade-off between depth and breadth
Be mindful of the trade-offs involved in your approach. Starting with a narrow focus, such as specifically prioritizing between anti-factory farming interventions, can yield more precise outcomes. However, this also means you might overlook other potentially impactful areas. Consider how you want to strike this balance between depth and breadth.
Decide what time commitment makes sense and hold on for dear life to time-capping
From my own experience I can tell you that it is easy to spend way too much time on this. You will discover a million uncertainties you need to look into to feel like you’re making an informed choice. This makes it easy to get stuck in ‘analysis paralysis’. Try to prevent this by pre-committing to how much time you want to invest into this research phase - and then sticking to it (a.k.a. time-capping).
You’ll have to trust in the 80/20 principle: 80% of results come from 20% of the efforts. This involves making choices even when you feel like you need more information to do so.
How much time you put into this should probably vary depending on how big the future commitment is - are you starting a new org or just a 3-month research project?
It can also vary depending on how important it is that you identify a very impactful next step. For instance, if you want to just learn a lot in your upcoming project, the impact of the project might matter less to you.
Seek feedback
Throughout every stage of this process, I encourage you to actively seek out feedback from trusted individuals. Their insights can help you identify blind spots in your reasoning and approach.
Find someone to do this with
I would recommend going through this process together with a small group of people. I didn’t but would if I were to do it again. This can help you stay on track, stick to the time-capping, and stay motivated. If you don’t know where to find your prioritization research partner, you can start by just posting a message about it in the EA Anywhere Slack channel.
Further resources
• Charity Entrepreneurship handbook - Chapter 19 (on how to decide on a charity idea).
• Workshop: How to prioritize | Renan Araujo, Marie Buhl, Jam Kraprayoon | EAG London 23.
Good luck!
Many thanks to Marie Buhl for the useful additions and input, and thank you to Gábor Szórád, Veerle Korsten and Bryce Robertson for providing feedback on drafts of this post.