Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Comment author: Benquo 29 November 2016 08:42:14PM *  3 points [-]

Can you say more about the perceived smugness? It seems to me like a straightforward account of the obvious limitation to GiveWell's scope. I only didn't upvote because it seemed too obvious.

Comment author: peter_hurford 30 November 2016 06:55:10PM 6 points [-]

To me, the tone came across as "Ho ho ho, look at those stupid GiveWell people who have never heard of the streetlight effect! They're blinded by their own metrics and can't even see how awesome MIRI is!" when there's no interaction or acknowledgement with (a) materials from GiveWell that address the streetlight effect argument, (b) OpenPhil, or (c) how to actually start to resolve the problem (or even that the problem is particularly hard).

I don't want to have a high demand for rigor, especially for Discussion-type posts -- for me, it's more about the lack of humility.

Comment author: peter_hurford 29 November 2016 08:02:42PM 6 points [-]

I downvoted because this feels overly smug to me. I think it's a legitimate issue, but GiveWell has made many arguments for why they do what they do, and OpenPhil has made some progress on figuring out how to evaluate AI organizations. Sure, many fields might very well be vastly more fruitful, but they also might not. How do we know which ones?

Using a Spreadsheet to Make Good Decisions: Five Examples

11 peter_hurford 28 November 2016 05:10PM

I've been told that LessWrong is coming back now, so I'm cross-posting this rationality post of interest from the Effective Altruism forum.

-

We all make decisions every day. Some of these decisions are pretty inconsequential, such as what to have for an afternoon snack. Some of these decisions are quite consequential, such as where to live or what to dedicate the next year of your life to. Finding a way to make these decisions better is important.

The folks at Charity Science Health and I have been using the same method to make many of our major decisions for the past for years -- everything from where to live to even deciding to create Charity Science Health. The method isn’t particularly novel, but we definitely think the method is quite underused.

Here it is, as a ten step process:

  1. Come up with a well-defined goal.

  2. Brainstorm many plausible solutions to achieve that goal.

  3. Create criteria through which you will evaluate those solutions.

  4. Create custom weights for the criteria.

  5. Quickly use intuition to prioritize the solutions on the criteria so far (e.g., high, medium, and low)

  6. Come up with research questions that would help you determine how well each solution fits the criteria

  7. Use the research questions to do shallow research into the top ideas (you can review more ideas depending on how long the research takes per idea, how important the decision is, and/or how confident you are in your intuitions)

  8. Use research to rerate and rerank the solutions

  9. Pick the top ideas worth testing and do deeper research or MVP testing, as is applicable

  10. Repeat steps 8 and 9 until sufficiently confident in a decision.

 

Which charity should I start?

The definitive example for this process was the Charity Entrepreneurship project, where our team decided which charity would be the best possible charity to create.

Come up with a well-defined goal: I want to start an effective global poverty charity, where effective is taken to mean a low cost per life saved comparable to current GiveWell top charities.

Brainstorm many plausible solutions to achieve that goal: For this, we decided to start by looking at the intervention level. Since there are thousands of potential interventions, we placed a lot of emphasis on plausibly highly effectve, and chose to look at GiveWell’s priority programs plus a few that we thought were worthy additions.

Create criteria through which you will evaluate those solutions / create custom weights for the criteria: For this decision, we spent a full month of our six month project thinking through the criteria. We weighted criteria based on both importance and the expected varaince that would occur between our options. We decided to strongly value cost-effectiveness, flexibility , and scalability. We moderately valued strength of evidence, metric focus, and indirect effects. We weakly valued logistical possibility and other factors.
 

Come up with research questions that would help you determine how well each solution fits the criteria: We came up with the following list of questions and research process.

Use the research questions to do shallow research into the top ideas, use research to rerate and rerank the solutions: Since this choice was important and we were pretty uninformed about the different interventions, we did shallow research into all of the choices. We then produced the following spreadsheet:

Afterwards, it was pretty easy to drop 22 out of the 30 possible choices and go with a top eight (the eight that ranked 7 or higher on our scale).

 

Pick the top ideas worth testing and do deeper research or MVP testing, as is applicable / Repeat steps 8 and 9 until sufficiently confident in a decision: We then researched the top eight more deeply, with a keen idea to turn them into concrete charity ideas rather than amorphous interventions. When re-ranking, we came up with a top five, and wrote up more detailed reports --SMS immunization reminders,tobacco taxation,iron and folic acid fortification,conditional cash transfers, and a poverty research organization. A key aspect to this narrowing was also talking to relevant experts, which we wish we did earlier on in the process as it could quickly eliminate unpromising options.

Pick the top ideas worth testing and do deeper research or MVP testing, as is applicable: As we researched further, it became more clear that SMS immunization reminders performed best on the criteria being highly cost-effective, with a high strength of evidence and easy testability. However, the other four finalists are also excellent opportunities and we strongly invite other teams to invest in creating charities in those four areas.

 

Which condo should I buy?

Come up with a well-defined goal: I want to buy a condo that is (a) a good place to live and (b) a reasonable investment.
 

Brainstorm many plausible solutions to achieve that goal: For this, I searched around on Zillow and found several candidate properties.

Create criteria through which you will evaluate those solutions: For this decision, I looked at the purchasing cost of the condo, the HOA fee, whether or not the condo had parking, the property tax, how much I could expect to rent the condo out, whether or not the condo had a balcony, whether or not the condo had a dishwasher, how bright the space was, how open the space was, how large the kitchen was, and Zillow’s projection of future home value.
 

Create custom weights for the criteria: For this decision, I wanted to turn things roughly into a personal dollar value, where I could calculate the benefits minus the costs. The costs were the purchasing cost of the condo turned into a monthly mortgage payment, plus the annual HOA fee, plus the property tax. The benefits were the expected annual rent plus half of Zillow’s expectation for how much the property would increase in value over the next year, to be a touch conservative. I also added some more arbitrary bonuses: +$500 bonus if there was a dishwasher, a +$500 bonus if there was a balcony, and up to +$1000 depending on how much I liked the size of the kitchen. I also added +$3600 if there was a parking space, since the space could be rented out to others as I did not have a car. Solutions would be graded on benefits minus costs model.

Quickly use intuition to prioritize the solutions on the criteria so far: Ranking the properties was pretty easy since it was very straightforward, I could skip to plugging in numbers directly from the property data and the photos.

 

Property

Mortgage

Annual fees

Annual increase

Annual rent

Bonuses

Total

A

$7452

$5244

$2864

$17400

+$2000

+$9568

B

$8760

$4680

$1216

$19200

+$1000

+$7976

C

$9420

$4488

$1981

$19200

+$1200

+$8473

D

$8100

$8400

$2500

$19200

+$4100

+$9300

E

$6900

$4600

$1510

$15000

+$3600

+$8610

  

Come up with research questions that would help you determine how well each solution fits the criteria: For this, the research was just to go visit the property and confirm the assessments.

Use the research questions to do shallow research into the top ideas, use research to rerate and rerank the solutions: Pretty easy, not much changed as I went to actually investigate.

Pick the top ideas worth testing and do deeper research or MVP testing, as is applicable: For this, I just ended up purchasing the highest ranking condo, which was a mostly straightforward process. Property A wins! 
 
This is a good example of how easy it is to re-adapt the process and how you can weight criteria in nonlinear ways.
 

How should we fundraise? 

Come up with a well-defined goal: I want to find the fundraising method with the best return on investment. 

Brainstorm many plausible solutions to achieve that goal: For this, our Charity Science Outreach team conducted a literature review of fundraising methods and asked experts, creating a list of the 25 different fundraising ideas. 

Create criteria through which you will evaluate those solutions / Create custom weights for the criteria: The criteria we used here was pretty similar to the criteria we later used for picking a charity -- we valued ease of testing, the estimated return on investment, the strength of the evidence, and the scalability potential roughly equally. 

Come up with research questions that would help you determine how well each solution fits the criteria: We created this rubric with questions

  • What research says on it (e.g. expected fundraising ratios, success rates, necessary pre-requisites)

  • What are some relevant comparisons to similar fundraising approaches? How well do they work?

  • What types/sizes of organizations is this type of fundraising best for?

  • How common is this type of fundraising, in nonprofits generally and in similar nonprofits (global health)?

  • How one would run a minimum cost experiment in this area?

  • What is the expected time, cost, and outcome for the experiment?

  • What is the expected value?

  • What is the expected time cost to get best time per $ ratio (e.g., would we have to have 100 staff or huge budget to make this effective)?

  • What further research should be done if we were going to run this approach?

Use the research questions to do shallow research into the top ideas, use research to rerate and rerank the solutions: After reviewing, we were able to narrow the 25 down to eight finalists: legacy fundraising, online ads, door-to-door, niche marketing, events, networking, peer-to-peer fundraising, and grant writing.
 
Pick the top ideas worth testing and do deeper research or MVP testing, as is applicable: We did MVPs of all eight of the top ideas and eventually decided that three of the ideas were worth pursuing full-time: online ads, peer-to-peer fundraising, and legacy fundraising.
 
 

Who should we hire? 

Come up with a well-defined goal: I want to hire the employee who will contribute the most to our organization. 

Brainstorm many plausible solutions to achieve that goal: For this, we had the applicants who applied to our job ad.

Create criteria through which you will evaluate those solutions / Create custom weights for the criteria: We thought broadly about what good qualities a hire would have, and decided to heavily weight values fit and prior experience with the job, and then roughly equally value autonomy, communication skills, creative problem solving, the ability to break down tasks, and the ability to learn new skills.
 
Quickly use intuition to prioritize the solutions on the criteria so far: We started by ranking hires based on their resumes and written applications. (Note that to protect the anonymity of our applicants, the following information is fictional.)
 

Person

Autonomy

Communication

Creativity

Break down

Learn new skills

Values fit

Prior experience

A

High

Medium

Low

Low

High

Medium

Low

B

Medium

Medium

Medium

Medium

Medium

Medium

Low

C

High

Medium

Medium

Low

High

Low

Medium

D

Medium

Medium

Medium

High

Medium

Low

High

E

Low

Medium

High

Medium

Medium

Low

Medium

 

Come up with research questions that would help you determine how well each solution fits the criteria: The initial written application was already tailored toward this, but we designed a Skype interview to further rank our applicants. 

Use the research questions to do shallow research into the top ideas, use research to rerate and rerank the solutions: After our Skype interviews, we re-ranked all the applicants. 

 

Person

Autonomy

Communication

Creativity

Break down

Learn new skills

Values fit

Prior experience

A

High

High

Low

Low

High

High

Low

B

Medium

Medium

Medium

Medium

Low

Low

Low

C

High

Medium

Low

High

High

Medium

Medium

D

Medium

Low

Medium

High

Medium

Low

High

E

Low

Medium

High

Medium

Medium

Low

Medium

  

Pick the top ideas worth testing and do deeper research or MVP testing, as is applicable: While “MVP testing” may not be polite to extend to people, we do a form of MVP testing by only offering our applicants one month trials before converting to a permanent hire.

 

Which television show should we watch? 

Come up with a well-defined goal: Our friend group wants to watch a new TV show together that we’d enjoy the most. 

Brainstorm many plausible solutions to achieve that goal: We all each submitted one TV show, which created our solution pool. 

Create criteria through which you will evaluate those solutions / Create custom weights for the criteria: For this decision, the criteria was the enjoyment value of each participant, weighted equally. 

Come up with research questions that would help you determine how well each solution fits the criteria: For this, we watched the first episode of each television show and then all ranked each one. 

Pick the top ideas worth testing and do deeper research or MVP testing, as is applicable: We then watched the winning television show, which was Black Mirror. Fun! 

 

Which statistics course should I take? 

Come up with a well-defined goal: I want to learn as much statistics as fast as possible, without having the time to invest in taking every course. 

Brainstorm many plausible solutions to achieve that goal: For this, we searched around on the internet and found ten online classes and three books.

Create criteria through which you will evaluate those solutions / Create custom weights for the criteria: For this decision, we heavily weighted breadth and time cost, weighted depth and monetary cost, and weakly weighted how interesting the course was and whether the course provided a tangible credential that could go on a resume.
 
Quickly use intuition to prioritize the solutions on the criteria so far: By looking at the syllabi, table of contents, and reading around online, we came up with some initial rankings:
 
 

Name

Cost

Estimated hours

Depth score

Breadth score

How interesting

Credential level

Master Statistics with R

$465

150

10

9

3

5

Probability and Statistics, Statistical Learning, Statistical Reasoning

$0

150

8

10

4

2

Critically Evaluate Social Science Research and Analyze Results Using R

$320

144

6

6

5

4

http://online.stanford.edu/Statistics_Medicine_CME_Summer_15

$0

90

5

2

7

0

Berkley stats 20 and 21

$0

60

6

5

6

0

Statistical Reasoning for Public Health

$0

40

5

2

4

2

Khan stats

$0

20

1

4

6

0

Introduction to R for Data Science

$0

8

3

1

5

1

Against All Odds

$0

5

1

2

10

0

Hans Rosling doc on stats

$0

1

1

1

11

0

Berkeley Math

$0

60

6

5

6

0

OpenIntro Statistics

$0

25

5

5

2

0

Discovering Statistics Using R by Andy Field

$25

50

7

3

3

0

Naked-Statistics by Charles Wheelan

$17

20

2

4

8

0

 

Come up with research questions that would help you determine how well each solution fits the criteria: For this, the best we could do would be to do a little bit from each of our top class choices, while avoiding purchasing the expensive ones unless free ones did not meet our criteria. 

Pick the top ideas worth testing and do deeper research or MVP testing, as is applicable: Only the first three felt deep enough. Only one of them was free, but we were luckily able to find a way to audit the two expensive classes. After a review of all three, we ended up going with “Master Statistics with R”.

Comment author: Soothsilver 26 November 2015 06:42:11PM 1 point [-]

Maybe this should be posted in Discussion? That way more people will see it.

Comment author: peter_hurford 01 December 2015 05:08:35PM 1 point [-]

Take the EA survey, help the EA movement grow and potentially win $250 to your favorite charity

5 peter_hurford 01 December 2015 05:07PM

This year's EA Survey is now ready to be shared! This is a survey of all EAs to learn about the movement and how it can improve. The data collected in the survey is used to help EA groups improve and grow EA. Data is also used to populate the map of EAs, create new EA meetup groups, and create EA Profiles and the EA Donation Registry.

If you are an EA or otherwise familiar with the community, we hope you will take it using this link. All results will be anonymised and made publicly available to members of the EA community. As an added bonus, one random survey taker will be selected to win a $250 donation to their favorite charity.

Take the EA Survey

Please share the survey with others who might be interested using this link rather than the one above: http://bit.ly/1OqsVWo

-

(Cross-posted on Discussion from Main by popular request.)

Take the EA survey, help the EA movement grow and potentially win $250 to your favorite charity

18 peter_hurford 01 December 2015 01:56AM

This year's EA Survey is now ready to be shared! This is a survey of all EAs to learn about the movement and how it can improve. The data collected in the survey is used to help EA groups improve and grow EA. Data is also used to populate the map of EAs, create new EA meetup groups, and create EA Profiles and the EA Donation Registry.

If you are an EA or otherwise familiar with the community, we hope you will take it using this link. All results will be anonymised and made publicly available to members of the EA community. As an added bonus, one random survey taker will be selected to win a $250 donation to their favorite charity.

Take the EA Survey

Please share the survey with others who might be interested using this link rather than the one above: http://bit.ly/1OqsVWo

Comment author: [deleted] 17 October 2015 10:52:51AM *  1 point [-]

Great offer.

Would you consider extending the offer to MIRI, Living Goods or Development Media International (for those unfamiliar: the first being the AI organisation that sponsors this website, the last two being 2 of GiveWell's standout charities?

Comprehensive, evidence-based toolkits for policy makers and planners on Improving Educational Quality through Interactive Radio Instruction exists. Development Media International is one of GiveWell's standout organisations and implements radio-based behaviour change health education to "save the greatest number of lives in the most cost effective way.

GiveWell's most compelling reason for not including Living Good's as one of their 4 recommended charities is room for more funding. However, the premise for the lack of room for more funding rested on the assumption that it would be funded by a big donor soon. There is no evidence to suggest this has taken place since GiveWell's analysis.

The size of Living Goods' funding gap for the next year is highly uncertain because major funders are considering supporting the program. If Living Goods raises enough funds to scale up the program studied with the RCT, it may allocate additional funds to programs with less of a track record.

I favour living goods since their approach doesn't disincentivise rational market behaviour and there are methodological issues re: the effectiveness of the 4 recommended GiveWell charities that are described elsewhere.

I favour DMI for complex reasons including the integrity of their frontier scientific methodology which I may elaborate upon at a later stage. They do their work very well but I still have concerns that it may be increasing the survivability of those who neglect somewhat easily researched solutions to everyday problems and therefore sustain unwellbeing in the long term.

Update: I have been doing some thinking about making a bequest, prompted by this offer. So, suddenly the urgency of donating feels less great to me, and dying with savings is suddenly valuable. When I think about how to maximise my savings, one thing that comes to mind is maximising my inheritence. My sister rents a house from my parents below market price. I feel like this is unjust, since it will take away from the some of my inheritence. She is a wealthy quant and just 'doesn't have the time or need' to change arrangements. I feel bad about this, like I'm being robbed or they are favouring her. At the same time, I'm not entitled to their money. This is stressful!

Comment author: peter_hurford 18 October 2015 07:04:07PM 0 points [-]

Would you consider extending the offer to MIRI, Living Goods or Development Media International (for those unfamiliar: the first being the AI organisation that sponsors this website, the last two being 2 of GiveWell's standout charities?

At the moment we would prefer to not extend the offer to MIRI. This is because we think it’s valuable to keep an organization fairly focused on doing a few things well and fundraising for MIRI currently falls out of Charity Science’s scope. It’s also legally dubious whether or not Charity Science can use its resources to influence money to groups not involved in alleviating poverty.

We are more than happy to extend the offer to Living Goods or Development Media International. We do recommend bequests to GiveWell’s top charities though because GiveWell’s fluidity and flexibility will allow their recommendations to change over time, which means that they’re an excellent choice to leave a bequest to. This is in contrast to another charity that may be effective now but may not be 30 years from now.

We’ll write you a will for free if you leave a gift to GiveWell’s top charities

4 peter_hurford 16 October 2015 03:33AM

Would you like to leave money in your will to GiveWell’s top rated charities at the time of your passing? If so, Charity Science will you help you write it for free.

To make it as easy as possible for you, we at Charity Science have made a simple form that takes as little as 5 minutes to complete. After that you come out with a ready made will. And don’t worry if you’re not sure what to put in it; it’s easy to change and you can always come back to it later. So give it a shot here. The default option should be to set it up just in case something terrible does happen, that way you always have something ready.

A few more reasons to take the time to write a will include:

  • Reducing the inheritance tax incurred - leaving money to charity being an excellent way to do so.

  • Making provisions for your children if you have any, for example by choosing who will take care of them and setting aside funds for this.

  • Making any other necessary provisions, such as for your pets, or your business, or other responsibilities that you have.

  • Specifying what sort of funeral you would like, which will spare your family from having to make the decision.

  • Naming your executors for your will (family members are a standard choice).


But most of all it’s because you have the incredible opportunity to do an epic amount of good.


You can set it up here. After that consider talking to your friends, parents and grandparents to see if they would be interested in doing the same. It’s really important you mention it because the average amount left to charities in a will is in the thousands of dollars so a few words may go a very long way.


If this doesn’t appeal to you then there are other things that you could do. You can always run a fundraiser for Christmas, your Birthday or any event you like.

Comment author: [deleted] 13 July 2015 06:42:53AM 1 point [-]

Bill Gates said something along the lines learn computer science by studying great code, not studying computer science formally. I'm trying to take that advice, starting with making things to improve my productivity (stuff I'll actually use). I might start with this Twilio API tutorial. Can you forsee any problems with this approach?

Comment author: peter_hurford 16 July 2015 04:17:40PM 1 point [-]

I think it's good advice generally speaking, but it won't work for beginners. Studying great code is a great way to go from intermediate to expert, but if you haven't already gone from beginner to intermediate than you probably won't be able to recognize what is good code or understand why it is good.

Comment author: peter_hurford 14 May 2015 02:09:38PM 7 points [-]

When I was 16, I spent a lot of time "analyzing signals", and it never went well.

View more: Next