Thanks for the feedback.

I think there are some important qualifications to make about this post, as others have noted.

My hunch is that most significant problem with the MWA approach is the assumption of (weak) independence, in the sense that in practice, when sophisticated use of MWA fails, it's usually because the weak lines of evidence are all being driven by the same selection effect. A hypothetical example that jumps to mind is:

A VC is evaluating a startup. He or she reasons

1. The sector is growing
2. My colleagues think that the sector is good to invest in
3. On an object level, their plan looks good
4. The people are impressive

and the situation is

Re: #1 — The reason that the sector is growing is because there's a bubble

Re: #2 — The reason that the VC's colleagues think that the sector is good to invest in is because, like the VC, they don't recognize that there's a bubble.

Re: #3 — The VC's views on the object level merit of the project are colored by the memes that have been spreading around that are causing the bubble

Re: #4 — The reason that impressive people are going into the sector is because there's a bubble, so everyone's going into the sector – the people's impressiveness isn't manifesting itself in their choosing a good focus.

I don't know whether this situation occurs in practice, but it seems very possible.

Givewell tends to emphasize the MWA approach, and has been remarkably successful at figuring out the parts of the world they're trying to understand.

GiveWell is an interesting case, insofar as it's done more ORSA work than I've seen in most contexts. The page on long lasting insecticide treated nets provides examples. Part of why I'm favoring MWA is because GiveWell has done both and of the two, leans toward MWA.