James_Miller comments on Rationality Quotes March 2014 - Less Wrong

4 Post author: malcolmocean 01 March 2014 03:34PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (326)

You are viewing a single comment's thread.

Comment author: James_Miller 01 March 2014 05:27:27PM *  22 points [-]

[A]lmost no innovative programs work, in the sense of reliably demonstrating benefits in excess of costs in replicated RCTs [randomized controlled trials]. Only about 10 percent of new social programs in fields like education, criminology and social welfare demonstrate statistically significant benefits in RCTs. When presented with an intelligent-sounding program endorsed by experts in the topic, our rational Bayesian prior ought to be “It is very likely that this program would fail to demonstrate improvement versus current practice if I tested it.”

In other words, discovering program improvements that really work is extremely hard. We labor in the dark -- scratching and clawing for tiny scraps of causal insight.

Megan McArdle quoting or paraphrasing Jim Manzi.

[Edited in response to Kaj's comment.]

Comment author: CCC 03 March 2014 02:03:33PM 6 points [-]

Only about 10 percent of new social programs in fields like education, criminology and social welfare demonstrate statistically significant benefits in RCTs

This is a higher rate than I'd expected. It implies that current policies in these three fields are not really thoroughly thought out, or at least not to the extent that I had expected. It seems that there is substantial room for improvement.

I would have expected perhaps one or two percent.

Comment author: pgbh 06 March 2014 04:50:38AM 5 points [-]

Remember that programs will not even be tested unless there are good reasons to expect improvement over current protocol. Most programs that are explicitly considered are worse than those that are tested, and most possible programs are worse than those that are explicitly considered. Therefore we can expect that far, far fewer than ten percent of possible programs would yield significant improvements.

Comment author: CCC 07 March 2014 08:32:24AM 2 points [-]

That is true. However, there is a second filtering process, after filtering by experts; and that is what I will refer to as filtering by experiment (i.e. we'll try this, and if it works we keep doing it, and if it doesn't we don't). Evolution is basically a mix of random mutation and filtering by experiment, and it shows that, given enough time, such a filter can be astonishingly effective. (That time can be drastically reduced by adding another filter - such as filtering-by-experts - before the filtering-by-experiment step)

The one-to-two percent expectation that I had was a subconscious expectation of the comparison of the effectiveness of the filtering-by-experts in comparison to the filtering-by-experiment over time. Investigating my reasoning more thoroughly, I think that what I had failed to appreciate is probably that there really hasn't been enough time for filtering-by-experiment to have as drastic an effect as I'd assumed; societies change enough over time that what was a good idea a thousand years ago is probably not going to be a good idea now. (Added to this, it likely takes more than a month to see whether such a social program actually is effective or not; so there hasn't really been time for all that many consecutive experiments, and there hasn't really been a properly designed worldwide experimental test model, either).

Comment author: Lumifer 11 March 2014 05:22:06PM 3 points [-]

It implies that current policies in these three fields are not really thoroughly thought out, or at least not to the extent that I had expected.

That's one possible explanation.

Another possible explanation is that there is a variety of powerful stakeholders in these fields and the new social programs are actually designed to benefit them and not whoever the programs claim to help.

Comment author: maia 13 March 2014 02:26:35PM 2 points [-]

Remember, you expect 5% to give a statistically significant result just by chance...

Comment author: CCC 14 March 2014 07:49:04AM 2 points [-]

That's only true of the programs which can be expected to produce no detriments, surely?

Comment author: Will_Sawin 01 March 2014 06:57:46PM 6 points [-]

10% isn't that bad as long as you continue the programs that were found to succeed and stop the programs that were found to fail. Come up with 10 intelligent-sounding ideas, obtain expert endorsements, do 10 randomized controlled trials, get 1 significant improvement. Then repeat.

Comment author: ChristianKl 03 March 2014 11:53:40AM 8 points [-]

10% isn't that bad as long as you continue the programs that were found to succeed and stop the programs that were found to fail.

Unfortunately we don't really have the political system to do this.

Comment author: [deleted] 11 March 2014 05:00:26PM 4 points [-]

But I have this great idea that will change that!

...Oh.

Comment author: Eugine_Nier 01 March 2014 07:20:31PM 4 points [-]

Unfortunately, governments are really bad at doing this.

Comment author: RolfAndreassen 02 March 2014 12:21:46AM 28 points [-]

Humans in general are very bad at this. The only reason capitalism works is that the losing experiments run out of money.

Comment author: RichardKennaway 03 March 2014 07:08:55AM 9 points [-]

The only reason capitalism works is that the losing experiments run out of money.

That's a very powerful reason.

Comment author: Will_Sawin 03 March 2014 05:31:05PM 1 point [-]

True, but that doesn't mean we're laboring in the dark. It just means we've got our eyes closed.

Comment author: Eugine_Nier 04 March 2014 01:48:54AM -1 points [-]

Unfortunately, the people involved have an incentive to keep them closed.

Comment author: Will_Sawin 04 March 2014 02:42:39AM 1 point [-]

I don't think that's really relevant to the original quote.

Comment author: ThrustVectoring 02 March 2014 10:46:31AM 0 points [-]

It depends on how many completely ineffectual programs would demonstrate improvement versus current practices.

Comment author: gwern 01 April 2014 10:05:30PM *  4 points [-]
Comment author: Kaj_Sotala 01 March 2014 06:00:17PM 4 points [-]

I think the quote is from Jim Manzi rather than Megan McArdle, given that McArdle starts the article with

I asked Jim Manzi, who has literally written the book on randomized controlled trials, to share his thoughts. Below is what he said:

and later on in the article it says

I agree with the weight and seriousness of each of these objections. My agreement is not ad hoc; I wrote a book that tried to describe how businesses have implemented experimental processes that operate in the face of all of these issues.

suggesting that the whole article after the first paragraph is a quote (or possibly paraphrase).