CCC comments on Rationality Quotes March 2014 - Less Wrong

4 Post author: malcolmocean 01 March 2014 03:34PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (326)

You are viewing a single comment's thread. Show more comments above.

Comment author: James_Miller 01 March 2014 05:27:27PM *  22 points [-]

[A]lmost no innovative programs work, in the sense of reliably demonstrating benefits in excess of costs in replicated RCTs [randomized controlled trials]. Only about 10 percent of new social programs in fields like education, criminology and social welfare demonstrate statistically significant benefits in RCTs. When presented with an intelligent-sounding program endorsed by experts in the topic, our rational Bayesian prior ought to be “It is very likely that this program would fail to demonstrate improvement versus current practice if I tested it.”

In other words, discovering program improvements that really work is extremely hard. We labor in the dark -- scratching and clawing for tiny scraps of causal insight.

Megan McArdle quoting or paraphrasing Jim Manzi.

[Edited in response to Kaj's comment.]

Comment author: CCC 03 March 2014 02:03:33PM 6 points [-]

Only about 10 percent of new social programs in fields like education, criminology and social welfare demonstrate statistically significant benefits in RCTs

This is a higher rate than I'd expected. It implies that current policies in these three fields are not really thoroughly thought out, or at least not to the extent that I had expected. It seems that there is substantial room for improvement.

I would have expected perhaps one or two percent.

Comment author: pgbh 06 March 2014 04:50:38AM 5 points [-]

Remember that programs will not even be tested unless there are good reasons to expect improvement over current protocol. Most programs that are explicitly considered are worse than those that are tested, and most possible programs are worse than those that are explicitly considered. Therefore we can expect that far, far fewer than ten percent of possible programs would yield significant improvements.

Comment author: CCC 07 March 2014 08:32:24AM 2 points [-]

That is true. However, there is a second filtering process, after filtering by experts; and that is what I will refer to as filtering by experiment (i.e. we'll try this, and if it works we keep doing it, and if it doesn't we don't). Evolution is basically a mix of random mutation and filtering by experiment, and it shows that, given enough time, such a filter can be astonishingly effective. (That time can be drastically reduced by adding another filter - such as filtering-by-experts - before the filtering-by-experiment step)

The one-to-two percent expectation that I had was a subconscious expectation of the comparison of the effectiveness of the filtering-by-experts in comparison to the filtering-by-experiment over time. Investigating my reasoning more thoroughly, I think that what I had failed to appreciate is probably that there really hasn't been enough time for filtering-by-experiment to have as drastic an effect as I'd assumed; societies change enough over time that what was a good idea a thousand years ago is probably not going to be a good idea now. (Added to this, it likely takes more than a month to see whether such a social program actually is effective or not; so there hasn't really been time for all that many consecutive experiments, and there hasn't really been a properly designed worldwide experimental test model, either).

Comment author: Lumifer 11 March 2014 05:22:06PM 3 points [-]

It implies that current policies in these three fields are not really thoroughly thought out, or at least not to the extent that I had expected.

That's one possible explanation.

Another possible explanation is that there is a variety of powerful stakeholders in these fields and the new social programs are actually designed to benefit them and not whoever the programs claim to help.

Comment author: maia 13 March 2014 02:26:35PM 2 points [-]

Remember, you expect 5% to give a statistically significant result just by chance...

Comment author: CCC 14 March 2014 07:49:04AM 2 points [-]

That's only true of the programs which can be expected to produce no detriments, surely?