Trial and error in policy making

7 DavidAgain 15 June 2011 10:39AM

I’m working in government in the UK, and interested in rationality in policy making: at the individual level, but also how we can build systems (in terms of requirements during policy creation, creation of independent bodies, incentive structures for officials/ministers…) that encourage more rational policies. This is the context for the stuff below.

 

I recently went to a talk on the use of trial and error in policy making. It was hosted by the Institute for Government (a non-political think tank advising ministers and civil servants on good government), and included the Undercover Economist, Tim Harford.  Details including a recording can be found here: http://www.instituteforgovernment.org.uk/our-events/105/government-by-trial-and-error-a-discussion-with-tim-harford

Ben Goldacre was also attending in the audience, and has written about this: http://www.badscience.net/2011/05/we-should-so-blatantly-do-more-randomised-trials-on-policy/

The basic message of the talk was that much of policy could benefit from being trialled more effectively, whether in the form of formalised, controlled trials: following the medical example inasmuch as that is practically possible.  This approach seems to be being seriously considered by this government in several areas: one example of how it might work in theory can be found here: http://www.straightstatistics.org/article/what-works-criminal-justice-time-find-out

Of course, the reality is often messier, and ministers are looking for both swifter results and more security. Other forms of ‘experimentation’ involve devolving power to local councils (Localism Bill) or schools (Free Schools) etc. In these cases there aren’t the same formal comparison methods but you at least get lots of ideas and some ability to judge which ones have gone well. An underlying issue is the quality of assessments: a policy that has clear criteria for appraisal built into its design will be much easier to judge, whereas one that is simply acted upon and studied later might suffer from poor evidence and be more open to being interpreted in line with preconcieved or desired outcomes.

A final approach is ‘payment by results’, where tasks such as preventing reoffending or getting people off drugs is paid for on the basis of successes achieved rather than services delivered, the idea being that this provides a fiscal incentive to get it right. A critical view of one such scheme is here

http://www.straightstatistics.org/article/flaws-mojs-reoffending-study

 

I was wondering what people here felt about the idea of trial-and-error in policy. I think it seems like a good idea, with the following limitations

-          a) some policies probably can’t be trialled on a national scale (can you have half the country making cuts while the other half keeps spending?)

-          b) some cases will have ethical issues as with medicine

-          c) some cases might not allow clear-cut results, eliminating the ability to learn and simply creating a chaotic mess of different approaches

-          d) the biggy: I don’t know if the political culture is open to it.

Any thoughts?

[prize] new contest for Spaced Repetition literature review ($365+)

15 jsalvatier 18 June 2011 06:31PM

Update: the prize is now finished!

The previous contest was poorly formatted for eliciting the most useful reviews of the spaced repetition literature so I've created a new slightly different contest. 

I'm interested in making projects happen on Less Wrong. In order to find out what works and to inspire others to try things too, I'm sponsoring the following small project:

Spaced Repetition is often mentioned on Less Wrong as a technique for adding facts to memory. I've started using Anki and it certainly seems to be useful. However, I haven't seen a good summary of evidence on Spaced Repetition and I would like to change that.

I hereby offer a prize, currently $385, to the best literature review submitted by August 1st. 'Best' will be judged by voting with discussion beforehand by the Seattle LW meetup group. People are not allowed to vote for their own submissions.

The summary should address questions such as:

  • What spacing is best?
  • How much does spaced repetition actually help memory?
  • Does spaced repetition have hidden benefits or costs?
  • Does the effectiveness vary across domains? How much? 
  • Is there research on the kinds of questions that work best? Especially for avoiding 'guessing the password' and memorizing the card per se rather than the material.
  • What questions do researchers think are most important?
  • Is there any interesting ongoing research? If so, what is it on?
  • What, if any, questions do researchers think it is important to answer? Are there other unanswered questions that would jump out at a smart person?
  • What does spaced repetition not do that people might expect it to?

The post should summarize the state of current evidence and provide citations to back up the claims in the article. Referencing both academic and non-academic research is encouraged. Lukeprog's The Science of Winning At Life sequence contains several examples of good literature review posts.

If you think you would benefit from the result of this project, please add to the prize! You can contribute to the prize on the ChipIn page.

If you have suggestions, questions or comments, please leave them in the comments. Prizes demotivating? Due date too soon/far? Specification too vague? Judgement procedure not credible enough?

This project is tagged with the 'project' tag and listed on the Projects wiki page.

A Master-Slave Model of Human Preferences

58 Wei_Dai 29 December 2009 01:02AM

[This post is an expansion of my previous open thread comment, and largely inspired by Robin Hanson's writings.]

In this post, I'll describe a simple agent, a toy model, whose preferences have some human-like features, as a test for those who propose to "extract" or "extrapolate" our preferences into a well-defined and rational form. What would the output of their extraction/extrapolation algorithms look like, after running on this toy model? Do the results agree with our intuitions about how this agent's preferences should be formalized? Or alternatively, since we haven't gotten that far along yet, we can use the model as one basis for a discussion about how we want to design those algorithms, or how we might want to make our own preferences more rational. This model is also intended to offer some insights into certain features of human preference, even though it doesn't capture all of them (it completely ignores akrasia for example).

continue reading »

Seeing Red: Dissolving Mary's Room and Qualia

38 orthonormal 26 May 2011 05:47PM

Essential Background: Dissolving the Question

How could we fully explain the difference between red and green to a colorblind person?

Well, we could of course draw the analogy between colors of the spectrum and tones of sound; have them learn which objects are typically green and which are typically red (or better yet, give them a video camera with a red filter to look through); explain many of the political, cultural and emotional associations of red and green, and so forth... but it seems that the actual difference between our experience of redness and our experience of greenness is something much harder to convey. If we focus in on that aspect of experience, we end up with the classic philosophical concept of qualia, and the famous thought experiment known as Mary’s Room1.

Mary is a brilliant neuroscientist who has been colorblind from birth (due to a retina problem; her visual cortex would work normally if it were given the color input). She’s an expert on the electromagnetic spectrum, optics, and the science of color vision. We can postulate, since this is a thought experiment, that she knows and fully understands every physical fact involved in color vision; she knows precisely what happens, on various levels, when the human eye sees red (and the optic nerve transmits particular types of signals, and the visual cortex processes these signals, etc).

One day, Mary gets an operation that fixes her retinas, so that she finally sees in color for the first time. And when she wakes up, she looks at an apple and exclaims, "Oh! So that's what red actually looks like."2

Now, this exclamation poses a challenge to any physical reductionist account of subjective experience. For if the qualia of seeing red could be reduced to a collection of basic facts about the physical world, then Mary would have learned those facts earlier and wouldn't learn anything extra now– but of course it seems that she really does learn something when she sees red for the first time. This is not merely the god-of-the-gaps argument that we haven't yet found a full reductionist explanation of subjective experience, but an intuitive proof that no such explanation would be complete.

The argument in academic philosophy over Mary's Room remains unsettled to this day (though it has an interesting history, including a change of mind on the part of its originator). If we ignore the topic of subjective experience, the arguments for reductionism appear to be quite overwhelming; so why does this objection, in a domain in which our ignorance is so vast3, seem so difficult for reductionists to convincingly reject?

Veterans of this blog will know where I'm going: a question like this needs to be dissolved, not merely answered.

continue reading »

Edinburgh LW meetup Saturday 4th of June, 2pm

1 Elias_Kunnas 02 June 2011 05:03PM

Another meetup at the Delhi cafe on Saturday at 2pm (67 Nicolson Street - see http://goo.gl/Wwws0).

No set topic really this time, though suggestions are welcome (for now and for the long-term). Not sure what kind of things would work best in maintaining interest in these...
I would personally be somewhat interest in the mindfulness practice as well discussing the somewhat associated recent post by Kaj Sotala.

I will have the book on the Flow with me:

Flow: The Psychology of Happiness

Suffering as attention-allocational conflict

49 Kaj_Sotala 18 May 2011 03:12PM

I previously characterized Michael Vassar's theory on suffering as follows: "Pain is not suffering. Pain is just an attention signal. Suffering is when one neural system tells you to pay attention, and another says it doesn't want the state of the world to be like this." While not too far off the mark, it turns out this wasn't what he actually said. Instead, he said that suffering is a conflict between two (or more) attention-allocation mechanisms in the brain.

I have been successful at using this different framing to reduce the amount of suffering I feel. The method goes like this. First, I notice that I'm experiencing something that could be called suffering. Next, I ask, what kind of an attention-allocational conflict is going on? I consider the answer, attend to the conflict, resolve it, and then I no longer suffer.

An example is probably in order, so here goes. Last Friday, there was a Helsinki meetup with Patri Friedman present. I had organized the meetup, and wanted to go. Unfortunately, I already had other obligations for that day, ones I couldn't back out from. One evening, I felt considerable frustration over this.

Noticing my frustration, I asked: what attention-allocational conflict is this? It quickly become obvious that two systems were fighting it out:

* The Meet-Up System was trying to convey the message: ”Hey, this is a rare opportunity to network with a smart, high-status individual and discuss his ideas with other smart people. You really should attend.”
* The Prior Obligation System responded with the message: ”You've already previously agreed to go somewhere else. You know it'll be fun, and besides, several people are expecting you to go. Not going bears an unacceptable social cost, not to mention screwing over the other people's plans.”

Now, I wouldn't have needed to consciously reflect on the messages to be aware of them. It was hard to not be aware of them: it felt like my consciousness was in a constant crossfire, with both systems bombarding it with their respective messages.

But there's an important insight here, one which I originally picked up from PJ Eby. If a mental subsystem is trying to tell you something important, then it will persist in doing so until it's properly acknowledged. Trying to push away the message means it has not been properly addressed and acknowledged, meaning the subsystem has to continue repeating it.

continue reading »

Overcoming suffering: Emotional acceptance

38 Kaj_Sotala 29 May 2011 10:57AM

Follow-up to: Suffering as attention-allocational conflict.

In many cases, it may be possible to end an attention-allocational conflict by looking at the content of the conflict and resolving it. However, there are also many cases where this simply won't work. If you're afraid of public speaking, say, the "I don't want to do this" signal is going to keep repeating itself regardless of how you try to resolve the conflict. Instead, you have to treat the conflict in a non-content-focused way.

In a nutshell, this is just the map-territory distinction as applied to emotions. Your emotions have evolved as a feedback and attention control mechanism: their purpose is to modify your behavior. If you're afraid of a dog, this is a fact about you, not about the dog. Nothing in the world is inherently scary, bad or good. Furthermore, emotions aren't inherently good or bad either, unless we choose to treat them as such.

We all know this, right? But we don't consistently apply it to our thinking of emotions. In particular, this has two major implications:

1. You are not the world: It's always alright to feel good. Whether you're feeling good or bad won't change the state of the world: the world is only changed by the actual actions you take. You're never obligated to feel bad, or guilty, or ashamed. In particular, since you can only influence the world through your actions, you will accomplish more and be happier if your emotions are tied to your actions, not states of the world.
2. Emotional acceptance: At the same time, "negative" emotions are not something to suppress or flinch away from. They're a feedback mechanism which imprints lessons directly into your automatic behavior (your elephant). With your subconsciousness having been trained to act better in the future, your conscious mind is free to concentrate on other things. If the feedback system is broken and teaching you bad lessons, then you should act to correct it. But if the pain is about some real mistake or real loss you suffered, then you should welcome it.

Internalizing these lessons can have some very powerful effects. I've been making very good progress on consistently feeling better after starting to train myself to think like this. But some LW posters are even farther along; witness Will Ryan:

continue reading »

Bangalore Meetup: 28th May

5 [deleted] 16 May 2011 06:36AM

Bangalore hopefully has enough LessWrongers to have its own meet up. I suggest having one in the afternoon on Saturday, 28th May at Cubbon Park Bandstand. (Behind the high court in Cubbon park).

I commit to be there from 4 pm-7 pm with a LW meetup sign and a book (can't commit to which book I'll be reading two weeks hence, so will edit that in later. I'll be wearing a red kurta though). Since I don't know any Bangaloreans here and have never stood around in a public park holding up a sign before, comments showing interest will be much appreciated as morale-boosters. :)

I'm open to suggestions regarding both time and place.

continue reading »

A Suite of Pragmatic Considerations in Favor of Niceness

82 Alicorn 05 January 2010 09:32PM

tl;dr: Sometimes, people don't try as hard as they could to be nice.  If being nice is not a terminal value for you, here are some other things to think about which might induce you to be nice anyway.

There is a prevailing ethos in communities similar to ours - atheistic, intellectual groupings, who congregate around a topic rather than simply to congregate - and this ethos says that it is not necessary to be nice.  I'm drawing on a commonsense notion of "niceness" here, which I hope won't confuse anyone (another feature of communities like this is that it's very easy to find people who claim to be confused by monosyllables).  I do not merely mean "polite", which can be superficially like niceness when the person to whom the politeness is directed is in earshot but tends to be far more superficial.  I claim that this ethos is mistaken and harmful.  In so claiming, I do not also claim that I am always perfectly nice; I claim merely that I and others have good reasons to try to be.

The dispensing with niceness probably springs in large part from an extreme rejection of the ad hominem fallacy and of emotionally-based reasoning.  Of course someone may be entirely miserable company and still have brilliant, cogent ideas; to reject communication with someone who just happens to be miserable company, in spite of their brilliant, cogent ideas, is to miss out on the (valuable) latter because of a silly emotional reaction to the (irrelevant) former.  Since the point of the community is ideas; and the person's ideas are good; and how much fun they are to be around is irrelevant - well, bringing up that they are just terribly mean seems trivial at best, and perhaps an invocation of the aforementioned fallacy.  We are here to talk about ideas!  (Interestingly, this same courtesy is rarely extended to appalling spelling.)

The ad hominem fallacy is a fallacy, so this is a useful norm up to a point, but not up to the point where people who are perfectly capable of being nice, or learning to be nice, neglect to do so because it's apparently been rendered locally worthless.  I submit that there are still good, pragmatic reasons to be nice, as follows.  (These are claims about how to behave around real human-type persons.  Many of them would likely be obsolete if we were all perfect Bayesians.)

continue reading »

Buenos Aires meetup: Saturday, May 14th, 3pm

4 Pablo_Stafforini 13 May 2011 09:15PM

View more: Prev | Next