Over at Scott Adams' Blog you can find a very fine example of using the 'Rationality Engine' to solve the social problem of assisted dying.

A piece I saw that Benjamin Todd adapted from THINK's module on charity assessment. Some of you may recall the network's recent launch. 

Lots of social interventions end up doing more harm than good. Many more make no difference at all, and are just a waste of resources. At times, we’ve probably argued with friends about which interventions we’d like to see, and which we wouldn’t. But are we any good at judging what’s likely to work?

Here’s a cool bit of content adapted from THINK. Try and guess which of these eight programs made a difference, which had no effect, and which made things worse.

cipergoth said that it should be emphasised that this isn't a trick question where the answer is they all worked or none did.

---

Round #1: Scared Straight

Program description: “In the 1970s, inmates serving life sentences at a New Jersey (USA) prison began a program to ‘scare’ or deter at‐risk or delinquent children from a future life of crime. The program, known as ‘Scared Straight’, featured as its main component an aggressive presentation by inmates to juveniles visiting the prison facility. The presentation depicted life in adult prisons, and often included exaggerated stories of rape and murder … The program received considerable and favorable media attention and was soon replicated in over 30 jurisdictions nationwide … Although the harsh and sometimes vulgar presentation in the earlier New Jersey version is the most famous, inmate presentations are now sometimes designed to be more educational than confrontational but with a similar crime prevention goal. Some of these programs featured interactive discussions between the inmates and juveniles, also referred to as ‘rap sessions.’(2)

Did the program decrease the rate of juvenile crime?

Round #2: Nurse‐Family Partnership

Program description: “The Nurse‐Family Partnership program provides nurse home visits to pregnant women with no previous live births, most of whom are i) low‐income, ii) unmarried, and iii) teenagers. The nurses visit the women approximately once per month during their pregnancy and the first two years of their children’s lives. The nurses teach i) positive health related behaviors, ii) competent care of children, and iii) maternal personal development (family planning, educational achievement, and participation in the workforce). The program costs approximately $12,500 per woman over the three years of visits (in 2010 dollars).”(6)

Did the program improve the quality of child care?

Round #3: Drug Abuse Resistance Education (DARE)

Program description: “DARE is a highly‐structured substance‐abuse prevention program taught by uniformed police officers … The program is typically provided over the course of 10‐20 weekly hour‐long sessions, during which the police officers use lectures, class discussion, role plays, and homework assignments to i) teach students about substance use and its effects, ii) teach students decision‐making and peer pressure resistance skills, and iii) boost students’ self‐esteem. Prior to teaching, the police officers take an 80‐hour training course on teaching techniques, classroom management, and the DARE curriculum … DARE costs approximately $130 (in 2004 dollars) per student and, as of 2001, was operating in 75% of American school districts.”(8)

Did the program decrease the rate of drug use?

Rounds #4 and #5: 21st Century Community Learning Centers

Program description: “21st Century Community Learning centers is a large ($1 billion per year) US Department of Education program which funds optional after‐school programs for elementary and middle school students in mostly high‐poverty schools. Key goals of the program are to i) provide students with a safe place after school, and ii) improve their academic performance. Recipients of program funds (ie, school districts and/or non‐profit educational/community organizations) are required to provide academically focused “extended learning activities” (e.g., instructional enrichment programs, tutoring, or homework assistance). Most centers also offer enrichment/recreational activities such as martial arts, sports, dance, art and/or music … (Elementary school) centers vary in the activities they offer and other key features, and thus comprise a range of after‐school interventions rather than a single intervention. In a typical center,

1. students may spend an hour doing homework and having a snack, an hour on additional academic activity (eg, a lesson or working in a computer lab), and an hour doing recreational or cultural activities;

2. the center’s staff are a mixture of certified teachers, instructional aides, and representatives of community youth organizations;

3. the center is open 4‐5 days per week for three hours after school, and serves approximately 85 students per day; and

4. the average student attends the center 2‐3 days per week.

Centers spend approximately $1,000 (in 2005 dollars) on each enrolled student per year.”(10)

Did the program increase the students’ academic achievement?

Did the program improve the behavioural problems at the schools?

Round #6: Even Start Family Literacy program

Program description: “The Even Start program is intended to ‘help break the cycle of poverty and illiteracy by improving the educational opportunities of the nation’s low‐income families by integrating early childhood education, adult literacy or adult basic education, and parenting education into a unified family literacy program’. In 2000‐2001, there were 855 Even Start projects serving 31,896 families … Even Start grantees had considerable flexibility in designing services to meet the needs of the low‐income families, but all were required to offer four services:

1. adult education to develop basic educational and literacy skills;

2. early childhood education services to provide developmentally appropriate services to help prepare children for school;

3. parenting education to help parents support the educational growth of their children; and

4. parent‐child literacy activities.”(13)

Did the program increase literacy?

Round #7: Big Brothers Big Sisters

Program description: “Big Brothers Big Sisters’ community‐based mentoring program matches youths aged 6‐18, predominantly from low‐income, single‐parent households, with adult volunteer mentors who are typically young (20‐34) and well‐educated (the majority are college graduates) … The mentor and youth typically meet for 2‐4 times per month for at least a year, and engage in activities of their choosing (e.g., studying, cooking, playing sports). The typical meeting lasts 3‐4 hours … For the first year, Big Brothers Big Sisters case workers maintain monthly contact with the mentor, as well as the youth and his or her parent, to insure a positive mentor‐youth match, and to help resolve any problems in the relationship. Mentors are encouraged to form a supportive friendship with the youths, as opposed to modifying the youth’s behavior or character… In 2008, Big Brothers Big Sisters served 255,000 youths and 470 agencies nationwide. The national average cost of making and supporting a match is approximately $1,300 in 2009 dollars.”(14)

Did the program decrease drug use and violent behavior?

Round #8: Top 16 Educational Software

Program description: “In the No Child Left Behind Act of 2002, Congress called for a rigorous study of the effectiveness of educational technology for improving student academic achievement … In fall 2003, developers and vendors of educational technology products responded to a public invitation and submitted products for possible inclusion in the national study. Mathematica Policy Research, Inc. staff selected 40 of the 160 submissions for further review by two panels of outside experts, one for reading products and one for math products … In January 2004, (the US Department of Education] considered the panel’s recommendations and selected 16 products for the study. In selecting products, (the US Department of Education) grouped them into four areas:

1. early reading (first grade),

2. reading comprehension (fourth grade),

3. pre‐algebra (sixth grade), and

4. algebra (ninth grade).

The products ranged widely in their instructional approaches and how long they had been in use. In general, however, the criteria weighted the selection towards products that had evidence of effectiveness from previous research, or, for newer products, evidence that their designs were based on approaches found to be effective by research. Twelve of the sixteen products had received awards or been nominated for awards (some as recently as 2006) by trade associations, media, teachers, or parents.”(15)

Did the program improve test scores?

---

Here are the answers!

---

Round #1: Scared Straight

Negative! Several randomized controlled trials have shown that Scared Straight had a negative effect. Going through Scared Straight made children more likely to commit crimes in the future (3). Fun fact: Scared Straight programs are still being run today (4), and people promote them as being effective, despite the fact that they are harmful (5).

Round #2: Nurse‐Family Partnership

Positive! Three randomized controlled trials have shown that the Nurse‐Family Partnership had a positive effect. The program led to a reduction in child abuse/neglect, child injuries (20‐50% reduction) and an improvement in cognitive/educational outcomes for children of mothers with low mental health/confidence/intelligence (e.g., 6 percentile point increase in grade 1‐6 in reading/math achievement) (7).

Round #3: Drug Abuse Resistance Education (DARE)

No effect!Two randomized controlled trials have shown that DARE did not have an effect on the rate of drug use among participants. The rate of drug use did not increase or decrease (9).

Round #4: 21st Century Community Learning Centers

No effect! A randomized controlled trial has shown that the 21st Century Community Learning Centers had no effect on participating students’ academic performance. Students who participated were neither helped nor harmed by the program.(11)

Round #5: 21st Century Community Learning Centers

Negative! A randomized controlled trial has shown that the 21st Century Community Learning Centers caused an increase in the behavioral problems of participating students (12).

Round #6: Even Start Family Literacy Program

No effect! A randomized controlled trial on a subset of Even Start programs found no evidence of an increase or decrease in literacy in parents or children (17).

Round #7: Big Brothers Big Sisters

Positive! A randomized controlled trial has shown that Big Brothers Big Sisters caused youths to be 46% less likely to have started using illegal drugs, 27% less likely to have started using alcohol, 32% less likely to have hit someone in the previous year and fewer days of skipping school during the past year (18).

Round #8: Top 16 Educational Software

No effect!The study described was a randomized controlled trial, and showed that the software did not make a noticeable difference in any of the categories. It did not help or hurt with 1) early reading (first grade), 2) reading comprehension (fourth grade), 3) pre‐ algebra (sixth grade), or 4) algebra (ninth grade) (19).

---

How did you do?

If you got 7-8 right, there’s less than a 1% chance you were guessing. If you got 5-6 right, there was only an 8.5% chance you were guessing, so it might be skill. If you got 1-4 right, then you did no better than randomly guessing. If you got zero right … we could get useful information by always doing the opposite of what you do.

The effects of social interventions are extremely complex. All of these programs sound good, but unintended consequences can get in the way. It’s very difficult to work out what’s going to be successful ahead of time. Instead, we need to test, measure the results, and take it from there.

I thought Round 2 would have no effect and expected Round #5 to have no effect not a negative one, I got 6 out of 8 correct. How well did you do?

I recommend checking out the links and references. Gwern's comment there was also interesting.

The Economist has a new article on ethical dilemmas faced by machine designers.

Evidently:

1. In the event of an immoral decision by a machine, neural networks make it too hard to know who is at fault--the programmer, the operator, the manufacturer, or the designer. Thus, neural networks might be a bad idea.

2. Robots' ethical systems ought to resonate with "most people."

3. Proper robot consciences are more likely to arise given greater collaboration among engineers, ethicists, policymakers, and lawyers. Key quotation:

Both ethicists and engineers stand to benefit from working together: ethicists may gain a greater understanding of their field by trying to teach ethics to machines, and engineers need to reassure society that they are not taking any ethical short-cuts.

The second clause of the above sentence is quite similar to something Yudkowsky wrote, perhaps more than once, about the value of approaching ethics from an AI standpoint. I do not recall where he wrote it, nor did my search turn up the appropriate post.

Evolution. Morality. Strategy. Security/Cryptography. This hits so many topics of interest, I can't imagine it not being discussed here. Bruce Schneier blogs about his book-in-progress, The Dishonest Minority:

Humans evolved along this path. The basic mechanism can be modeled simply. It is in our collective group interest for everyone to cooperate. It is in any given individual's short-term self interest not to cooperate: to defect, in game theory terms. But if everyone defects, society falls apart. To ensure widespread cooperation and minimal defection, we collectively implement a variety of societal security systems.

I am somewhat reminded of Robin Hanson's Homo Hypocritus writings from the above, although it is not the same. Schneier says that the book is basically a first draft at this point, and might still change quite a bit. Some of the comments focus on whether "dishonest" is actually the best term to use for defecting from social norms.

From "Academics Doubt Impact of Osama bin Laden’s Death":

"...Fifty-three percent of the terrorist organizations that suffered such a violent leadership loss fell apart — which sounds impressive until you discover that 70 percent of groups who did not deal with an assassination no longer exist.

Further crunching of the numbers revealed that leadership decapitation becomes more counterproductive the older the group is. The difference in collapse rates (between groups that did and did not have a leader assassinated) is fairly small among organizations less than 20 years old but quite large for those more than 20 years in age, and even larger for those that have been around more than 30 years.

Assassination of a leader does seem to negatively impact smaller terrorist groups: The data shows organizations with fewer than 500 members are more likely to collapse if they suffer such a leadership loss. But organizations with more than 500 members are actually more likely to survive after an assassination, making this strategy “highly counterproductive for larger groups,” Jordan writes."

See also Lost Purposes, The Importance of Goodhart's Law, & Faster than Science.

During discussion in my previous post, when we touched the subject of human statistical majorities, I had a side-thought. If taking the Less Wrong audience as an example, the statistics say that any given participant is strongly likely to be white, male, atheist, and well, just going by general human statistics, probably heterosexual.

But in my actual interaction, I've taken as a rule not to make any assumptions about the other person. Does it mean, I thought, that I reset my prior probabilities, and consciously choose to discard information? Not relying on implicit assumptions seems the socially right thing to do, I thought; but is it rational?

When I discussed it on IRC, this quote by sh struck me as insightful:

I.e. making the guess incorrectly probably causes far more friction than deliberately not making a correct guess you could make.

I came up with the following payoff matrix:

In this case, the second option is strictly preferable. In other words, I don't discard the information, but the repercussions to our social interaction in case of an incorrect guess outweigh the benefit from guessing correctly. And it also matters whether either Alice or Bob is an Asker or a Guesser.

One consequence I can think of is that with a sufficiently low p, or if Bob wouldn't be particularly offended by Alice's incorrect guess, taking the guess would be preferable. Now I wonder if we do that a lot in daily life with issues we don't consider controversial ("hmm, are you from my country/state too?"), and if all the "you're overreacting/too sensitive" complaints come from Alice incorrectly assessing a too low-by-absolute-value negative payoff in (0, 1).