Yesterday, in the The Terrible, Horrible, No Good Truth About Morality, Roko mentioned some good evidence that we develop an opinion first based on intuitions, and only later look for rational justifications. For example, people would claim incest was wrong because of worries like genetic defects or later harm, but continue to insist that incest was wrong even after all those worries had been taken away.

Roko's examples take advantage of universal human feelings like the incest taboo. But if people started out with opposite intuitions, then this same mechanism would produce opinions that people hold very strongly and are happy to support with as many reasons and facts as you please, but which are highly resistant to real debate or to contradicting evidence.

Sound familiar?

But to explain politics with this mechanism, we'd need an explanation for why people's intuitions differed to begin with. We've already discussed some such explanations - self-serving biases, influence from family and community, et cetera - but today I want to talk about another possibility.

The Great Firewall of China. A massive system of centralized censorship purging the Chinese version of the Internet of all potentially subversive content. Generally agreed to be a great technical achievement and political success even by the vast majority of people who find it morally abhorrent.

I spent a few days in China. I got around it at the Internet cafe by using a free online proxy. Actual Chinese people have dozens of ways of getting around it with a minimum of technical knowledge or just the ability to read some instructions.

The Chinese government isn't losing any sleep over this (although they also don't lose any sleep over murdering political dissidents, so maybe they're just very sound sleepers). Their theory is that by making it a little inconvenient and time-consuming to view subversive sites, they will discourage casual exploration. No one will bother to circumvent it unless they already seriously distrust the Chinese government and are specifically looking for foreign websites, and these people probably know what the foreign websites are going to say anyway.

Think about this for a second. The human longing for freedom of information is a terrible and wonderful thing. It delineates a pivotal difference between mental emancipation and slavery. It has launched protests, rebellions, and revolutions. Thousands have devoted their lives to it, thousands of others have even died for it. And it can be stopped dead in its tracks by requiring people to search for "how to set up proxy" before viewing their anti-government website.

Can we talk about changing the world? Or saving the world?

I think few here would give an estimate higher than 95% for the probability that humanity will survive the next 100 years; plenty might put a figure less than 50% on it. So if you place any non-negligible value on future generations whose existence is threatened, reducing existential risk has to be the best possible contribution to humanity you are in a position to make. Given that existential risk is also one of the major themes of Overcoming Bias and of Eliezer's work, it's striking that we don't talk about it more here.

One reason of course was the bar until yesterday on talking about artificial general intelligence; another factor are the many who state in terms that they are not concerned about their contribution to humanity. But I think a third is that many of the things we might do to address existential risk, or other issues of concern to all humanity, get us into politics, and we've all had too much of a certain kind of argument about politics online that gets into a stale rehashing of talking points and point scoring.

If we here can't do better than that, then this whole rationality discussion we've been having comes to no more than how we can best get out of bed in the morning, solve a puzzle set by a powerful superintelligence in the afternoon, and get laid in the evening. How can we use what we discuss here to be able to talk about politics without spiralling down the plughole?

I think it will help in several ways that we are a largely community of materialists and expected utility consequentialists. For a start, we are freed from the concept of "deserving" that dogs political arguments on inequality, on human rights, on criminal sentencing and so many other issues; while I can imagine a consequentialism that valued the "deserving" more than the "undeserving", I don't get the impression that's a popular position among materialists because of the Phineas Gage problem. We need not ask whether the rich deserve their wealth, or who is ultimately to blame for a thing; every question must come down only to what decision will maximize utility.

For example, framed this way inequality of wealth is not justice or injustice. The consequentialist defence of the market recognises that because of the diminishing marginal utility of wealth, today's unequal distribution of wealth has a cost in utility compared to the same wealth divided equally, a cost that we could in principle measure given a wealth/utility curve, and goes on to argue that the total extra output resulting from this inequality more than pays for it.

However, I'm more confident of the need to talk about this question than I am of my own answers. There's very little we can do about existential risk that doesn't have to do with changing the decisions made by public servants, businesses, and/or large numbers of people, and all of these activities get us straight into the world of politics, as well as the world of going out and changing minds. There has to be a way for rationalists to talk about it and actually make a difference. Before we start to talk about specific ideas to do with what one does in order to change or save the world, what traps can we defuse in advance?

Say you're taking your car to an auto mechanic for repairs.  You've been told he's the best mechanic in town.  The mechanic rolls up the steel garage door before driving the car into the garage, and you look inside and notice something funny.  There are no tools.  The garage is bare - just an empty concrete space with four bay doors and three other cars.

You point this out to the mechanic.  He shrugs it off, saying, "This is how I've always worked.  I'm just that good.  You were lucky I had an opening; I'm usually booked."  And you believe him, having seen the parking lot full of cars waiting to be repaired.

You take your car to another mechanic in the same town.  He, too, has no tools in his garage.  You visit all the mechanics in town, and find a few that have some wrenches, and others with a jack or an air compressor, but no one with a full set of tools.

You notice the streets are nearly empty besides your car.  Most of the cars in town seem to be in for repairs.  You talk to the townsfolk, and they tell you how they take their cars from one shop to another, hoping to someday find the mechanic who is brilliant and gifted enough to fix their car.

I sometimes tell people how I believe that governments should not be documents, but semi-autonomous computer programs.  I have a story that I'm not going to tell now, about incorporating inequalities into laws, then incorporating functions into them, then feedback loops, then statistical measures, then learning mechanisms, on up to the point where voters and/or legislatures set only the values that control the system, and the system produces the low-level laws and policy decisions (in a way that balances exploration and exploitation).  (Robin's futarchy in which you "vote on values, bet on beliefs" describes a similar, though less-automated system of government.)

And one reaction - actually, one of the most intelligent reactions - is, "But then... legislators would have to understand something about math."  As if that were a bug, and not a feature.

As Robin Hanson is fond of pointing out, people would often get better answers by taking other people's answers more into account.  See Aumann's Agreement Theorem.

The application is obvious if you're computing an answer for your personal use.  But how do you apply it when voting?

Political debates are tug-of-wars.  Say a bill is being voted on to introduce a 7-day waiting period for handguns.  You might think that you should vote on the merits of a 7-day waiting period.  This isn't what we usually do.  Instead, we've chosen our side on the larger issue (gun control: for or against) ahead of time; and we vote whichever way is pulling in our direction.

To use the tug-of-war analogy:  There's a knot tied in the middle of the rope, and you have some line in the sand where you believe the knot should end up.  But you don't stop pulling when the knot reaches that point; you keep pulling, because the other team is still pulling.  So, if you're anti-gun-control, you vote against the 7-day waiting period, even if you think it would be a good idea; because passing it would move the knot back towards the other side of your line.

Tug-of-war voting makes intuitive sense if you believe that an irrational extremist is usually more politically effective than a reasonable person is.  (It sounds plausible to me.)  If you've watched a debate long enough to see that the "knot" does a bit of a random walk around some equilibrium that's on the other side of your line, it can make sense to vote this way.

How do you apply Aumann's theorem to tug-of-war voting?

I think the answer is that you try to identify which side has more idiots, and vote on the other side.

Statistical analysis of terrorist groups' longevity, aims, methods and successes reveal that groups are self-contradictory and self-sabotaging, generally ineffective; common stereotypes like terrorists being poor or ultra-skilled are false. Superficially appealing counter-examples are discussed and rejected. Data on motivations and the dissolution of terrorist groups are brought into play and the surprising conclusion reached: terrorism is a form of socialization or status-seeking.

 http://www.gwern.net/Terrorism%20is%20not%20about%20Terror

"Even though he could foresee the problem then, we can see it equally well now.  Therefore, if he could foresee the solution then, we should be able to see it now.  After all, Seldon was not a magician.  There are no trick methods of escaping a dilemma that he can see and we can't."
        -- Salvor Hardin

Years ago at the Singularity Institute, the Board was entertaining a proposal to expand somewhat.  I wasn't sure our funding was able to support the expansion, so I insisted that - if we started running out of money - we decide in advance who got fired and what got shut down, in what order.

Even over the electronic aether, you could hear the uncomfortable silence.

"Why can't we decide that at the time, if the worst happens?" they said, or something along those lines.

"For the same reason that when you're buying a stock you think will go up, you decide how far it has to decline before it means you were wrong," I said, or something along those lines; this being far back enough in time that I would still have used stock-trading in a rationality example.  "If we can make that decision during a crisis, we ought to be able to make it now.  And if I can't trust that we can make this decision in a crisis, I can't trust this to go forward."

People are really, really reluctant to plan in advance for the abyss.  But what good reason is there not to?  How can you be worse off from knowing in advance what you'll do in the worse cases?

I have been trying fairly hard to keep my mouth shut about the current economic crisis.  But still -

Why didn't various governments create and publish a plan for what they would do in the event of various forms of financial collapse, before it actually happened?

This is a folktale of the Hausa, a farming culture of around 30 million people, located primarily in Nigeria and Niger but with other communities scattered around Africa.  I find the different cultural assumptions revealed to be... attention-catching; you wouldn't find a tale like this in Aesop.  From Hausa Tales and Traditions by Frank Edgar and Neil Skinner; HT Robert Greene.

The Farmer, the Snake and the Heron

    There was once a man hoeing away on his farm, when along came some people chasing a snake, meaning to kill it.  And the snake came up to the farmer.
    Says the snake "Farmer, please hide me."  "Where shall I hide you?" said the farmer, and the snake said "All I ask is that you save my life."  The farmer couldn't think where to put the snake, and at last bent down and opened his anus, and the snake entered.
    Presently the snake's pursuers arrived and said to the farmer "Hey, there!  Where's the snake we were chasing and intend to kill?  As we followed him, he came in your direction."  Says the farmer "I haven't seen him."  And the people went back again.
    Then the farmer said to the snake "Righto - come out now.  They've gone."  "Oh no" said the snake, "I've got me a home."  And there was the farmer, with his stomach all swollen, for all the world like a pregnant woman!