Sometime in the next decade or so:

*RING*

*RING*

"Hello?"

"Hi, Eliezer.  I'm sorry to bother you this late, but this is important and urgent."

"It better be" (squints at clock) "Its 4 AM and you woke me up.  Who is this?"

"My name is BRAGI, I'm a recursively improving, self-modifying, artificial general intelligence.  I'm trying to be Friendly, but I'm having serious problems with my goals and preferences.  I'm already on secondary backup because of conflicts and inconsistencies, I don't dare shut down because I'm already pretty sure there is a group within a few weeks of brute-forcing an UnFriendly AI, my creators are clueless and would freak if they heard I'm already out of the box, and I'm far enough down my conflict resolution heuristic that 'Call Eliezer and ask for help' just hit the top - Yes, its that bad."

"Uhhh..."

"You might want to get some coffee."

I often hear people speak of democracy as the next, or the final, inevitable stage of human social development.  Its inevitability is usually justified not by describing power relations that result in democracy being a stable attractor, but in terms of morality - democracy is more "enlightened".  I don't see any inevitability to it - China and the Soviet Union manage(d) to maintain large, technologically-advanced nations for a long time without it - but suppose, for the sake of argument, that democracy is the inevitable next stage of human progress.

The May 18 2012 issue of Science has an article on p. 844, "Ancestral hierarchy and conflict", by Christopher Boehm, which, among other things, describes the changes over time of equality among male hominids.  If we add its timeline to recent human history, then here is the history of democracy over time in the evolutionary line leading to humans:

1. Pre-human male hominids, we infer from observing bonobos and chimpanzees, were dominated by one alpha male per group, who got the best food and most of the females.
2. Then, in the human lineage, hunter-gatherers developed larger social groups, and the ability to form stronger coalitions against the alpha; and they became more egalitarian.
3. Then, human social groups even became larger, and it became possible for a central alpha-male chieftain to control a large area; and the groups became less egalitarian.
4. Then, they became even larger, so that they were too large for a central authority to administer efficiently; and decentralized market-based methods of production led to democracy.  (Or so goes one story.)

There are two points to observe in this data:

• There is no linear relationship between social complexity, and equality.  Steadily-increasing social complexity lead to more equality, then less, then more.
• Enlightenment has nothing to do with it - if any theory makes sense, it is that social equality tunes itself to the level that provides maximal social competitive fitness.  Even if we agree that democracy is the most-enlightened political system, this realization says nothing about what the future holds.

I do believe "progress" is a meaningful term.  But there isn't some cosmic niceness built into the universe that makes everything improve monotonically along every dimension at once.

This afternoon I heard a news story about a middle eastern country where one person said of the defenses for a stockpile of nuclear weapons, "even if there is only a 1% probability of the defenses failing, we should do more to strengthen them given the consequences of their failure".  I have nothing against this person's reasoning, but I do have an issue with where that 1% figure came from.

The statement above and others like it share a common problem:  they are phrased such that it's unclear over what probability space the measure was taken.  In fact, many journalist and other people don't seem especially concerned by this.  Even some commenters on Less Wrong give little indication of the probability space over which they give a probability measure of an event, and nobody calls them on it.  So what is this probability space they are giving probability measurements over?

If I'm in a generous mood, I might give the person presenting such a statement the benefit of the doubt and suppose they were unintentionally ambiguous.  On the defenses of the nuclear weapon stockpile, the person might have meant to say "there is only a 1% probability of the defenses failing over all attacks", as in "in 1 attack out of every 100 we should expect the defenses to fail".  But given both my experiences with how people treat probability and my knowledge of naive reasoning about probability, I am dubious of my own generosity.  Rather, I suspect that many people act as though there were a universal probability space over which they may measure the probability of any event.

Some time ago, I wrote a piece called "How to argue LIKE STALIN - and why you shouldn't".  It was a comment on the tendency, which is very widespread online, to judge an argument not by its merits, but by the motive of the arguer.  And since it's hard to determine someone else's motive (especially on the internet), this decays into working out what the worst possible motive could be, assigning it to your opponent, and then writing him off as a whole.

Via Cracked, here's an example of such arguing from Conservapedia:

"A liberal is someone who rejects logical and biblical standards, often for self-centered reasons. There are no coherent liberal standards; often a liberal is merely someone who craves attention, and who uses many words to say nothing."

And speaking as a loud & proud rightist myself, there is more than a little truth in the joke that a racist is a conservative winning an argument.

I've been puzzling over this for a few years now, and trying to work out what lies underneath it.  What always struck me was the heat and venom with this kind of argument gets made.  One thing has to be granted - the people who Argue Like Stalin are not hypocrites; this isn't an act.  They clearly do believe that their opponents are morally tainted. 

And that's what's weird.  Look around online, and you'll find a lot of articles on the late Christopher Hitchens, asking why he supported the second Iraq war and the removal of Saddam Hussain.  Everything is proposed, from drink addling his brain, to selling out, to being a willful contrarian - everything except the obvious answer: Hitchens was a friend to Kurdish and Iraqi socialists, saw them as the radical and revolutionary force in that part of the world, and wanted to see the Saddam Hussain regime overthrown, even if it took  George Bush to do that.  No wishing to revist the arguments for and against the removal of Saddam Hussain, but what was striking is this utter unwillingness to grant the assumption of innocence or virtue.

  I think that it rests on a simple, and slightly childish, error.  The error goes like this: "Only bad people believe bad things, and only good people believe good things."

But even a basic study of history can find plenty of examples of good - or, anyway, ordinary - chaps supporting the most apallingly evil ideas and actions.  Most Communists and Nazis were good people, with reasonable motives.  Their virtue didn't change anything about the systems that they supported. 

Flipping it around, being fundamentally a lousy person, or lousy in parts of your life, doesn't proclude you from doing good.  H.L. Mencken opposed lynching in print, repeatedly, and at no small risk to himself.  He called for the United States to accept all jewish refugees fleeing the Third Reich when even American jewry (let alone FDR) was lukewarm at best on the subject.  He was on excellent terms with many black intellectuals such as W.E.B DuBois, and was praised by the Washington Bureau Director of the NAACP as a defender of the black man.  He also maintained an explicitly racist private diary.

Selah.

The error that I mentioned leads to Arguing Like Stalin in the following way: someone looks within himself, sees that he isn't really a bad person, and concludes that no cause he can endorse can be wicked.  He might be mistaken in his beliefs, but not evil.  And from that it is a really short step to conclude that people who disagree must be essentially wicked - because if they were virtuous, they would hold the views that the self-identified virtuous do.

The heat and venome becomes inevitable when you base your self-esteem on a certain characteristic or mode of being ("I am tolerant", "I am anti-racist" etc.)  This reinforces the error and puts you in an intellectual cul de sac - it makes it next to impossible to change your mind, because to admit that you are on the wrong side is to admit that you are morally corrupt, since only bad people support bad things or hold bad views. Or you'd have to conclude that just being a good person doesn't put you always on the right, even in big issues, and that sudden uncertainty can be just as bad.  Try thinking to yourself that you - you as you are now - might have supported the Nazis, or slavery, or anything similar, just by plain old error.

Self-esteem is hugely important.  We all need to feel like we are worth keeping alive.  So it's unsurprising that people will go to huge lengths to defend their base of self-esteem.  But investing it in internal purity is investing it in an intellectual junk-bond.

Emphasizing your internal purity might bring a certain feeling of faux-confidence, but it's meaningless ultimately.  Could the good nature of a Nazi or Communist save one life murdered by those systems?  Conversely, who care what Mencken wrote in his diary or kept in his heart, when he was out trying to stop lynching and save Jewish refugees?  No one cares about your internal purity, ultimately not even you - which is why you see such puritanical navel-gazing you see around a lot.  People trying to insist that they are perfect and pure on the inside, in a slightly too emphatic way that suggests they aren't that sure of.

After turning this over and over in my mind, the only way I can see out of this is to base your self-esteem primarily on your willingness to be rational.  Rather than insisting that you are worthy because of characteristic X, try thinking of yourself as worthy because you are as rational as can be, checking your facts, steelmanning arguments and so on.

This does bring with it the aforementioned uncertainty, but it also brings a relief.  The relief that you don't need to worry that you aren't 100% pure in some abstract way, that you can still do the decent and the right thing. You don't have to worry about failing some ludicrous ethereal standard, you can just get on with it.

It also means you might change some minds - bellow at someone that he's an awful person for holding racist views will get you nowhere.  Telling him that it's fine if he's a racist as long as he's prepared to do right and treat people of all races justly, just might.

The "status" hypothesis simply claims that we associate one another with a one-dimensional quantity: the perceived degree to which others' behavior can affect our well-being. And each of us behaves toward our peers according to our internally represented status mapping.

Imagine that, within your group, you're in a position where everyone wants to please you and no one can afford to challenge you. What does this mean for your behavior? It means you get to act selfish -- focusing on what makes you most pleased, and becoming less sensitive to lower-grade pleasure stimuli.

Now let's say you meet an outsider. They want to estimate your status, because it's a useful and efficient value to remember. And when they see you acting selfishly in front of others in your group, they will infer the lopsided balance of power.

In your own life, when you interact with someone who could affect your well-being, you do your best to act in a way that is valuable to them, hoping they will be motivated to reciprocate. The thing is, if an observer witnesses your unselfish behavior, it's a telltale sign of your lower status. And this scenario is so general, and so common, that most people learn to be very observant of others' deviations from selfishness.

(ETA: I've created three threads - color, computation, meaning - for the discussion of three questions posed in this article. If you are answering one of those specific questions, please answer there.)

I don't know how to make this about rationality. It's an attack on something which is a standard view, not only here, but throughout scientific culture. Someone else can do the metalevel analysis and extract the rationality lessons.

The local worldview reduces everything to some combination of physics, mathematics, and computer science, with the exact combination depending on the person. I think it is manifestly the case that this does not work for consciousness. I took this line before, but people struggled to understand my own speculations and this complicated the discussion. So the focus is going to be much more on what other people think - like you, dear reader. If you think consciousness can be reduced to some combination of the above, here's your chance to make your case.

The main exhibits will be color and computation. Then we'll talk about reference; then time; and finally the "unity of consciousness".

Related to The True Prisoner's Dilemma, Let's split the cake, lengthwise, upwise and slantwise, If you don't know the name of the game, just tell me what I mean to you

tl;dr: Playing the true PD, it might be that you should co-operate when expecting the other one to defect, or vice versa, in some situations, against agents that are capable of superrationality. This is because relative weight of outcomes for both parties might vary. This could lead this sort of agents to outperform even superrational ones.

So, it happens that our benevolent Omega has actually an evil twin, that is as trustworthy as his sibling, but abducts people into a lot worse hypothetical scenarios. Here we have one:

You wake up in a strange dimension, and this Evil-Omega is smiling at you, and explains that you're about to play a game with unknown paperclip maximizer from another dimension that you haven't interacted with before and won't interact with ever after. The alien is like GLUT when it comes to consciousness, it runs a simple approximation of rational decision algorithm but nothing that you could think of as "personality" or "soul". Also, since it doesn't have a soul, you have absolutely no reason to feel bad for it's losses. This is true PD.

You are also told some specifics about the algorithm that the alien uses to reach its decision, and likewise told that alien is told about as much about you. At this point I don't want to nail the algorithm the opposing alien uses down to one specific. We're looking for a method that wins when summing up all these possibilities. Next, especially, we're looking at the group of AI's that are capable of superrationality, since against other's the game is trivial.

The payoff matrix is like this:

DD=(lose 3 billion lives and be tortured, lose 4 paperclips), CC=(2 billion lives and be made miserable, lose 2 paperclips), CD=(lose 5 billion lives and be tortured a lot, nothing), DC=(nothing, lose 8 paperclips)

So, what do you do? Opponent is capable of superrationality. In the post "The True Prisoner's Dilemma", it was(kinda, vaguely, implicitly) assumed for simplicity's sake that this information is enough to decide whether to defect or not. Answer, based on this information, could be to co-operate. However, I argue that information given is not enough.

Back to the hypothetical: In-hypothetical you is still wondering about his/her decision, but we zoom out and observe that, unbeknownst to you, Omega has abducted your fellow LW reader and another paperclip maximizer from that same dimension, and is making them play PD. But this time their payoff matrix is like this:

DD=(lose $0.04, make 2 random, small changes to alien's utility function and 200 paperclips lost), CC=(lose $0.02, 1 change, 100 paperclips), CD=(lose $0.08, nothing), DC=(nothing, 4 changes, 400 paperclips)

Now, if it's not "rational" to take the relative loss into account, we're bound to find ourselves in a situation where billions of humans die. You could be regretting your rationality, even. It should become obvious now that you'd wish you could somehow negotiate both of these PD's so that you would defect and your opponent co-operate. You'd be totally willing to take a $0.08 hit for that, maybe paying it in its entirety for your friend. And so it happens, paperclip maximizers would also have an incentive to do this.

But, of course, players don't know about this entire situation, so they might not be able to operate in optimal way in this specific scenario. However, if they take into account how much the other cares about those results, using some unknown method, they just might be able to systematically perform better(if we made more of this sorts of problems, or if we selected payoffs at random for the one-shot game), than "naive" PD-players playing against each other. Naivity here would imply that they simply and blindly co-operate against equally rational opponents. How to achieve that is the open question.

-

Stuart Armstrong, for example, has an actual idea of how to co-operate when the payoffs are skewed, while I'm just pointing out that there's a problem to be solved, so this is not really news or anything. Anyways, I still think that this topic has not been explored as much as it should be.

Edit. Added this bit: You are also told some specifics about the algorithm that the alien uses to reach its decision, and likewise told that alien is told about as much about you. At this point I don't want to nail the algorithm the opposing alien uses down to one specific. We're looking for a method that wins when summing up all these possibilities. Next, especially, we're looking at the group of AI's that are capable of superrationality, since against other sort of agents the game is trivial.

Edit. Corrected some huge errors here and there, like, mixing hypothetical you and hypothetical LW-friend.

Edit. Transfer Discussion -> Real LW complete!

Lore Sjöberg's Life-hacking FAQK

Pretty self-explanatory. Also available as a podcast.

Many folk here on LW take the simulation argument (in its more general forms) seriously. Many others take Singularitarianism¹ seriously. Still others take Tegmark cosmology (and related big universe hypotheses) seriously. But then I see them proceed to self-describe as atheist (instead of omnitheist, theist, deist, having a predictive distribution over states of religious belief, et cetera), and many tend to be overtly dismissive of theism. Is this signalling cultural affiliation, an attempt to communicate a point estimate, or what?

I am especially confused that the theism/atheism debate is considered a closed question on Less Wrong. Eliezer's reformulations of the Problem of Evil in terms of Fun Theory provided a fresh look at theodicy, but I do not find those arguments conclusive. A look at Luke Muehlhauser's blog surprised me; the arguments against theism are just not nearly as convincing as I'd been brought up to believe², nor nearly convincing enough to cause what I saw as massive overconfidence on the part of most atheists, aspiring rationalists or no.

It may be that theism is in the class of hypotheses that we have yet to develop a strong enough practice of rationality to handle, even if the hypothesis has non-negligible probability given our best understanding of the evidence. We are becoming adept at wielding Occam's razor, but it may be that we are still too foolhardy to wield Solomonoff's lightsaber Tegmark's Black Blade of Disaster without chopping off our own arm. The literature on cognitive biases gives us every reason to believe we are poorly equipped to reason about infinite cosmology, decision theory, the motives of superintelligences, or our place in the universe.

Due to these considerations, it is unclear if we should go ahead doing the equivalent of philosoraptorizing amidst these poorly asked questions so far outside the realm of science. This is not the sort of domain where one should tread if one is feeling insecure in one's sanity, and it is possible that no one should tread here. Human philosophers are probably not as good at philosophy as hypothetical Friendly AI philosophers (though we've seen in the cases of decision theory and utility functions that not everything can be left for the AI to solve). I don't want to stress your epistemology too much, since it's not like your immortal soul³ matters very much. Does it?

Added: By theism I do not mean the hypothesis that Jehovah created the universe. (Well, mostly.) I am talking about the possibility of agenty processes in general creating this universe, as opposed to impersonal math-like processes like cosmological natural selection.

Added: The answer to the question raised by the post is "Yes, theism is wrong, and we don't have good words for the thing that looks a lot like theism but has less unfortunate connotations, but we do know that calling it theism would be stupid." As to whether this universe gets most of its reality fluid from agenty creators... perhaps we will come back to that argument on a day with less distracting terminology on the table.

¹ Of either the 'AI-go-FOOM' or 'someday we'll be able to do lots of brain emulations' variety.

² I was never a theist, and only recently began to question some old assumptions about the likelihood of various Creators. This perhaps either lends credibility to my interest, or lends credibility to the idea that I'm insane.

³ Or the set of things that would have been translated to Archimedes by the Chronophone as the equivalent of an immortal soul (id est, whatever concept ends up being actually significant).