Hrm. I'd think "avoid destroying the world" itself to be an ethical injunction too. (modulo all relevant caveats like all minds on earth uploading and deciding collectively to rest of the matter composing the planet for some other purpose, blah blah blah, you know what I mean)

"Even intrapersonal ethics can help protect you from black swans and fundamental mistakes. That logic doesn't change its structure when you double the value of the stakes, or even raise them to the level of a world."

• I'm not so sure. The kind of deontological ethics that you are talking about works well in human social interactions. When removed from that context, why do you think that it will still work?

For example, Knut Haukelid broke a deontological rule in order to make a significant dent in the subjective probability (given his knowledge) of the the nazis winning WWII. I think that he did the right thing, and I think that in such an extreme case one ought to act according to the greater good.

The problem is working out when one is in a sufficiently extreme case. For the readers of Overcoming Bias, and those interested in the singularity, this is a tough question. Clever men, such as yourself, tell us that the fate of the entire human race rests upon solving the FAI problem. Does this count as extreme? Does it count as extreme enough to justify damaging one's personal life, one's friends or family?

My answer to such questions of "greater good" versus "duty" used to be to favor the former, but my experiences in life have shown me that it is better to try to avoid such choices. Looking back on the times when I have stuck by my friends or my duties to my disadvantage, and the times where I have betrayed people or lied, (Yes, I have done both several times), I realize that in every single case there was a third option available if I had just thought about the problem clearly enough.

If somebody was planning to destroy the world, the rationalist could stop him and not break his oath of honesty by simply killing the psychopath. Then if the rationalist were caught and arrested and still didn't reveal why he had committed murder, perhaps even being condemned to death for the act but never breaking his oath of honesty, now that would make an awesome movie.

It might make an awesome movie, but if it were expected behaviour, it would defeat the point of the injunction. In fact if rationalists were expected to look for workarounds of any kind it would defeat the point of the injunction. So the injunction would have to be, not merely to be silent, but not to attempt to use the knowledge divulged to thwart the one making the confession in any way except by non-coercive persuasion.

Or alternatively, not to ever act in a way such that if the person making the confession had expected it they would have avoided making the confession.

If a serial killer comes to a confessional, and confesses that he's killed six people and plans to kill more, should the priest turn him in? I would answer, "No." If not for the seal of the confessional, the serial killer would never have come to the priest in the first place.

It's important to distinguish two ways this argument might work. The first is that the consequences of turning him in are bad, because future killers will be (or might be) less likely to seek advice from priests. That's a fairly straightforward utilitarian argument.

But the second is that turning him in is somehow bad, regardless of the consequences, because the world in which every "confessor" did as you do is a self-defeating, impossible world. This is more of a Kantian line of thought.

Eliezer, can you be explicit which argument you're making? I thought you were a utilitarian, but you've been sounding a bit Kantian lately. :)

Your deontological ethics are tiresome. Why not just be a utilitarian and lie your way to a better tomorrow?

Put more seriously, I would think that being believed to put the welfare of humanity ahead of concerns about personal integrity could have significant advantages itself.

Or put another way, when it's time to shut up and do the impossible (save humanity, say), that doesn't seem like a good time to defer to pre-established theories, of ethics or anything else. Refer, yes; defer, no. You say to beware of cleverness, be wary of thinking you're smarter than your ethics (meaning deontological beliefs and intuitions). That discussion sounded like a Hofstadter's Law ("It always takes longer than you expect, even when you take Hofstadter's Law into account.") for ethics. Yet, when the chips are down, when you've debugged your hardware as best you can, isn't our cleverness what we have to trust? What else could there be? After all, as you yourself said, rationality is ultimately about winning, and so however much you hedge against black swans and corrupt hardware, it can't be an infinite amount, and there must come a point where you should stop and do what your brain computes is the right thing to do.

If my ethics don't tell me to save the world, I have no ethics at all.

There seems to be a conflict here between not lying to yourself, and holding a traditional rule that suggests you ignore your rationality.

This is A way to deal with running on untrusted hardware, but I am far from convinced it is optimal.

Crossman: there's a third argument, which is that even if the consequences of keeping the secret are overall worse than those of betraying the confidence even after the effect you discuss, turning yourself into someone who will never betray these secrets no matter what the consequences and advertising yourself as such in an impossible-to-fake way may overall have good consequences. In other words, you might turn away from consequentialism on consequentialist grounds.

Another example where unfakeably advertising irrationality can (at least in theory) serve you is threats. My only way of stopping you from taking over the world is that I have the power to destroy the world and you. Now, if you take over the world, there's no possible advantage to destroying it, so I won't, so you can take the world over. But if I put a lunatic in charge of the button who believably will carry out the threat, you will be deterred; the same applies if I can become that lunatic.

However, overall I think that the arguments against turning yourself into a lunatic are pretty strong, and in fact I suspect that consequentialism has the best consequences.

Given that there are already lots of people seeking to stop atrocities, the presence of one more person trying to do the same seems likely to be irrelevant. But there are very few people who have even a chance to speak with atrocity-planners and possibly persuade them to do otherwise - effectively none.

Trying to decide what to do without looking at what most other people are likely to do is impossible. Taking the behavior of others into account, it is quite reasonable for one person to put another strategy into play.

If talking has even a small chance of working, the utility of applying it as well as other prevention strategies is greater than the others alone.

I am glad Stanislav Petrov, contemplating his military oath to always obey his superiors and the appropriate guidelines, never read this post.

Yvain: I am glad Stanislav Petrov, contemplating his military oath to always obey his superiors and the appropriate guidelines, never read this post.

An interesting point, for several reasons.

First, did Petrov actually swear such an oath, and would it apply in such fashion as to require him to follow the written policy rather than using his own military judgment?

Second, you might argue that Petrov's oath wasn't intended to cover circumstances involving the end of the world, and that a common-sense exemption should apply when the stakes suddenly get raised hugely beyond the intended context of the original oath. I think this fails, because Petrov was regularly in charge of a nuclear-war installation and so this was exactly the sort of event his oath would be expected to apply to.

Third, the Soviets arguably implemented what I called strategy 1 in this comment: Petrov did the right thing, and was censured for it anyway.

Fourth - maybe, on sober reflection, we wouldn't have wanted the Soviets to act differently! Yes, the written policy was stupid. And the Soviet Union was undoubtedly censuring Petrov out of bureaucratic coverup, not for reasons of principle. But do you want the Soviet Union to have a written, explicit policy that says, "Anyone can ignore orders in a nuclear war scenario if they think it's a good idea," or even an explicit policy that says "Anyone who ignores orders in a nuclear war scenario, who is later vindicated by events, will be rewarded and promoted"?

Paul, that's a good point.

Eliezer: If all I want is money, then I will one-box on Newcomb's Problem.

Mmm. Newcomb's Problem features the rather weird case where the relevant agent can predict your behaviour with 100% accuracy. I'm not sure what lessons can be learned from it for the more normal cases where this isn't true.

@Allan: Agent need not predict your overall behavior, only the outcome. If you are creating such agent, you are creating the situation where you have a system that will arrange the future context based on deep analysis of environment, and your other actions are forming this environment. Orchestrating actions consisting of intelligent agents requires this kind of reasoning.

"One suspects that Haukelid's price was far higher than most people would charge"

Or that he routinely killed people and just didn't mind.

Your sense of morality is so wayward.

Question - would you lie in order to win the AI box experiment?

Eliezer,

Crossman and Crowley make very good points above, delineating three possible types of justification for some of the things you say:

1) Don't turn him in because the negative effects of the undermining of the institution will outweigh the benefits

2) Don't turn him in because [some non-consequentialist reason on non-consequentialist grounds]

3) Don't turn him in because you will have rationally/consequentialistly tied yourself to the mast making it impossible to turn him in to achieve greater benefits.

(1) and (3) are classic pieces of consequentialism, the first dating back at least to Mill. If your reason is like those, then you are probably a consequentialist and there is no need to reinvent the wheel: I can provide some references for you. If you support (2), perhaps on some kind of Newcomb's problem grounds, then this deserves a clear explanation. Why, on account of a tricky paradoxical situation that may not even be possible, will you predictably start choosing to make things worse in situations that are not Newcomb situations? Unless you are explicit about your beliefs, we can't help debug them effectively, and you then can't hold them with confidence for they won't have undergone peer scrutiny. [The same still goes for your meta-ethical claims].

Toby, my actual stance on the core issue is that it is a Newcomblike problem. You observe the seal of the confessional for the same reason that you one-box on Newcomb's Problem, cooperate in the oneshot Prisoner's Dilemma, or keep your word as Parfit's Hitchhiker: namely, to win.

And if we were talking about superintelligences dealing with other superintelligences, this would be the whole of the law.

It's not easy to transport Newcomblike problems to humans - who cannot make rigorous inferences about each other's probable initial conditions, cannot make rigorous deductions about decisions given the initial condition, and who can only guess at the degree of similarity of decision processes.

But it's by no means obvious that a human should two-box on Newcomb's Problem - it seems that people's choices on Newcomb's Problem do correlate to other facets of their personality, which means that one-boxers against a human Omega might still do better on average. It's by no means clear that humans should go around defecting in the Prisoner's Dilemma, because for us, such situations are often iterated. Our PDs are rarely True PDs where you really don't care at all about the other person. It's by no means clear that humans should believe themselves obligated to break their word to Parfit's Hitchhiker, because we are not perfect liars.

If that lacks the crispness of, for example, the rule that you should not adopt mysterious answers to mysterious questions - well, not every question that I consider has a nice, crisp, massively supported answer. Some of them do. Those are nice. And I even prefer to write about questions that are clear to me, than areas where the borders are fuzzy. But I felt that I had to write about ethics anyway - all things considered.

Eliezer,

So you have a form of deontological ethics based on Newcomb's problem? Now that is very unusual. I can't see how that could be plausible, but hope that you will surprise me. Obviously it is something important enough for a post (or many), so I won't ask you to elaborate any further in the comments.

Googling Parfit and hitchhiker returns some fans of both Derek and Guide to the Galaxy, and a few academic papers behind a paywall. Is there a summary of his example online somewhere?

Since no one so far seems to have mentioned it, there was in fact a Hollywood-style film made (albeit in the UK, not Hollywood), with a mixture of British and American stars, based on the Haukelid/Norsk Hydro story, called "The Heroes of Telemark". Despite being an war/action movie, it actually somehow managed to understate the historical reality.

http://www.imdb.com/title/tt0059263/

Keith, Eliezer: from what I remember of Catholic doctrine (I grew up one), breaking the seal of confession is a lesser sin than murder - as murder is a mortal sin. You go straight to hell for that one, no passing go - Especially as Jesus specifically said 'do not kill' is one of the strongest commandments - but breaking the seal, IIRC, is 'just' de-frocking and excommunication (which may or may not condemn you to hell), which are only undoable by the Pope.

However, mortal sins can be forgiven, and I recall that self-defense lessens the gravity of the offense. So given the hypothetical case of a sinner who is going to kill the priest, I think the thing to do would be to kill the sinner; but in the case of the sinner killing a bunch of other people & specifically excepting the priest (so he can't claim self-defense as in the first case), that's harder. I suppose it comes down to whether you think you can convince the Pope that you were justified in breaking the seal.

TGGP: page 85.

TGGP, regarding Parfit's Hitchhiker:

As near as I can tell, it's from his 1984 book "Reasons and Persons." (http://en.wikipedia.org/wiki/Reasons_and_Persons) The hitchhiker is, indeed, living in Douglas Adams' universe, where teleportation "is as nice and instinctive as a kick in the head." I believe he's using teleportation as a metaphor for the thread of personal identity between T1 and T2, and the moral obligation person at T1 has for person at T2. A quote from the wiki goes:

"Part 3 argues for a reductive account of personal identity; rather than accepting the claim that our existence is a deep, significant fact about the world, Parfit's account of personal identity is like this:

At time 1, there is a person. At a later time 2, there is a person. These people seem to be the same person. Indeed, these people share memories and personality traits. But there are no further facts in the world that make them the same person."

I'm including this here so nobody else has to spend an hour searching down broken google links. I might be a little bit off - I had to make a lot of inferences, so don't take it as gospel - and if anyone knows better, please correct my errors.

:)

If ethics must be held to in the face of the annihilation of everything, then I will proudly state that I have no ethics, only value judgments. Would I kill babies? Yes, to save the life of the mother. Would I kill innocents who had helped me? Yes, to save more. On an interesting aside, I would not torture an innocent for 40 years to prevent 3^^^^3 people from getting a speck of dust in their eyes assuming no further consequences from any of that dust. I would not walk away from Omelas, I would stay to tear it down.

I think this this is the quote you were looking for.

(Obligatory)

‘No, Lady,’ [Sam] answered. ‘To tell the truth, I wondered what you were talking about. I saw a star through your finger. But if you’ll pardon my speaking out, I think my master was right. I wish you’d take his Ring. You’d put things to rights. You’d stop them digging up the gaffer and turning him adrift. You’d make some folk pay for their dirty work.’

‘I would,’ she said. ‘That is how it would begin. But it would not stop with that, alas! We will not speak more of it. Let us go!’

I was thinking about this a few months ago, and since people have multiple "one shot" instances of the Prisoner's Dilemma within their lifetimes it might make sense for general rules about "one shot" instances to arise. This sort of interacts with Hofstadter's ideas about superrationality, too. I don't remember the thought very well, but hopefully this comment sort of gets the idea across.

Can you imagine a Hollywood movie in which the hero did that, instead of coming up with some amazing clever way to save the civilians on the ship?

Jack Bauer might do it.