I vote in favor of banning Caledonian. He isn't just dissenting, which many commenters do often enough. He isn't even trying to be right, he's just trying to say Eliezer is wrong.

Eliezer, the money pump results from circular preferences, which should exist according to your description of the inconsistency. Suppose we have a million statements, each of which you believe to be true with equal confidence, one of which is "The LHC will not destroy the earth."

Suppose I am about to pick a random statement from the list of a million, and I will destroy the earth if I happen to pick a false statement. By your own admission, you estimate that there is more than one false statement in the list. You will therefore prefer that I play a lottery with odds of 1 in a million, destroying the earth only if I win.

It makes no difference if I pick a number randomly between one and a million, and then play the lottery mentioned (ignoring the number picked.)

But now if I pick a number randomly between one and a million, and then play the lottery mentioned only if I didn't pick the number 500,000, while if I do pick the number 500,000, I destroy the earth only if the LHC would destroy the earth, then you would prefer this state of affairs, since you prefer "destroy the earth if the LHC would destroy the earth" to "destroy the earth with odds of one in a million."

But now I can also substitute the number 499,999 with some other statement that you hold with equal confidence, so that if I pick 499,999, instead of playing the lottery, I destroy the earth if this statement is false. You will also prefer this state of affairs for the same reason, since you hold this statement with equal confidence to "The LHC will not destroy the earth."

And so on. It follows that you prefer to go back to the original state of affairs, which constitutes circular preferences and implies a money pump.

Eliezer, you are thinking of Utilitarian (also begins with U, which may explain the confusion.) See http://utilitarian-essays.com/pascal.html

I'll get back to the other things later (including the money pump.) Unfortunately I will be busy for a while.

Can't give details, there would be a risk of revealing my identity.

I have come up with a hypothesis to explain the inconsistency. Eliezer's verbal estimate of how many similar claims he can make, while being wrong on average only once, is actually his best estimate of his subjective uncertainty. How he would act in relation to the lottery is his estimate influenced by the overconfidence bias. This is an interesting hypothesis because it would provide a measurement of his overconfidence. For example, which would he stop: The "Destroy the earth if God exists" lottery, or "Destroy the earth at odds of one in a trillion"? How about a quadrillion? A quintillion? A googleplex? One in Graham's number? At some point Eliezer will have to prefer to turn off the God lottery, and comparing this to something like one in a billion, his verbal estimate, would tell us exactly how overconfident he is.

Since the inconsistency would allow Eliezer to become a money-pump, Eliezer has to admit that some irrationality must be responsible for it. I assign at least a 1% chance to the possibility that the above hypothesis is true. Given even such a chance, and given Eliezer's work, he should come up with methods to test the hypothesis, and if it is confirmed, he should change his way of acting in order to conform with his actual best estimate of reality, rather than his overconfident estimate of reality.

Unfortunately, if the hypothesis is true, by that very fact, Eliezer is unlikely to take these steps. Determining why can be left as an exercise to the reader.

Recently I did some probability calculations, starting with "made-up" numbers, and updating using Bayes' Rule, and the result was that something would likely happen which my gut said most firmly would absolutely not, never, ever, happen.

I told myself that my probability assignments must have been way off, or I must have made an error somewhere. After all, my gut couldn't possibly be so mistaken.

The thing happened, by the way.

This is one reason why I agree with RI, and disagree with Eliezer.

I've touched things a few thousand years old. But I think I get more psychological effect from just looking at a bird, for example, and thinking of its ancestors flying around in the time of the dinosaurs.

I've mentioned in the past that human brains evaluate moral propositions as "true" and "false" in the same way as other propositions.

It's true that it there are possible minds that do not do this. But the first AI will be programmed by human beings who are imitating their own minds. So it is very likely that this AI will evaluate moral propositions in the same way that human minds do, namely as true or false. Otherwise it would be very difficult for human beings to engage this AI in conversation, and one of the goals of the programmers would be to ensure that it could converse.

This is why, as I've said before, that programming an AI does not require an understanding of morality, it just requires enough knowledge to program general intelligence. And this is what is going to actually happen, in all probability; the odds that Eliezer's AI will be the very first AI are probably less than 1 in a 1000, given the number of people trying.

Poke, in the two sentences:

"You should open the door before attempting to walk through it."

"You should not murder."

The word "should" means EXACTLY the same thing. And since you can understand the first claim, you can understand the second as well.

Mike Blume: "Intelligence is a product of structure, and structure comes from an ordering of lower levels."

I agree with that (at least for the kind of intelligence we know about), but the structure rests on universal laws of physics: how did those laws get to be universal?

We might be living in a simulation. If we are, then as Eliezer pointed out himself, we have no idea what kind of physics exist in the "real world." In fact, there is no reason to assume any likeness at all between our world and the real world. For example, the fundamental entities in the real world could be intelligent beings, instead of quarks. If so, then there could be some "shadowy figure" after all. This might be passing the buck, but at least it would be passing it back to somewhere where we can't say anything about it anymore.