Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Eliezer_Yudkowsky comments on Pascal's Mugging: Tiny Probabilities of Vast Utilities - Less Wrong

39 Post author: Eliezer_Yudkowsky 19 October 2007 11:37PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (335)

Sort By: Old

You are viewing a single comment's thread.

Comment author: Eliezer_Yudkowsky 22 October 2007 09:17:00PM 6 points [-]

It might be more promising to assume that states with many people hurt have a low correlation with what any random person claims to be able to effect.

Robin: Great point about states with many people having low correlations with what one random person can effect. This is fairly trivially provable.

Aha!

For some reason, that didn't click in my mind when Robin said it, but it clicked when Vassar said it. Maybe it was because Robin specified "many people hurt" rather than "many people", or because Vassar's part about being "provable" caused me to actually look for a reason. When I read Robin's statement, it came through as just "Arbitrarily penalize probabilities for a lot of people getting hurt."

But, yes, if you've got 3^^^^3 people running around they can't all have sole control over each other's existence. So in a scenario where lots and lots of people exist, one has to penalize by a proportional factor the probability that any one person's binary decision can solely control the whole bunch.

Even if the Matrix-claimant says that the 3^^^^3 minds created will be unlike you, with information that tells them they're powerless, if you're in a generalized scenario where anyone has and uses that kind of power, the vast majority of mind-instantiations are in leaves rather than roots.

This seems to me to go right to the root of the problem, not a full-fledged formal answer but it feels right as a starting point. Any objections?

Comment author: taryneast 31 December 2010 05:16:50PM -1 points [-]

This seems intuitively plausible.

The more outrageous the claim, the correspondingly less plausible is their ability to pull it off.

Especially when you evaluate the amount of resources they are demanding vs the number of resources that you would expect their implausibly difficult plan would require to be achieved.

Comment author: Will_Sawin 31 December 2010 08:29:26PM 1 point [-]

That's not the point. None of those probabilities are as strong as 3^^^3. Maybe big, buy not THAT big.

The point is that no more than 1/3^^^3 people have sole control over the life or death of 3^^3 people. This improbability, that you would be one of those very special people, IS big enough.

(This answer fails unless your ethics and anthropics use the same measure. That's how the pig example works.)

Comment author: wedrifid 01 January 2011 01:18:01AM *  0 points [-]

Even if the Matrix-claimant says that the 3^^^^3 minds created will be unlike you, with information that tells them they're powerless, if you're in a generalized scenario where anyone has and uses that kind of power, the vast majority of mind-instantiations are in leaves rather than roots.

The point is that no more than 1/3^^^3 people have sole control

I was about to express mild amusement about how cavalier we are with jumping to, from and between numbers like 3^^^^3 and 3^^^3. I had to squint to tell the difference. Then it occurred to me that:

The point is that no more than 1/3^^^3 people have sole control over the life or death of 3^^3 people. This improbability, that you would be one of those very special people, IS big enough.

3^^3 is not even unimaginably big, Knuth arrows or no. It's about 1/5th the number of people that can fit in the MCG.

Comment author: Will_Sawin 01 January 2011 06:42:07AM 1 point [-]

Being cavalier with proofreading =/= being cavalier with number size.

But that is indeed amusing.

Comment author: wedrifid 01 January 2011 07:07:29AM 0 points [-]

Being cavalier with proofreading =/= being cavalier with number size.

Well, I didn't want to declare a proofreading error because 3^^^3 does technically fit correctly in the context, even if you may not have meant it. ;)

I was thinking the fact that we are so cavalier makes it easier to slip between them if not paying close attention. Especially since 3^^^3 is more commonly used than 3^^^^3. I don't actually recall Eliezer going beyond pentation elsewhere.

I know if I go that high I tend to use 4^^^^4. It appeals more aesthetically and is more clearly distinct. Mind you it isn't nearly as neat as 3^^^3 given that 3^^^3 can also be written and visualized conceptually as 3 -> 3 -> 3 while 4^^^^4 is just 4 -> 4 -> 4 not 4 -> 4 -> 4 -> 4.

Comment author: taryneast 01 January 2011 09:39:49PM 0 points [-]

So you're saying that the implausibility is that I'd run into a person that just happened to have that level of "power" ?

Is that different in kind to what I was saying?

If I find it implausible that the person I'm speaking to can actually do what they're claiming, is that not the same as it being implausible that I happen to have met a person that can do what this person is claiming/ (leaving aside the resource-question which is probably just my rationalisation as to why I think he couldn't pull it off).

Basically I'm trying to taboo the actual BigNum... and trying to fit the concepts around in my head.

Comment author: Will_Sawin 02 January 2011 12:12:02AM 0 points [-]

It's implausible that you're the person with that power. We could easily imagine a world in which everyone runs into a single absurdly powerful person. We could not imagine a world in which everyone was absurdly powerful (in their ability to control other people), because then multiple people would have control over the same thing.

If you knew that he had the power, but that his action wasn't going to depend on yours, then you wouldn't give him the money. So you're only concerned with the situation where you have the power.

Comment author: taryneast 02 January 2011 10:06:40AM 1 point [-]

Ok, sure thing. I get what you're saying. I managed to encompass that implausibility also into the arguments I made in my restatement anyway, but yeah, I agree that these are different kinds of "unlikely thing"

Comment author: taryneast 01 January 2011 10:38:10PM *  -1 points [-]

In fact... let me restate what I think I was trying to say.

The mugger is making an extraordinary claim. One for which he has provided no evidence.

The amount of evidence required to make me believe that his claim is possible, grows at the same proportion as the size of his claim.

Think about it at the lower levels of potential claims.

1) If he claimed to be able to kill one person - I'd believe that he was capable of killing one person. I'd then weigh that against the likelihood that he'd pick me to blackmail, and the low blackmail amount that he'd picked... and consider it more likely that he's lying to make a fast buck, than that he actually has a hostage somewhere ready to kill.

2) If he claimed to be able to kill 3^3 people, I'd consider it plausible... with a greatly diminished likelihood. I'd have to weigh the evidence that he was a small-time terrorist, willing to take the strong risk of being caught while preparing to blow up a buildings-worth of people... or to value his life so low as to actually do it and die in the process. It's not very high, but we've all seen people like this in our lifetime both exist and carry out this threat. So it's "plausible but extremely unlikely".

The likelihood that I've: a) happened to run into one of these rare people and b) that he'd pick me (pretty much a nobody) to blackmail combine to be extremely unlikely... and I'd reckon that those two, balanced against the much higher prior likelihood that he's just a con-artist, would fairly well cancel out against the actual value of a buildings-worth of people.

Especially when you consider that the resources to do this would far outweigh the money he's asked for. As far as I know about people wiling to kill large numbers of people - most of them do it for a reason, and that reason is almost never a paltry amount of cash. It's still possible... after all the school-killers have done crazy stunts to kill people for a tiny reason... but usually there's fame or revenge involved... not blackmail of a nobody.

3) So now we move to 3^^3 people. Now, I personally have never seen that many die in one sitting (or even as the result of a single person)... but my Grandfather did, and using technology from 65 years ago.

It is plausible, though even less likely than before, that the person I've just run into happens to be willing and able to use a nuke on a large city, or to have the leadership capabilities (and luck) required to take over a country and divert it's resources to killing that number of people.

I would consider it exponentially less likely that he'd pick me to blackmail about this... and certainly not for such a pitiful amount of cash. People that threaten this kind of thing are either after phenomenal amounts of money, recognition or some kind of political or religious statement.... they are extremely unlikely to find a random citizen to blackmail for a tiny amount of cash. The likelihood that this is a con seems about as high as the number of people to potentially die.

4) Now we hit the first real BigNum. AFAIK, the world has never seen 3^^^3 sentient intelligences ever die in one sitting. We don't have that many people on the Earth right now. Maybe the universe has seen it somewhere... some planetary system wiped out in a supernova. It's plausible... but now think of the claims the guy is making:

a) that he can create (or knows of) a civilisation that contains that number of sentient beings.

b) that he (and he alone) has the ability to destroy that civilisation, and can do so at whim and that

c) it's worthwhile him doing so for the mere pittance he's demanding from a complete, unrelated nobody... (or potentially the whim of watching said nobody squirm).

I actually think that the required (and missing) evidence for his outrageous claims stack fairly evenly against the potential downside of his claims actually being true.

So, to get back to the original point: In my mind, as each step grows exponentially more extreme, so does the evidence required to support such a ludicrous claim. These two cancel out roughly evenly, leaving the leftovers of "is he likely to have picked me?" and other smaller probabilities to actually sway the balance.

Those, added with the large disutility of "encouraging the guy to do it again" would sway me to choose not to give him £5, but to walk away, then immediately find the nearest police officer...

Comment author: Will_Sawin 02 January 2011 12:16:50AM -1 points [-]

3+3, 3*3, 3^3, 3^^3, 3^^^3, etc. grows much faster than exponentially. a^b, for any halfway reasonable a and b, can't touch 3^^^3

3^^^3=3^^(3^^3)=3^^(7625597484987)=3^(3^^(7625597484986))

It's not an exponential, it's a huge, huge tower of exponentials. It is simply too big for that argument to work.

Comment author: taryneast 02 January 2011 09:50:41AM *  -1 points [-]

Yes, I should not have used the word exponential... but I don't know the word for "grows at a rate that is a tower of exponentials"... "hyperexponential" perhaps?

However - I consider that my argument still holds. That the evidence required grows at the same rate as the size of the claim.

The evidence must be of equal value to the claim.

(from "extraordinary claims require extraordinary evidence")

My point in explaining the lower levels is that is that we don't demand evidence from most claimants of small amounts of damage because we've already seen evidence that these threats are plausible. But if we start getting to the "hyperexponential" threats, we hit a point where we suddenly realise that there is no evidence supporting the plausibility of the claim... so we automatically assume that the person is a crank.

Comment author: Vaniver 02 January 2011 12:35:07AM -1 points [-]

3) So now we move to 3^^3 people. Now, I personally have never seen that many die in one sitting (or even as the result of a single person)... but my Grandfather did, and using technology from 65 years ago.

3^^3 is a thousand times larger than the number of people currently alive.

Comment author: taryneast 02 January 2011 09:43:42AM -1 points [-]

oops, yes I mixed up 3^^3 with 3^^^3

Ok, so skip step 3 and move straight on to 4 ;)

Comment author: steven0461 02 August 2011 07:27:18AM *  0 points [-]

The point is that no more than 1/3^^^3 people have sole control over the life or death of 3^^3 people. This improbability, that you would be one of those very special people, IS big enough.

(This answer fails unless your ethics and anthropics use the same measure. That's how the pig example works.)

So can we solve the problem by putting some sort of upper bound on the degree to which ethics and anthropics can differ, along the lines of "creation of 3^^^^3 people is at most N times less probable than creation of 3^^^^3 pigs, so across the ensemble of possible worlds the prior against your being in a position to influence that many pigs still cuts down the expected utility from something vaguely like 3^^^^3 to something vaguely like N"?

Comment author: Ben123 07 June 2012 05:52:41PM 1 point [-]

Is that a general solution? What about this: "Give me five dollars or I will perform an action, the disutility of which will be equal to twice that of you giving me five dollars, multiplied by the reciprocal of the probability of this statement being true."

Comment author: [deleted] 07 June 2012 06:54:22PM 1 point [-]

Well, I'd rather lose twenty dollars than be kicked in the groin very hard, and the probability of you succeeding in doing that given you being close enough to me and trying to do so is greater than 1/2, so...

Comment author: MugaSofer 18 September 2012 11:14:09AM *  0 points [-]

But anthropically, since you exist within the matrix, and so does he, and hostages outside the matrix cannot reach you to make such an offer ...

You don't have 3^^^^3 people "running around". You have the population of earth running around, plus one matrix lord.

More to the point, if you create 3^^^^3 people, surely a LOT of them are going to be identical, purely by coincidence? Aren't you double-counting most of them?