# Pascal's Mugging: Tiny Probabilities of Vast Utilities

The most common formalizations of Occam's Razor, Solomonoff induction and Minimum Description Length, measure the program size of a computation used in a hypothesis, but don't measure the running time or space requirements of the computation. What if this makes a mind vulnerable to finite forms of Pascal's Wager? A compactly specified wager can grow in size *much* faster than it grows in complexity. The utility of a Turing machine can grow much faster than its prior probability shrinks.

Consider Knuth's up-arrow notation:

- 3^3 = 3*3*3 = 27
- 3^^3 = (3^(3^3)) = 3^27 = 3*3*3*3*3*3*3*3*3*3*3*3*3*3*3*3*3*3*3*3*3*3*3*3*3*3*3 = 7625597484987
- 3^^^3 = (3^^(3^^3)) = 3^^7625597484987 = 3^(3^(3^(... 7625597484987 times ...)))

In other words: 3^^^3 describes an exponential tower of threes 7625597484987 layers tall. Since this number can be computed by a simple Turing machine, it contains very little information and requires a very short message to describe. This, even though writing out 3^^^3 in base 10 would require *enormously* more writing material than there are atoms in the known universe (a paltry 10^80).

Now suppose someone comes to me and says, "Give me five dollars, or I'll use my magic powers from outside the Matrix to run a Turing machine that simulates and kills 3^^^^3 people."

Call this Pascal's Mugging.

"Magic powers from outside the Matrix" are easier said than done - we have to suppose that our world is a computing simulation run from within an environment that can afford simulation of arbitrarily large finite Turing machines, and that the would-be wizard has been spliced into our own Turing tape and is in continuing communication with an outside operator, etc.

Thus the Kolmogorov complexity of "magic powers from outside the Matrix" is larger than the mere English words would indicate. Therefore the Solomonoff-inducted probability, two to the *negative* Kolmogorov complexity, is exponentially tinier than one might naively think.

But, small as this probability is, it isn't anywhere *near* as small as 3^^^^3 is large. If you take a decimal point, followed by a number of zeros equal to the length of the Bible, followed by a 1, and multiply this unimaginably tiny fraction by 3^^^^3, the result is pretty much 3^^^^3.

Most people, I think, envision an "infinite" God that is nowhere near as large as 3^^^^3. "Infinity" is reassuringly featureless and blank. "Eternal life in Heaven" is nowhere near as intimidating as the thought of spending 3^^^^3 years on one of those fluffy clouds. The notion that the diversity of life on Earth springs from God's infinite creativity, sounds more plausible than the notion that life on Earth was created by a superintelligence 3^^^^3 bits large. Similarly for envisioning an "infinite" God interested in whether women wear men's clothing, versus a superintelligence of 3^^^^3 bits, etc.

The original version of Pascal's Wager is easily dealt with by the gigantic multiplicity of possible gods, an Allah for every Christ and a Zeus for every Allah, including the "Professor God" who places only atheists in Heaven. And since all the expected utilities here are allegedly "infinite", it's easy enough to argue that they cancel out. Infinities, being featureless and blank, are all the same size.

But suppose I built an AI which worked by some bounded analogue of Solomonoff induction - an AI sufficiently Bayesian to insist on calculating complexities and assessing probabilities, rather than just waving them off as "large" or "small".

If the probabilities of various scenarios considered did not *exactly* cancel out, the AI's action in the case of Pascal's Mugging would be *overwhelmingly* dominated by whatever tiny differentials existed in the various tiny probabilities under which 3^^^^3 units of expected utility were actually at stake.

You or I would probably wave off the whole matter with a laugh, planning according to the dominant mainline probability: Pascal's Mugger is just a philosopher out for a fast buck.

But a silicon chip does not look over the code fed to it, assess it for reasonableness, and correct it if not. An AI is not given its code like a human servant given instructions. An AI *is* its code. What if a philosopher tries Pascal's Mugging on the AI for a joke, and the tiny probabilities of 3^^^^3 lives being at stake, override *everything* else in the AI's calculations? What is the mere Earth at stake, compared to a tiny probability of 3^^^^3 lives?

How do *I* know to be worried by this line of reasoning? How do *I* know to rationalize reasons a Bayesian shouldn't work that way? A mind that worked strictly by Solomonoff induction would not know to rationalize reasons that Pascal's Mugging mattered less than Earth's existence. It would simply go by whatever answer Solomonoff induction obtained.

It would seem, then, that I've implicitly declared my existence as a mind that does not work by the logic of Solomonoff, at least not the way I've described it. What am I comparing Solomonoff's answer to, to determine whether Solomonoff induction got it "right" or "wrong"?

Why do I think it's unreasonable to focus my entire attention on the magic-bearing possible worlds, faced with a Pascal's Mugging? Do I have an instinct to resist exploitation by arguments "anyone could make"? Am I unsatisfied by any visualization in which the dominant mainline probability leads to a loss? Do I drop sufficiently small probabilities from consideration entirely? Would an AI that lacks these instincts be exploitable by Pascal's Mugging?

Is it me who's wrong? Should I worry more about the possibility of some Unseen Magical Prankster of very tiny probability taking this post literally, than about the fate of the human species in the "mainline" probabilities?

It doesn't feel to me like 3^^^^3 lives are *really* at stake, even at very tiny probability. I'd sooner question my grasp of "rationality" than give five dollars to a Pascal's Mugger because I thought it was "rational".

Should we penalize computations with large space and time requirements? This is a hack that solves the problem, but is it *true?* Are computationally costly explanations less likely? Should I think the universe is probably a coarse-grained simulation of my mind rather than real quantum physics, because a coarse-grained human mind is *exponentially *cheaper than real quantum physics? Should I think the galaxies are tiny lights on a painted backdrop, because that Turing machine would require less space to compute?

Given that, in general, a Turing machine can increase in utility vastly faster than it increases in complexity, how should an Occam-abiding mind avoid being dominated by tiny probabilities of vast utilities?

If I could formalize whichever internal criterion was telling me I didn't want this to happen, I might have an answer.

I talked over a variant of this problem with Nick Hay, Peter de Blanc, and Marcello Herreshoff in summer of 2006. I don't feel I have a satisfactory resolution as yet, so I'm throwing it open to any analytic philosophers who might happen to read Overcoming Bias.

## Comments (334)

OldWhy would not giving him $5 make it more likely that people would die, as opposed to less likely? The two would seem to cancel out. It's the same old "what if we are living in a simulation?" argument- it is, at least, possible that me hitting the sequence of letters "QWERTYUIOP" leads to a near-infinity of death and suffering in the "real world", due to AGI overlords with wacky programming. Yet I do not refrain from hitting those letters, because there's no entanglement which drives the probabilities in that direction as opposed to some other random direction; my actions do not alter the expected future state of the universe. You could just as easily wind up saving lives as killing people.

*13 points [-]Because he said so, and people tend to be true to their word more often than dictated by chance.

That observation applies to humans, who also tend not to kill large numbers of people for no payoff (that is, if you've already refused the money and walked away).

That's a symmetric effect, though.

Yes, but they're more likely to kill large numbers of people conditional on you not doing what they say than conditional on you doing what they say.

The mugger claims to not be a 'person' in the conventional sense, but rather an entity with outside-Matrix powers. If this statement is true, then generalized observations about the reference class of 'people' cannot necessarily be considered applicable.

Conversely, if it is false, then this is not a randomly-selected person, but rather someone who has started off the conversation with an outrageous profit-motivated lie, and as such cannot be trusted.

They claim to not be a human. They're still a person, in the sense of a sapient being. As a larger class, you'd expect lower correlation, but it would still be above zero.

I am not convinced that, even among humans speaking to other humans, truth-telling can be assumed when there is such a blatantly obvious incentive to lie.

I mean, say there actually

issomeone who can destroy vast but currently-unobservable populations with less effort than it would take them to earn $5 with conventional economic activity, and the ethical calculus works out such that you'd be better served to pay them $5 than let it happen. At that point, aren't they better served to exaggerate their destructive capacity by an order of magnitude or two, and ask you for $6? Or $10?Once the number the mugger quotes exceeds your ability to independently confirm, or even properly imagine, the number itself becomes irrelevant. It's either a display of incomprehensibly overwhelming force, to which you must submit utterly or be destroyed, or a bluff you should ignore.

There is no blatantly obvious reason to want to torture the people only if you do give him money.

So, you're saying that the problem is that, if they really were going to kill 3^^^3 people, they'd lie? Why? 3^^^3 isn't just enough to get $5. It's enough that the expected seriousness of the threat is unimaginably large.

Look at it this way: If they're going to lie, there's no reason to exaggerate their destructive capacity by an order of magnitude when they can just make up a number. If they choose to make up a number, 3^^^3 is plenty high. As such, if it really is 3^^^3, they might as well just tell the truth. If there's any chance that they're not lying given that they really can kill 3^^^3 people, their threat is valid. It's one thing to be 99.9% sure they're lying, but here, a 1 - 1/sqrt(3^^^3) certainty that they're lying still gives more than enough doubt for an unimaginably large threat.

You're not psychic. You don't know which it is. In this case, the risk of the former is enough to overwhelm the larger probability of the latter.

Not the way I do the math.

Let's say you're a sociopath, that is, the only factors in your utility function are your own personal security and happiness. Two unrelated people approach you simultaneously, one carrying a homemade single-shot small-caliber pistol (a 'zip gun') and the other apparently unarmed. Both of them, separately, demand $10 in exchange for not killing you immediately. You've got a $20 bill in your wallet; the unarmed mugger, upon learning this, obligingly offers to make change. While he's thus distracted, you propose to the mugger with the zip gun that he shoot the unarmed mugger, and that the two of you then split the proceeds. The mugger with the zipgun refuses, explaining that the unarmed mugger claims to be close personal friends with a professional sniper, who is most likely observing this situation from a few hundred yards away through a telescopic sight and would retaliate against anyone who hurt her friend the mugger. The mugger with the zip gun has never actually met the sniper or directly observed her handiwork, but is sufficiently detered by rumor alone.

If you don't pay the zip-gun mugger, you'll definitely get shot at, but only once, and with good chances of a miss or nonfatal injury. If you don't pay the unarmed mugger, and the sniper is real, you will almost certainly die before you can determine her position or get behind sufficiently hard cover. If you pay them both, you will have to walk home through a bad part of town at night instead of taking the quicker-and-safer bus, which apart from the inconvenience might result in you being mugged a third time.

How would you respond to that?

I don't need to be psychic. I just do the math. Taking any sort of infinitessimally-unlikely threat so seriously that it dominates my decisionmaking means anyone can yank my chain just by making a few unfounded assertions involving big enough numbers, and then once word gets around, the world will no longer contain acceptable outcomes.

In your example, only you die. In Pascal's mugging, it's unimaginably worse.

Do you accept that, in the circumstance you gave, you are more likely to be shot by a sniper if you only pay one mugger? Not significantly more likely, but still more likely? If so, that's analogous to accepting that Pascal's mugger will be more likely to make good on his threat if you don't pay.

In my example, the person making the decision was specified to be a sociopath, for whom there is no conceivable worse outcome than the total loss of personal identity and agency associated with death.

The two muggers are indifferent to each other's success. You could pay off the unarmed mugger to eliminate the risk of being sniped (by that particular mugger's friend, at least, if she exists; there may well be other snipers elsewhere in town with unrelated agendas, about whom you have even less information) and accept the risk of being shot with the zip gun, in order to afford the quicker, safer bus ride home. In that case you would only be paying one mugger, and still have the lowest possible sniper-related risk.

The three possible expenses were meant as metaphors for existential risk mitigation (imaginary sniper), infrastructure development (bus), and military/security development (zip gun), the latter two forming the classic guns-or-butter economic dilemma. Historically speaking, societies that put too much emphasis, too many resources, toward preventing low-probability high-impact disasters, such as divine wrath, ended up succumbing to comparatively banal things like famine, or pillaging by shorter-sighted neighbors. What use is a mathematical model of utility that would steer us into those same mistakes?

Is your problem that we'd have to keep the five dollars in case of another mugger? I'd hardly consider the idea of steering our life around pascal's mugging to be disagreeing with it. For what it's worth, if you look for hypothetical pascal's muggings, expected utility doesn't converge and decision theory breaks down.

excellent point, sir.

*0 points [-][comment deleted]

Very interesting thought experiment!

One place where it might fall down is that our disutility for causing deaths is probably not linear in the number of deaths, just as our utility for money flattens out as the amount gets large. In fact, I could imagine that its value is connected to our ability to intuitively grasp the numbers involved. The disutility might flatten out *really quickly* so that the disutility of causing the death of 3^^^^3 people, while large, is still small enough that the small probabilities from the induction are not overwhelmed by it.

*1 point [-]That just means you have to change the experiment. Suppose he just said he'll cause a certain amount of net disutility, without specifying how.

This works unless you assume a maximum possible disutility.

You are not entitled to assume a maximum disutility, even if you think you see a proof for it (see Confidence Levels Inside and Outside an Argument).

People say the fact that there are many gods neutralizes Pascal’s wager - but I don't understand that at all. It seems to be a total non sequetor. Sure, it opens the door to other wagers being valid, but that is a different issue.

Lets say I have a simple game against you where, if I choose 1 I win a lotto ticket and if I choose 0 I loose. There is also a number of other games tables around the room with people winning or not winning lotto tickets. If I want to win the lotto, what number should I pick?

Also I don't tink there is a fundimental issue with having favour with Allah, Christ and Zeus simultaniously. (so you could actualy win, then get up and go play at another table - although there would be a time cost to that).

Now there is the more detailed argument where you argue that a god who desired you disbelieve in him and oppose his will is equally likely to one that desires that you believe in him and supports his will. But as long as there is any imperfection in the mirror then there is a Pascal’s wager to be had.

> What if a philosopher tries Pascal's Mugging on the AI for a joke, and the tiny probabilities of 3^^^^3 lives being at stake, override everything else in the AI's calculations?

Suppose that depends on how he calculates the probability of the threat of the mugger. The very act of giving a specific probability to a threat like that opens one up to an infinite risk (i.e. that they will demand infinite things in exchange for infinity x 3^^^^3 lives). So this is a bit like comparing what I might call naive utilitarianism (where one doesn’t consider the wider effects of one’s acts and rules) with pure utilitarianism (where one takes everything into account).

Whether that neutralizes Pascal’s wager relates to how one resolves the mirror issue I mentioned. If that produces a tidy result then the problem above doesn’t occur.

*1 point [-]There is one problem with having favor of several gods simultaneously:

In fact, one could argue that being a true orthodox christian would lead you to the muslim, hindu, protestant and scientology (etc.) hells, while choosing anyone of them would subtract that hell but add the hell of whatever religion you left...

I try to stay away for safety's sake :)

[edit: spelling]

This is an instance of the general problem of attaching a probability to matrix scenarios. And you can pascal-mug yourself, without anyone showing up to assert or demand anything - just think:

what ifthings are set up so that whether I do, or do not do,something, determines whether those 3^^^^3 people will be created and destroyed? It's just as possible as the situation in which a messenger from Outside shows up and tells you so.The obvious way to attach probabilities to matrix scenarios is to have a unified notion of possible world capacious enough to encompass both matrix worlds and worlds in which your current experiences are veridical; and then you look at relative frequencies or portions of world-measure for the two classes of possibility. For example, you could assume the correctness of our current physics across all possible worlds, and then make a Drake/Bostrom-like guesstimate of the frequency of matrix construction across all those universes, and of the "demography" and "political character" of those simulations. Garbage in, garbage out; but you really can get an answer if you make enough assumptions. In that regard, it is not too different to any other complicated decision made against a background of profound uncertainty.

Tom and Andrew, it seems very implausible that someone saying "I will kill 3^^^^3 people unless X" is literally

zeroBayesian evidence that they will kill 3^^^^3 people unless X. Though I guess it could plausibly be weak enough to take much of the force out of the problem.Andrew, if we're in a simulation, the world containing the simulation could be able to support 3^^^^3 people. If you knew (magically) that it couldn't, you could substitute something on the order of 10^50, which is vastly less forceful but may still lead to the same problem.

Andrew and Steve, you could replace "kill 3^^^^3 people" with "create 3^^^^3 units of disutility according to your utility function". (I respectfully suggest that we all start using this form of the problem.)

Michael Vassar has suggested that we should consider any number of identical lives to have the same utility as one life. That could be a solution, as it's impossible to create 3^^^^3 distinct humans. But, this also is irrelevant to the create-3^^^^3-disutility-units form.

IIRC, Peter de Blanc told me that any consistent utility function must have an upper bound (meaning that we must discount lives like Steve suggests). The problem disappears if your upper bound is low enough. Hopefully any realistic utility function has such a low upper bound, but it'd still be a good idea to solve the general problem.

I see a similarity to the police chief example. Adopting a policy of paying attention to any Pascalian muggings would encourage others to manipulate you using them. At first it doesn't seem like this would have nearly enough disutility to justify ignoring muggings, but it might when you consider that it would interfere with responding to any

realthreat (unlikely as it is) of 3^^^^3 deaths.For all X:

If your utility function assigns values to outcomes that differ by a factor of X, then you are vulnerable to becoming a fanatic who banks on scenarios that only occur with probability 1/X. As simple as that.

If you think that banking on scenarios that only occur with probability 1/X is silly, then you have implicitly revealed that your utility function only assigns values in the range [1,Y], where Y<X, and where 1 is the lowest utility you assign.

... or your judgments of silliness are out of line with your utility function.

When I said "Silly" I meant from an axiological point of view, i.e. you think the scenario over, and you still think that you would be doing something that made you win less.

Of course in any such case, there are likely to be conflicting intuitions: one to behave as an aggregative consequentialist, and the another to behave like a sane human being.

Mitchell, it doesn't seem to me like any sort of accurate many-worlds probability calculation would give you a probability anywhere near low enough to cancel out 3^^^^3. Would you disagree? It seems like there's something else going on in our intuitions. (Specifically, our intuitions that an good FAI would need to agree with us on this problem.)

Sorry, the first link was supposed to be to Absence of Evidence is Evidence of Absence.

Mitchell, I don't see how you can Pascal-mug yourself. Tom is right that the possibility that typing QWERTYUIOP will destroy the universe can be safely ignored; there is no evidence either way, so the probability equals the prior, and the Solomonoff prior that typing QWERTYUIOP will save the universe is, as far as we know, exactly the same. But the mugger's threat is a shred of Bayesian evidence that you have to take into account, and when you do, it massively tips the expected utility balance. Your suggested solution does seem right but utterly intractable.

I don't think the QWERTYUIOP thing is literally zero Bayesian evidence either. Suppose the thought of that particular possibility was manually inserted into your mind by the simulation operator.

Tom and Andrew, it seems very implausible that someone saying "I will kill 3^^^^3 people unless X" is literally zero Bayesian evidence that they will kill 3^^^^3 people unless X. Though I guess it could plausibly be weak enough to take much of the force out of the problem.Nothing could possibly be that weak.

Tom is right that the possibility that typing QWERTYUIOP will destroy the universe can be safely ignored; there is no evidence either way, so the probability equals the prior, and the Solomonoff prior that typing QWERTYUIOP will save the universe is, as far as we know, exactly the same.Exactlythe same? These are different scenarios. What happens if an AI actually calculates the prior probabilities, using a Solomonoff technique, without any a priori desire that things should exactly cancel out?Why would an AI consider those two scenarios and no others? Seems more likely it would have to chew over every equivalently-complex hypothesis before coming to any actionable conclusion... at which point it stops being a worrisome, potentially world-destroying AI and becomes a brick, with a progress bar that won't visibly advance until after the last proton has decayed.

... which doesn't solve the problem, but at least that AI won't be giving anyone... five dollars? Your point is valid, but it doesn't expand on anything.

More generally I mean that an AI capable of succumbing to this particular problem wouldn't be able to function in the real world well enough to cause damage.

I'm not sure that was ever a question. :3

OK, let's try this one more time:

3. Even if you don't accept 1 and 2 above, there's no reason to expect that the person is telling the truth. He might kill the people even if you give him the $5, or conversely he might not kill them even if you don't give him the $5.

To put it another way, conditional on this nonexistent person having these nonexistent powers, why should you be so sure that he's telling the truth? Perhaps you'll only get what you want by _not_ giving him the $5. To put it mathematically, you're computing p*X, where p is the probability and X is the outcome, and you're saying that if X is huge, then just about any nonzero p will make p*X be large. But you're forgetting two things: first, if you have the imagination to imagine X to be super-huge, you should be able to have the imagination to imagine p to be super-small. (I.e., if you can talk about 3^^^^3, you can talk about 1/3^^^^3.) Second, once you allow these hypothetical super-large X's, you have to acknowledge the possibility that you got the sign wrong.

I have to go with Tom McGabe on this one; This is just a restatement of the core problem of epistemology. It's not unique to AI, either.

3. Even if you don't accept 1 and 2 above, there's no reason to expect that the person is telling the truth. He might kill the people even if you give him the $5, or conversely he might not kill them even if you don't give him the $5.But if a Bayesian AI actually calculates these probabilities by assessing their Kolmogorov complexity - or any other technique you like, for that matter - without desiring that they come out exactly equal, can you rely on them coming out exactly equal? If not, an expected utility differential of 2 to the negative googolplex times 3^^^^3 still equals 3^^^^3, so whatever tiny probability differences exist will dominate all calculations based on what we think of as the "real world" (the mainline of probability with no wizards).

if you have the imagination to imagine X to be super-huge, you should be able to have the imagination to imagine p to be super-smallBut we can't just set the probability to anything we like. We have to calculate it, and Kolmogorov complexity, the standard accepted method, will not be anywhere near that super-small.

Addendum: In computational terms, you can't avoid using a 'hack'. Maybe not the hack you described, but something, somewhere has to be hard-coded. How else would you avoid solipsism?

This case seems to suggest the existence of new interesting rationality constraints, which would go into choosing rational probabilities and utilities. It might be worth working out what constraints one would have to impose to make an agent immune to such a mugging.

Eliezer,

OK, one more try. First, you're picking 3^^^^3 out of the air, so I don't see why you can't pick 1/3^^^^3 out of the air also. You're saying that your priors have to come from some rigorous procedure but your utility comes from simply transcribing what some dude says to you. Second, even if for some reason you really want to work with the utility of 3^^^^3, there's no good reason for you not to consider the possibility that it's really -3^^^^3, and so you should be doing the opposite. The issue is not that two huge numbers will exactly cancel out; the point is that you're making up _all_ the numbers here but are artificially constraining the expected utility differential to be positive.

If I really wanted to consider this example realistically, I'd say that this guy has no magic powers, so I wouldn't worry about him killing 3^^^^3 people or whatever. A slightly more realistic scenario would be something like a guy with a bomb in a school, in which case I'd defer to the experts (presumably whoever in the police force deals with people like that) on their judgment of how best to calm him down. There I could see an (approximate) probability calculation being relevant, but, again, they key thing would be whether giving him $5 (or whatever) would make him more or less likely to set the fuse. It wouldn't be appropriate to say a priori that it could only help.

You're not picking 3^^^^3 out of the air. The other guy told you that number.

You can't pick probabilities out of the air. If you could, why not just set the probability that you're God to one?

With what probability? Would you give money to a mugger if their gun probably isn't loaded? Is this example fundamentally different?

pdf23ds, under certain straightforward physical assumptions, 3^^^^3 people wouldn't even fit in anyone's future light-cone, in which case the probability is literally zero. So the assumption that our apparent physics is the physics of the real world too, really could serve to decide this question. The only problem is that that assumption itself is not very reasonable.

Lacking for the moment a rational way to delimit the range of possible worlds, one can utilize what I'll call a Chalmers prior, which simply specifies directly how much time you will spend thinking about matrix scenarios. (I name it for David Chalmers because I once heard him give an estimate of the odds that we are in a matrix; I think it was about 10%.) The rationality of

havinga Chalmers prior can be justified by observing one's own cognitive resource-boundedness, and the apparently endless amount of time one could spend thinking about matrix scenarios. (Is there a name for this sort of scheduling meta-heuristic, in which one limits the processing time available for potentially nonterminating lines of thought?)I'm not aware of any (and I'm not sure it really solves this problem in particular), but there should be, because processing time is absolutely critical to bounded rationality.

Well... I think we act diffrently from the AI because we not only know Pascals Mugging, we know that it is known. I don't see why an AI could not know the knowledge of it, though, but you do not seem to consider that, which might simply show that it is not relevant, as you, er, seem to have given this some thought...

But maybe an AI cannot in fact know the knowledge of something.

What possible reason would you have to assume that? If we're talking about an actually intelligent AI, it'd presumably be as smart as any other intelligent being(like, say, a human). If we're talking about a dumb program, it can take into account anything that we want it to take into account.

Konrad:

In computational terms, you can't avoid using a 'hack'. Maybe not the hack you described, but something, somewhere has to be hard-coded.Well, yes. The alternative to code is not solipsism, but a rock, and even a rock can be viewed as being hard-coded as a rock. But we would prefer that the code be elegant and make sense, rather than using a local patch to fix specific problems as they come to mind, because the latter approach is guaranteed to fail if the AI becomes more powerful than you and refuses to be patched.

Andrew:

You're saying that your priors have to come from some rigorous procedureThe priors have to come from some computable procedure. We would prefer it to be a good one, as agents with nonsense priors will not attain sensible posteriors.

but your utility comes from simply transcribing what some dude says to you.No. Certain hypothetical scenarios, which we describe using the formalism of Turing machines, have fixed utilities - that is, if some description of the universe is true, it has a certain utility.

The problem with this scenario is not that we believe everything the dude tells us. The problem is that the description of a certain very large universe with a very large utility, does not have a correspondingly tiny prior probability if we use Solmonoff's prior. And then as soon as we see any evidence, no matter how tiny, anything whose entanglement is not as tiny as the very large universe is large, that expected utility differential instantly wipes out all other factors in our decision process.

Second, even if for some reason you really want to work with the utility of 3^^^^3, there's no good reason for you not to consider the possibility that it's really -3^^^^3, and so you should be doing the opposite.A Solomonoff inductor might indeed consider it, though there's the problem of any bounded rationalist not being able to consider all computations. It seems "reasonable" for a bounded mind to consider it here; you did, after all.

The issue is not that two huge numbers will exactly cancel out; the point is that you're making up _all_ the numbers here but are artificially constraining the expected utility differential to be positive.Let the differential be negative. Same problem. If the differential is not zero, the AI will exhibit unreasonable behavior. If the AI literally thinks in Solomonoff induction (as I have described), it won't

wantthe differential to be zero, it will just compute it.Well, are you going to give us your answer?

To solve this problem, the AI would need to calculate the probability of the claim being true, for which it would need to calculate the probability of 3^^^^3 people even existing. Given what it knows about the origins and rate of reproduction of humans, wouldn't the probability of 3^^^^3 people even existing be approximately 1/3^^^^3? It's as you said, multiply or divide it by the number of characters in the bible, it's still nearly the same damned incomprehensably large number. Unless you are willing to argue that there are some bizarre properties of the other universe that would allow so many people to spontaneously arise from nothing- but this is yet another explanatory assumption, and one that I see no way of assigning a probability to.

Here's one for you: Lets assume for arguement's sake that "humans" could include human cosciousnesses, not just breathing humans. Then, if a universe with 3^^^^3 "humans" actually existed, what would be the odds that they were NOT all copies of the same parasitic consciousness?

Pascal's wager type arguments fail due to their symmetry (which is preserved in finite cases).

Eliezer Sorry to say (because it makes me sound callous), but if someone can and is willing to create and then destroy 3^^^3 people for less than $5, then there is no value in life, and definitely no moral structure to the universe. The creation and destruction of 3^^^3 people (or more) is probably happening all the time. Therefore the AI is safe declining the wager on purely selfish grounds.

So, if there is someone out there committing grevious holocausts (if we use realistic numbers like "10 million deaths", "20 billion deaths", the probability of this is near 1), then none of us have any moral obligations ever?

Eliezer, I'd like to take a stab at the internal criterion question. One differerence between me and the program you describe is that I have a hoped for future. Say "I'd like to play golf on Wednesday." Now, I could calculate the odds of Wednesday not actually arriving (nuclear war,asteroid impact...), or me not being alive to see it (sudden heartattack...), and I would get an answer greater than zero. Why don't I operate on those non-zero probabilities? (The other difference between me and the program you describe) I think it has to do with faith. That is I have faith that my hoped for future will occur, or at least some semblance of it. I seem to have this faith despite previous losses. Take the field of AI. There is a hoped for future, a computer will demonstrate intelligence, some hope the machine will become conscious. There is a faith that "we can solve these problems" I'm not sure the machine you describe would have either characteristic. I don't know how to formalize this, but it seems an important aspect of the situation.

IIRC, Peter de Blanc told me that any consistent utility function must have an upper bound (meaning that we must discount lives like Steve suggests). The problem disappears if your upper bound is low enough. Hopefully any realistic utility function has such a low upper bound, but it'd still be a good idea to solve the general problem.Nick, please see my blog (just click on my name). I have a post about this.

"Let the differential be negative. Same problem. If the differential is not zero, the AI will exhibit unreasonable behavior. If the AI literally thinks in Solomonoff induction (as I have described), it won't want the differential to be zero, it will just compute it."

How can a computation arrive at a nonzero differential, starting with zero data? If I ask a rational AI to calculate the probability of me typing "QWERTYUIOP" saving 3^^^^3 human lives, it knows literally *nothing* about the causal interactions between me and those lives, because they are totally unobservable.

GeniusNZ, you have to consider not only all proposed gods, but all possible gods and reward/punishment structures. Since the number and range of conceivable divine rewards and punishments is infinite for each action, the incentives are all equally balanced, and thus give you no reason to prefer one action over another.

Ultimately, I think Tom McCabe is right -- the truth of a proposition depends in part on its meaningfulness.

What is the probability that the sun will rise tomorrow? Nearly 1, if you're thinking of dawns. Nearly 0, if you're thinking of Copernicus. Bayesian reasoning can evaluate propositions, but at the limit, one must already have a rational vocabulary in which to express hypotheses.

When someone threatens to kill 3^^^^3 people, this calls into question

1) whether the assertion is meaningful at all 2) whether the lives in question are equivalent to "human lives" already observed, or are unlike in kind -- in other words, whether they should be valued similarly.

After all, analogously to the original Wager's problem, these 3^^^^3 people could be of a negative moral value -- it could be good to kill them. And no, Pascal's Mugger cannot just respond that he means people like you and me, because they are obviously not exactly analogous, since they are unobservable.

I generally share Tom McCabe's conclusion, that is, that they exactly cancel out because a symmetry has not been broken. The reversed hypothesis has the same complexity as the original hypothesis, and the same evidence supporting it. No differential entanglement. However, I think that this problem is worth attention because a) so many people who normally agree disagree here, and b) I suspect that the problem is related to normal utilitarianism with no discounting and an unbounded future. Of course, we already have some solutions in that case and we should try them and see what we get. Our realistic AGI is boundedly rational in some respect to another. How does it's limitation in predicting the mundane consequences of any given action relate to its limitation in predicting the probabilities in Pascal's Mugging.

Benquo, replace "kill 3^^^^3 people" with "create 3^^^^3 disutility units" and the problem reappears.

Michael, do you really think the mugger's statement is _zero_ evidence?

It seems to me that the cancellation is an artifact of the particular example, and that it would be easy to come up with an example in which the cancellation does not occur. For example, maybe you have previous experience with the mugger. He has mugged you before about minor things and sometimes you have paid him and sometimes not. In all cases he has been true to his word. This would seem to tip the probabilities at least slightly in favor of him being truthful about his current much larger threat.

Even in that case I would assign enormously higher probability to the hypothesis that my deadbeat pal has caught some sort of brain disease that results in compulsive lying, than that such a person has somehow acquired reality-breaking powers but still has nothing better to do than hit me up for spare change.

Enormously higher probability is not 1. This still doesn't mean the statement is zero evidence.

I don't know - if he did actually have reality breaking powers, he would likely be tempted to put them to more effective use. If he would in fact be

lesslikely to be making the statement were it true, then it is evidence against, not evidence for, the truth of his statement.However clever your algorithm, at that level, something's bound to confuse it. Gimme FAI with checks and balances every time.

Is there a Godel sentence for human consciousness?

(My favorite proposal so far is: "I cannot honestly assert this sentence.")

You could always just give up being a consequentialist and ontologically refuse to give in to the demands of anyone taking part in a Pascal mugging because consistently doing so would lead to the breakdown of society.

Re: "However clever your algorithm, at that level, something's bound to confuse it. Gimme FAI with checks and balances every time."

I agree that a mature Friendly Artificial Intelligence should defer to something like humanity's volition.

However, before it can figure out what humanity's volition is and how to accomplish it, an FAI first needs to:

1. self-improve into trans-human intelligence while retaining humanity's core goals 2. avoid UnFriendly Behavior (for example, murdering people to free up their resources) in the process of doing step (1)

If the AI falls prey to a paradoxes early on in the process of self-improvement, the FAI has failed and has to be shut down or patched.

Why is that a problem? Because if the AI falls prey to a paradox later on in the process of self-improvement, when the computer can outsmart human beings, the result could be catastrophic. (As Eliezer keeps pointing out: a rational AI might not *agree* to be patched, just as Gandhi would not *agree* to have his brain modified into becoming a psychopath, and Hitler would not *agree* to have his brain modified to become an egalitarian. All things equal, rational agents will try to block any actions that would prevent them from accomplishing their current goals.)

So you want to create an elegant (to the point, ideally, of being "provably correct") structure that doesn't need patches or hacks. If you have to constantly patch or hack early on in the process, that increases the chances that you've missed something fundamental, and that the AI will fail later on, when it's too late to patch.

Rolf: I agree with everything you just said, especially the bit about patches and hacks. I just wouldn't be happy having a FAI's sanity dependent on

anysingle part of it's design, no matter how perfect and elegant looking, or provably safe on paper, or demonstrably safe in our experiments.However clever your algorithm, at that level, something's bound to confuse it.Odd, I've been reading moral paradoxes for many years and my brain never crashed once, nor have I turned evil. I've been confused but never catastrophically so (though I have to admit my younger self came close). My algorithm must be "beyond clever".

That's a remarkable level of resilience for a brain design which is, speaking professionally, a damn ugly mess. If I can't do aspire to do

at leastthat well, I may as well hang up my shingle and move in with the ducks.The modern human nervous system is the result of upwards of a hundred thousand years of brutal field-testing. The basic components, and even whole submodules, can be traced back even further. A certain amount of resiliency is to be expected. If you want to start from scratch and aspire to the same or higher standards of performance, it might be sensible to be prepared to invest the same amount of time and capital that the BIG did.

That you have not yet been crippled by a moral paradox or other standard rhetorical trick is comparable to saying that a server remains secure after a child spent an afternoon poking around with it and trying out lists of default passwords: a good sign, certainly, and a test many would fail, but not in itself proof of perfection.

Indeed, on a list of things we can expect evolved brains to be, ROBUST is very high on the list. ("rational" is actually rather hard to come by. To some degree, rationality improves fitness. But often its cost outweighs its benefit, hence the sea slug.)

Give me five dollars, or I will kill as many puppies as it takes to make you. And they'll go to hell. And there in that hell will be fire, brimstone, and rap with Engrish lyrics.

I think the problem is not Solomonoff inducton or Kolmogorov complexity or Bayesian rationality, whatever the difference is, but you. You don't want an AI to think like this because you don't want it to kill you. Meanwhile, to a true altruist, it would make perfect sense.

*Not really confident. It's obvious that no society of selfish beings whose members think like this could function. But they'd still, absurdly, be happier on average.*

Well, in that case, one possible response is for me to kill YOU (or report you to the police who will arrest you for threatening mass animal cruelty). But if you're really a super-intelligent being from beyond the simulation, then trying to kill you will inevitably fail and probably cause those 3^^^^3 people to suffer as a result.

(The most plausible scenario in which a Pascal's Mugging occurs? Our simulation is being tested for its coherence in expected utility calculations. Fail the test and the simulation will be terminated.)

You don't need a bounded utility function to avoid this problem. It merely has to have the property that the utility of a given configuration of the world doesn't grow faster than the length of a minimal description of that function. (Where "minimal" is relative to whatever sort of bounded rationality you're using.)

It actually seems quite plausible to me that our intuitive utility-assignments satisfy something like this constraint (e.g., killing 3^^^^^3 puppies doesn't *feel* much worse than killing 3^^^^3 puppies), though that might not matter much if you think (as I do, and I expect Eliezer does) that our intuitive utility-assignments often need a lot of adjustment before they become things a really rational being could sign up to.

Nick Tarleton, you say:

"Benquo, replace "kill 3^^^^3 people" with "create 3^^^^3 disutility units" and the problem reappears."

But

what isa disutility unit?Howcan there be that many? How do you know that what he supposes to be a disutility unit isn't from your persective a utility unit?Any similarly outlandish claim is a challenge not merely to your beliefs, but to your mental vocabulary. It can't be evaluated for probability until it's evaluated for meaning.

Utility functions have to be bounded basically because genuine martingales screw up decision theory -- see the St. Petersburg Paradox for an example.

Economists, statisticians, and game theorists are typically happy to do so, because utility functions don't really exist -- they aren't uniquely determined from someone's preferences. For example, you can multiply any utility function by a constant, and get another utility function that produces exactly the same observable behavior.

In the INDIVIDUAL case that is true. In the AGGREGATE case it's not.

Tiiba, keep in mind that to an altruist with a bounded utility function, or with any other of Peter's caveats, in may not "make perfect sense" to hand over the five dollars. So the problem is solveable in a number of ways, the problem is to come up with a solution that (1) isn't a hack and (2) doesn't create more problems than in solves.

Anyway, like most people, I'm not a complete utilitarian altruist, even at a philosophical level. Example: if an AI complained that you take up too much space and are mopey, and offered to kill you and replace you with two happy midgets, I would feel no guilt about refusing the offer, even if the AI could guarantee that overall utility would be higher after the swap.

Though, if the AI is a true utilitarian, why must it kill you in order to make the midgets? Aren't there plenty of asteroids that can be nanofabricated into midgets instead?

Candidate for weirdest sentence ever uttered: "Aren't there plenty of asteroids that can be nanofabricated into midgets instead?"

That's a remarkable level of resilience for a brain design which is, speaking professionally, a damn ugly mess....with vital functions inherited from reptiles. But it's been tested to death through history, serious failures thrown out at each step, and we've lots of practical experience and knowledge about how and why it fails. It wasn't built and run first go with zero unrecoverable errors.

I'm not advocating using evolutionary algorithms or to model from the human brain like Ray Kurzweil. I just mean I'd allow for unexpected breakdowns in any part of the system, however much you trust it. At least enough so if it fails it fails safe.

That's only my opinion, and it shouldn't be taken too seriously as I don't have much knowledge in the field at this time, but I thought I should explain what I meant.

I think that if you consider that the chance of a threat to cause a given amount of disutility being valid is a function of the amount of disutility then the problem mostly goes away. That is, in my experience any threat to cause me X units of disutility where X is beyond some threshold is less than 1/10 as credible as a threat to cause me 1 unit of disutility. If someone threatened to kill another person unless I gave them $5000 I would be worried. If they threatened to kill 10 poeple I would be very slightly less worried. If they threatened to kill 1000 people I would be roughly 10 times less worried. If they threatened to kill 1,000,000 people I wouldn't pay any attention at all. Taking these data points and extrapolating I form the heuristic that the chance of someone threatening me with X units of disutility over a threshold based on how much they are demanding and whether I can fulfill that demand decreases faster than linearly.

[i]Nothing could possibly be that weak.[/i]

On the contrary, I think it is not only that weak but actually far weaker. If you are willing to consider the existance of things like 3^^^3 units of disutility without considering the existence of chances like 1/4^^^4 then I believe that is the problem that is causing you so much trouble.

"Odd, I've been reading moral paradoxes for many years and my brain never crashed once, nor have I turned evil."

Even if it hasn't happened to you, it's quite common- think about how many people under Stalin had their brains programmed to murder and torture. Looking back and seeing how your brain could have crashed is *scary*, because it isn't particularly improbable; it almost happened to me, more than once.

g:

killing 3^^^^^3 puppies doesn't *feel* much worse than killing 3^^^^3 puppies...

..........................

I hereby award G the All-Time Grand Bull Moose Prize for Non-Extensional Reasoning and Scope Insensitivity.

Clough:

On the contrary, I think it is not only that weak but actually far weaker. If you are willing to consider the existance of things like 3^^^3 units of disutility without considering the existence of chances like 1/4^^^4 then I believe that is the problem that is causing you so much trouble.I'm certainly willing to consider the existence of chances like that, but to arrive at such a calculation, I can't be using Solomonoff induction.

Consider the plight of the first nuclear physicists, trying to calculate whether an atomic bomb could ignite the atmosphere. Yes, they had to do this calculation! Should they have not even bothered, because it would have killed so many people that the prior probability must be very low? The essential problem is that the universe doesn't care one way or the other and therefore events

do not in facthave probabilities that diminish with increasing disutility.Likewise, physics does not contain a clause prohibiting comparatively small events from having large effects. Consider the first replicator in the seas of ancient Earth.

Tiiba:

You don't want an AI to think like this because you don't want it to kill you. Meanwhile, to a true altruist, it would make perfect sense.So you're biting the bullet and saying that, faced with a Pascal's Mugger, you should give him the five dollars?

Would any commenters care to mug Tiiba? I can't quite bring myself to do it, but it needs doing.

Krishnaswami:

Utility functions have to be bounded basically because genuine martingales screw up decision theory -- see the St. Petersburg Paradox for an example.One deals with the St. Petersburg Paradox by observing that the resources of the casino are finite; it is not necessary to bound the utility function itself when you can bound the game within your world-model.

If you believe in the many worlds interpretation of quantum mechanics, you have to discount the utility of each of your future selves by his measure, instead of treating them all equally. The obvious generalization of this idea is for the altruist to discount the utility he assigns to other people by their measures, instead of treating them all equally.

But instead of using the QM measure (which doesn't make sense "outside the Matrix"), let the measure of each person be inversely related to his algorithmic complexity (his personal algorithmic complexity, which is equal to the algorithmic complexity of his universe plus the amount of information needed to locate him within that universe), and the problem is solved. The utility of a Turing machine can no longer grow much faster than its prior probability shrinks, since the sum of measures of people computed by a Turing machine can't be larger than its prior probability.

But there is another puzzle/paradox with Solomonoff induction that I don't know how to solve. I've written about it at http://groups.google.com/group/everything-list/browse_frm/thread/c7442c13ff1396ec/. Eliezer, do you think it would be suitable for a blog post here?

Wei, would it be correct to say that, under your interpretation, if our universe initially contains 100 super happy people, that creating one more person who is "very happy" but not "super happy" is a net negative, because the "measure" of all the 100 super happy people gets slightly discounted by this new person?

It's hard to see why I would consider this

the right thing to do- where does this mysterious "measure" come from?Eliezer, do you think it would be suitable for a blog post here?Mm... sure. "Bias against uncomputability."

That's a much more general problem, the problem of whether to use sums or averages in utility calculations with changing population size.

"Would any commenters care to mug Tiiba? I can't quite bring myself to do it, but it needs doing."

If you don't donate $5 to SIAI, some random guy in China will die of a heart attack because we couldn't build FAI fast enough. Please donate today.

Eli,

I agree that G's reasoning is an example of scope insensitivity. I suspect you meant this as a criticism. It seems undeniable that scope insensitivity leads to some irrational attitudes (e.g. when a person who would be horrified at killing one human shrugs at wiping out humanity). However, it doesn't seem obvious that scope insensitivity is pure fallacy. Mike Vassar's suggestion that "we should consider any number of identical lives to have the same utility as one life" seems plausible. An extreme example is, what if the universe were periodic in the time direction so that every event gets repeated infinitely. Would this mean that every decision has infinite utility consequence? It seems to me that, on the contrary, this would make no difference to the ethical weight of decisions. Perhaps somehow the utility binds to the information content of a set of events. Presumably, the total variation in experiences a puppy can have while being killed would be exhausted long before reaching 3^^^^^3.

Vann McGee has proven that if you have an agent with an unbounded utility function and who thinks there are infinitely many possible states of the world (ie, assigns them probability greater than 0), then you can construct a Dutch book against that agent. Next, observe that anyone who wants to use Solomonoff induction as a guide has committed to infinitely many possible states of the world. So if you also want to admit unbounded utility functions, you have to accept rational agents who will buy a Dutch book.

And if you do that, then the subjectivist justification of probability theory collapses, taking Bayesianism with it, since that's based on non-Dutch-book-ability.

I think the cleanest option is to drop unbounded utility functions, since they buy you

zeroadditional expressive power. Suppose you have an event space S, a preference relation P, and a utility function f from events to nonnegative real numbers such that if s1 P s2, then f(s1) < f(s2). Then, you can easily turn this into a bounded utility function g(s) = f(s)/(f(s) + 1). It's easily seen that g respects the preference relation P in exactly the same way as f did, but is now bounded to the interval [0, 1).G,

I was essentially agreeing with you that killing 3^^^^^3 vs 3^^^^3 puppies may not be ethically distinct. I would call this scope insensitivity. My suggestion was that scope insensitivity is not necessarily always unjustified.

Eliezer, creating another person in addition to 100 super happy people do not reduce the measures of those 100 super happy people. For example, suppose those 100 super happy people are living in a classical universe computed by some TM. The minimal information needed to locate each person in this universe is just his time/space coordinate. Creating another person does not cause an increase in that information for the existing people.

Is the value of my existence steadily shrinking as the universe expands and it requires more information to locate me in space?

If I make a large uniquely structured arrow pointing at myself from orbit so that a very simple Turing machine can scan the universe and locate me, does the value of my existence go up?

I am skeptical that this solution makes moral sense, however convenient it might be as a patch to this particular problem.

Yes.

Doing something like that proves you're clever enough to come up with a plan for something that's unique in all the universe, and then marshal the resources to make it happen. That's worth something.

No. He is either clever enough or not. Proving it doesn't change his value.

Stephen, you can't have been agreeing with me about that since I didn't say it, even though for some reason I don't understand (perhaps I was very unclear, but I don't see how) Eliezer chose to interpret me doing so and indeed going further to say that it *isn't* ethically distinct.

Random question:

The number of possible Turing machines is countable. Given a function that maps the natural numbers onto the set of possible Turing machines, one can construct a Turing machine that acts like this:

If machine #1 has not halted, simulate the execution of one instruction of machine #1

If machine #2 has not halted, simulate the execution of one instruction of machine #2

If machine #1 has not halted, simulate the execution of one instruction of machine #1

If machine #3 has not halted, simulate the execution of one instruction of machine #3

If machine #2 has not halted, simulate the execution of one instruction of machine #2

If machine #1 has not halted, simulate the execution of one instruction of machine #1

etc.

This Turing machine, if run, would eventually make all possible computations. (One could even run a program like this on a real, physical computer, subject to memory and time limitations.) Does running such a program have any ethical implications? If running a perfect simulation of a reality is essentially the same as creating that reality, would running this program for a long enough period of time actually cause all possible computable universes to come into existence? Does the existence of this program have any implications for the hypothesis that "our universe is a computer simulation being run in another universe?"

I've long felt that simulations are NOT the same as actual realities, though I can't precisely articulate the difference.

As others have basically said:

Isn't the point essentially that we believe the man's statement is uncorrelated with any moral facts? I mean if we did, then its pretty clear we can be morally forced into doing something.

Is it reasonable to believe the statement is uncorrelated with any facts about the existence of many lives? It seems so, since we have no substantial experience with "Matrices", people from outside the simulation visting us, 3^^^^^^3, the simulation of moral persons, etc...

Consider, the statement 'there is a woman being raped around the corner'. We are morally obliged to look around the corner. We have no more direct proof of the truth of this statement than of Pascal's mugger's statement. But we have good reason to believe the statement is correlated with a fact in one case, but no such reason in the other.

Can a machine be made that will consistently give zero correlation to this sort of thing? Hell if I know. Probably no, since you if iterate the known enough you get the absurd. But any claim that conditional probability of 3^^^^3 lives being simulated and destroyed is 1/(3^^^^^3) or something is a pile of horseshit.

Eliezer, you can interpret rocks as minds if you make the interpretation complex enough. Why do you ignore these rock-minds if not because you discount them for algorithmic complexity?

First, questions like "if the agent expects that I wouldn't be able to verify the extreme disutility, would its utility function be such as to actually go through spending the resources to cause the unverifiable disutility?"

That an entity with such a utility function exists would manage to stick around long enough in the first place itself may drop the probabilities by a whole lot.

Perhaps best to restrict ourselves to the case of the disutility being verifiable, but only after the fact. (Has this agent ever pulled this soft of thing before? etc..) and that verification doesn't open in the present a causal link allowing for other means of preventing the disutility. There's alot going on here.

I'm not sure, but maybe the reasoning would go not so much for the single specific case, but the process would reason by computing the expected utility of following a rule which would result in it being utterly vulnerable to any agent that merely claims to be capable of causing bignum units of disutility.

Something reasoning along the lines of following such a rule would allow agents in general to order the process to cause plenty of disutility. And that, in itself, would seem to have plenty of expected disutility.

However, _if_ after chugging through the math, it didn't balance out and still the expected disutility from the existance of the disutility threat was greater, then perhaps allowing oneself to be vulnerable to such threats is genuinely the correct outcome, however counterintuitive and absurd it would seem to us.

Eliezer> Is the value of my existence steadily shrinking as the universe expands and it requires more information to locate me in space?

Yes, but the value of everyone else's existence is shrinking by the same factor, so it doesn't disturb the preference ordering among possible courses of actions, as far as I can see.

Eliezer> If I make a large uniquely structured arrow pointing at myself from orbit so that a very simple Turing machine can scan the universe and locate me, does the value of my existence go up?

This is a more serious problem for my proposal, but the conspicuous arrow also increases the values of everyone near you by almost the same factor, so again perhaps it doesn't make as much difference as you expect.

Eliezer> I am skeptical that this solution makes moral sense, however convenient it might be as a patch to this particular problem.

I'm also skeptical, but I'd say it's more than just a patch to this particular problem. Treating everyone as equals no matter what their measures are, besides leading to counterintuitive results in this "Pascal's Mugging" thought experiment, is not even mathematically sound, since the sum of the small probabilities multiplied by the vast utilities do not converge to any finite value, no matter what course of action you choose.

The mathematics says that you *have* to discount each person's value by some function, otherwise your expected utilities won't converge. The only question is which function. Using the inverse of a person's algorithmic complexity seems to lead to intuitive results in many situations, but not all.

But I'm also open to the possibility that this entire approach is wrong... Are there other proposed solutions that make more sense to you at the moment?

I'll respond to a couple of other points I skipped over earlier.

Eliezer> It's hard to see why I would consider this the right thing to do - where does this mysterious "measure" come from?

Suppose you plan to measure the polarization of a photon at some future time and thereby split the universe into two branches of unequal weight. You do not treat people in these two branches as equals, but instead value the people in the higher-weight branch more, right? Can you answer why you consider that to be the right thing to do? That's not a rhetorical question, btw. If I knew the answer to that question I think I'd also know why discounting people by algorithmic complexity (or some other function) might be the right thing to do.

Stephen> Mentioning quantum mechanics serves only as a distraction.

In classical physics, the universe doesn't branch, but instead everything is predetermined by the starting conditions and laws of physics. There is no issue of people in unequal-weight branches, which I think might be analogous to people with different algorithmic complexities. That's why I brought up QM.

Maybe the origin of the paradox is that we are extending the principle of maximizing expected return beyond its domain of applicability. Unlike Bayes formula, which is an unassailable theorem, the principle of maximizing expected return is perhaps just a model of rational desire. As such it could be wrong. When dealing with reasonably high probabilities, the model seems intuitively right. With small probabilities it seems to be just an abstraction, and there is not much intuition to compare it to. When considering a game with positive expected return that comes from big payoffs and small probabilities, it reduces to the intuitive case if we have the opportunity to play the game many times, on the order of one over the payoff probability. This type of frequentist argument seems to be where the principle comes from in the first place. However, if the probabilities are so small that there is no possibility of playing the game that many times, then maybe a rational person just ignores it rather than dutifully investing in an essentially certain loss. Of course, if we relegate the principle of maximizing expected return to being just a limiting case, this leaves open the question of what more general model underlies it.

G: Sorry to put words in your mouth.

Wei:

You do not treat people in these two branches as equals, but instead value the people in the higher-weight branch more, right? Can you answer why you consider that to be the right thing to do?Robin Hanson's guess about mangled worlds seems very elegant to me, since it means that I can run a (large) computer with conventional quantum mechanics programmed into it, no magic in its transistors, and the resulting simulation will contain sentient beings who experience the same probabilities we do.

Even so, I'd have to confess myself confused about why I find myself in a simple universe rather than a noisy one.

Not all infinities are equal, there exists a hierarchy. Look at real numbers versus integers.

kthxbye

Stephen, no problem. Incidentally, I share your doubt about the optimality of optimizing *expected* utility (though I wonder whether there might be a theorem that says anything coherent can be squeezed into that form).

CC, indeed there are many infinities (not merely infinitely many, not merely more than we can imagine, but more than we can describe), but so what? Any sort of infinite utility, coupled with a nonzero finite probability, leads to the sort of difficulty being contemplated here. Higher infinities neither help with this nor make it worse, so far as I can see. (I suppose it's worth considering that it might conceivably make sense for an agent's utilities to live in some structure "richer" than the usual real numbers, like Conway's surreal numbers, where there are infinities and infinitesimals aplenty. But I think there are technical difficulties with this sort of scheme; for instance, doing calculus over the surreals is problematic. And of course we actually only have finite brains, so whatever utilities we have are presumably representable in finite terms even if they feature incommensurabilities of the sort that might be modelled in terms of something like the surreal numbers. But all this is a separate issue.)

I have a paper which explores the problem in a somewhat more general way (but see especially section 6.3).

Infinite Ethics: http://www.nickbostrom.com/ethics/infinite.pdf

People have been talking about assuming that states with many people hurt have a low (prior) probability. It might be more promising to assume that states with many people hurt have a low

correlationwith what any random person claims to be able to effect.Eliezer, I think Robin's guess about mangled worlds is interesting, but irrelevant to this problem. I'd guess that for you, P(mangled worlds is correct) is much smaller than P(it's right that I care about people in proportion to the weight of the branches they are in). So Robin's idea can't explain why you think that is the right thing to do.

Nick, your paper doesn't seem to mention the possibility of discounting people by their algorithmic complexity. Is that an option you considered?

Pascal's wager type arguments fail due to their symmetry (which is preserved in finite cases).Even if our priors are symmetric for equally complex religious hypotheses, our posteriors almost certainly won't be. There's too much evidence in the world, and too many strong claims about these matters, for me to imagine that posteriors would come out even. Besides, even if two religions are equally probable, there may be certainly be non-epistemic reasons to prefer one over the other.

However, _if_ after chugging through the math, it didn't balance out and still the expected disutility from the existance of the disutility threat was greater, then perhaps allowing oneself to be vulnerable to such threats is genuinely the correct outcome, however counterintuitive and absurd it would seem to us.I agree. If we really trust the AI doing the computations and don't have reason to think that it's biased, and if the AI has considered all of the points that have been raised about the future consequences of showing oneself vulnerable to Pascalian muggings, then I feel we should go along with the AI's conclusion. 3^^^^3 people is too many to get wrong, and if the probabilities come out asymmetric, so be it.

Maybe the origin of the paradox is that we are extending the principle of maximizing expected return beyond its domain of applicability.In addition to a frequency argument, one can in some cases make a different argument for maximizing expected value even in one-time-only scenarios. For instance, if you knew you would become a randomly selected person in the universe, and if your only goal was to avoid being murdered, then minimizing the expected number of people murdered would also minimize the probability that you personally would be murdered. Unfortunately, arguments like this make the assumption that your utility function on outcomes takes only one of two values ("good," i.e., not murdered, and "bad," i.e., murdered); it doesn't capture the fact that being murdered in one way may be twice as bad as being murdered in another way.

Even if there is nobody currently making a bignum-level threat, maybe the utility-maximizing thing to do is to devote substantial resources to search for low-probability, high-impact events and stop or encourage them depending on the utility effect. After all, you can't say the probability of

everypossibility as bad as killing 3^^^^3 people is zero.Nick Tarleton,

Yes, it is probably correct that one should devote substantial resources to low probability events, but what are the odds that the universe is not only a simulation, but that the containing world is *much* bigger; and, if so, does the universe just not count, because it's so small? The bounded utility function probably reaches the opposite conclusion that only this universe counts, and maybe we should keep our ambitions limited, out of fear of attracting attention.

"I find myself in a simple world rather than a noisy one."

Care to expand on that?

Robin: Great point about states with many people having low correlations with what one random person can effect. This is fairly trivially provable.

Utilitarian: Equal priors due to complexity, equal posteriors due to lack of entanglement between claims and facts.

Wei Dai, Eliezer, Stephen, g: This is a great thread, but it's getting very long, so it seems likely to be lost to posterity in practice. Why don't the three of you read the paper Neel Krishnaswami referenced, have a chat, and post it on the blog, possibly edited, as a main post?

"The paper I referenced:

Vann McGee (1999)

An airtight Dutch book

Analysis 59 (264), 257â€“265.

Posted by: Neel Krishnaswami | October 20, 2007 at 06:29 PM"

It might be more promising to assume that states with many people hurt have a low correlation with what any random person claims to be able to effect.Robin: Great point about states with many people having low correlations with what one random person can effect. This is fairly trivially provable.Aha!

For some reason, that didn't click in my mind when Robin said it, but it clicked when Vassar said it. Maybe it was because Robin specified "many people hurt" rather than "many people", or because Vassar's part about being "provable" caused me to actually look for a reason. When I read Robin's statement, it came through as just "Arbitrarily penalize probabilities for a lot of people getting hurt."

But, yes, if you've got 3^^^^3 people running around they can't

allhave sole control over each other's existence. So in a scenario where lots and lots of people exist, one has to penalizeby a proportional factorthe probability that any one person's binary decision can solely control the whole bunch.Even if the Matrix-claimant says that the 3^^^^3 minds created will be unlike you, with information that tells them they're powerless, if you're in a generalized scenario where anyone has and uses that kind of power, the vast majority of mind-instantiations are in leaves rather than roots.

This seems to me to go right to the root of the problem, not a full-fledged formal answer but it feels right as a starting point. Any objections?

This seems intuitively plausible.

The more outrageous the claim, the correspondingly less plausible is their ability to pull it off.

Especially when you evaluate the amount of resources they are demanding vs the number of resources that you would expect their implausibly difficult plan would require to be achieved.

That's not the point. None of those probabilities are as strong as 3^^^3. Maybe big, buy not THAT big.

The point is that no more than 1/3^^^3 people have sole control over the life or death of 3^^3 people. This improbability, that you would be one of those very special people, IS big enough.

(This answer fails unless your ethics and anthropics use the same measure. That's how the pig example works.)

*0 points [-]I was about to express mild amusement about how cavalier we are with jumping to, from and between numbers like 3^^^^3 and 3^^^3. I had to squint to tell the difference. Then it occurred to me that:

3^^3 is not even unimaginably big, Knuth arrows or no. It's about 1/5th the number of people that can fit in the MCG.

Being cavalier with proofreading =/= being cavalier with number size.

But that is indeed amusing.

Well, I didn't want to declare a proofreading error because 3^^^3 does technically fit correctly in the context, even if you may not have meant it. ;)

I was thinking the fact that we are so cavalier makes it easier to slip between them if not paying close attention. Especially since 3^^^3 is more commonly used than 3^^^^3. I don't actually recall Eliezer going beyond pentation elsewhere.

I know if I go that high I tend to use 4^^^^4. It appeals more aesthetically and is more clearly distinct. Mind you it isn't nearly as neat as 3^^^3 given that 3^^^3 can also be written and visualized conceptually as 3 -> 3 -> 3 while 4^^^^4 is just 4 -> 4 -> 4 not 4 -> 4 -> 4 -> 4.

So you're saying that the implausibility is that I'd run into a person that just happened to have that level of "power" ?

Is that different in kind to what I was saying?

If I find it implausible that the person I'm speaking to can actually do what they're claiming, is that not the same as it being implausible that I happen to have met a person that can do what this person is claiming/ (leaving aside the resource-question which is probably just my rationalisation as to why I think he couldn't pull it off).

Basically I'm trying to taboo the actual BigNum... and trying to fit the concepts around in my head.

It's implausible that

you'rethe person with that power. We could easily imagine a world in which everyone runs into a single absurdly powerful person. We could not imagine a world in which everyone was absurdly powerful (in their ability to control other people), because then multiple people would have control over the same thing.If you knew that he had the power, but that his action wasn't going to depend on yours, then you wouldn't give him the money. So you're only concerned with the situation where you have the power.

Ok, sure thing. I get what you're saying. I managed to encompass that implausibility also into the arguments I made in my restatement anyway, but yeah, I agree that these are different kinds of "unlikely thing"

*-1 points [-]In fact... let me restate what I think I was trying to say.

The mugger is making an extraordinary claim. One for which he has provided no evidence.

The amount of evidence required to make me believe that his claim is possible, grows at the same proportion as the size of his claim.

Think about it at the lower levels of potential claims.

1) If he claimed to be able to kill one person - I'd believe that he was capable of killing one person. I'd then weigh that against the likelihood that he'd pick

meto blackmail, and the low blackmail amount that he'd picked... and consider it more likely that he's lying to make a fast buck, than that he actually has a hostage somewhere ready to kill.2) If he claimed to be able to kill 3^3 people, I'd consider it plausible... with a greatly diminished likelihood. I'd have to weigh the evidence that he was a small-time terrorist, willing to take the strong risk of being caught while preparing to blow up a buildings-worth of people... or to value his life so low as to actually do it and die in the process. It's not very high, but we've all seen people like this in our lifetime both exist and carry out this threat. So it's "plausible but extremely unlikely".

The likelihood that I've: a) happened to run into one of these rare people and b) that he'd pick

me(pretty much a nobody) to blackmail combine to be extremely unlikely... and I'd reckon that those two, balanced against the much higher prior likelihood that he's just a con-artist, would fairly well cancel out against the actual value of a buildings-worth of people.Especially when you consider that the resources to do this would far outweigh the money he's asked for. As far as I know about people wiling to kill large numbers of people -

mostof them do it for a reason, and that reason is almost never a paltry amount of cash. It's still possible... after all the school-killers have done crazy stunts to kill people for a tiny reason... but usually there's fame or revenge involved... not blackmail of a nobody.3) So now we move to 3^^3 people. Now, I personally have never seen that many die in one sitting (or even as the result of a single person)... but my Grandfather did, and using technology from 65 years ago.

It is plausible, though even less likely than before, that the person I've just run into happens to be willing and able to use a nuke on a large city, or to have the leadership capabilities (and luck) required to take over a country and divert it's resources to killing that number of people.

I would consider it exponentially less likely that he'd pick

meto blackmail about this... and certainly not for such a pitiful amount of cash. People that threaten this kind of thing are either after phenomenal amounts of money, recognition or some kind of political or religious statement.... they are extremely unlikely to find a random citizen to blackmail for a tiny amount of cash. The likelihood that this is a con seems about as high as the number of people to potentially die.4) Now we hit the first real BigNum. AFAIK, the world has never seen 3^^^3 sentient intelligences ever die in one sitting. We don't have that many people on the Earth right now. Maybe the universe has seen it somewhere... some planetary system wiped out in a supernova. It's plausible... but now think of the claims the guy is making:

a) that he can create (or knows of) a civilisation that contains that number of sentient beings.

b) that he (and he alone) has the ability to destroy that civilisation, and can do so at whim and that

c) it's worthwhile him doing so for the mere pittance he's demanding from a complete, unrelated nobody... (or potentially the whim of watching said nobody squirm).

I actually think that the required (and missing) evidence for his outrageous claims stack fairly evenly against the potential downside of his claims actually being true.

So, to get back to the original point: In my mind, as each step grows exponentially more extreme, so does the evidence required to support such a ludicrous claim. These two cancel out roughly evenly, leaving the leftovers of "is he likely to have picked me?" and other smaller probabilities to actually sway the balance.

Those, added with the large disutility of "encouraging the guy to do it again" would sway me to choose not to give him Â£5, but to walk away, then immediately find the nearest police officer...

3+3, 3*3, 3^3, 3^^3, 3^^^3, etc. grows much faster than exponentially. a^b, for any halfway reasonable a and b, can't touch 3^^^3

3^^^3=3^^(3^^3)=3^^(7625597484987)=3^(3^^(7625597484986))

It's not an exponential, it's a huge, huge tower of exponentials. It is simply too big for that argument to work.

*-1 points [-]Yes, I should not have used the word exponential... but I don't know the word for "grows at a rate that is a tower of exponentials"... "hyperexponential" perhaps?

However - I consider that my argument still holds. That the evidence required grows

at the same rateas the size of the claim.The evidence must be of equal value to the claim.

(from "extraordinary claims require extraordinary evidence")

My point in explaining the lower levels is that is that we don't demand evidence from most claimants of small amounts of damage because we've already seen evidence that these threats are plausible. But if we start getting to the "hyperexponential" threats, we hit a point where we suddenly realise that there is no evidence supporting the plausibility of the claim... so we automatically assume that the person is a crank.

3^^3 is a thousand times larger than the number of people currently alive.

oops, yes I mixed up 3^^3 with 3^^^3

Ok, so skip step 3 and move straight on to 4 ;)

*0 points [-]So can we solve the problem by putting some sort of upper bound on the degree to which ethics and anthropics can differ, along the lines of "creation of 3^^^^3 people is at most N times less probable than creation of 3^^^^3 pigs, so across the ensemble of possible worlds the prior against your being in a position to influence that many pigs still cuts down the expected utility from something vaguely like 3^^^^3 to something vaguely like N"?

Robin's anthropic argument seems pretty compelling in this example, now that I understand it. It seems a little less clear if the Matrix-claimant tried to mug you with a threat not involving many minds. For example, maybe he could claim that there exists some giant mind, the killing of which would be as ethically significant as the killing of 3^^^^3 individual human minds? Maybe in that case you would anthropically expect with overwhelmingly high probability to be a figment inside the giant mind.

I think that Robin's point solves this problem, but doesn't solve the more general problem of an AGI's reaction to low probability high utility possibilities and the attendant problems of non-convergence.

The guy with the button could threaten to make an extra-planar factory farm containing 3^^^^^3 pigs instead of killing 3^^^^3 humans. If utilities are additive, that would be worse.

The guy with the button could threaten to make an extra-planar factory farm containing 3^^^^^3 pigs instead of killing 3^^^^3 humans. If utilities are additive, that would be worse.Congratulations, you made my brain asplode.

3^^^^^^3 copies of that brain, fates all dependent on the original pondering this thread.

All fates equal, I

thinktheir incentive to solve the mystery equals that for one alone.Eliezer, what if the mugger (Matrix-claimant) also says that he is the only person who has that kind of power, and he knows there is just one copy of you in the whole universe? Is the probability of that being true less than 1/3^^^^3?

Don't dollars have an infinite expected value (in human lives or utility) anyway, especially if you take into account weird low-probability scenarios? Maybe the next mugger will make even bigger threats.

nextmugger? There's a distinctly high probability thatthismugger will return with higher blackmail demands.Even if the Matrix-claimant says that the 3^^^^3 minds created will be unlike you, with information that tells them they're powerless, if you're in a generalized scenario where anyone has and uses that kind of power, the vast majority of mind-instantiations are in leaves rather than roots.You would have to abandon Solomonoff Induction (or modify it to account for these anthropic concerns) to make this work. Solomonoff Induction doesn't let you consider just "generalized scenarios"; you have to calculate each one in turn, and eventually one of them is guaranteed to be nasty.

To paraphrase Wei's example: the mugger says, "Give me five dollars, or I'll simulate and kill 3^^^^3 people,

and I'll make sure they're aware that they are at the leaf and not at the node". Congratulations, you now have over 3^^^^3 bits of evidence (in fact, it's a tautology with probability 1) that the following proposition is true: "ifthe mugger's statement is correct, then I am the one person at the node and am not one of the 3^^^^3 people at the leaf." By Solomonoff Induction, this scenario where his statement is literally true has > 1 / 2^(10^50) probability, as it's easily describable in much less than 10^50 bits. Once you try to evaluate the utility differential of that scenario, boom, we're right back where we started.On the other hand, you could modify Solomonoff Induction to reflect anthropic concerns, but I'm not sure it's any better than just modifying the utility function to reflect anthropic concerns.

And, of course, there's still the pig problem in either case.

Michael, your pig example threw me into a great fit of belly-laughing. I guess that's what my mind look likes when it explodes. And I recall that was Marvin Minsky's prediction in

Society of Minds.You would have to abandon Solomonoff Induction (or modify it to account for these anthropic concerns) to make this work.To be more specific, you would have to alter it in such a way that it accepted Brandon Carter's Doomsday Argument.

"Congratulations, you made my brain asplode."

Read http://www.spaceandgames.com/?p=22 if you haven't already. Your utility function should not be assigning things arbitrarily large additive utilities, or else you get precisely this problem (if pigs qualify as minds, use rocks), and your function will sum to infinity. If you "kill" by destroying the exact same information content over and over, it doesn't seem to be as bad, or even bad at all. If I made a million identical copies of you, froze them into complete stasis, and then shot 999,999 with a cryonics-proof Super-Plasma-Vaporizer, would this be immoral? It would certainly be less immoral than killing a million ordinary individuals, at least as far as I see it.

Wei, no I don't think I considered the possibility of discounting people by their algorithmic complexity.

I can see that in the context of Everett it seems plausible to weigh each observer with a measure proportional to the amplitude squared of the branch of the wave function on which he is living. Moreover, it seems right to use this measure both to calculate the anthropic *probability* of me finding myself as that observer and the moral *importance* of that observer's well-being.

Assigning anthropic probabilities over infinite domains is problematic. I don't know of a fully satisfactory explanation of how to do this. One natural approach might to explore might be to assign some Turing machine based measure to each of the infinite observers. Perhaps we could assign plausible probabilities by using such an approach (although I'd like to see this worked out in detail before accepting that it would work).

If I understand your suggestion correctly, you propose that the same anthropic probability measure should also be used as a measure of moral importance. But there seems to me to be a problem. Consider a simple classical universe with two very similar observers. On my reckoning they should each get anthropic probability measure 1/2 (rejecting SIA, the Self-Indication Assumption). Yet it appears that they should each have a moral weight of 1. Does your proposal require that one accepts the SIA? Or am I misinterpreting you? Or are you trying to explicate not total utilitarianism but average utilitarianism?

It seems like this may be another facet of the problem with our models of expected utility in dealing with very large numbers. For instance, do you accept the Repugnant conclusion?

I'm at a loss for how to model expected utility in a way that doesn't generate the repugnant conclusion, but my suspicion is that if someone finds it, this problem may go away as well.

Or not. It seems that our various heuristics and biases against having correct intuitions about very large and small numbers are directly tied up in producing a limiting framework that acts as a conservative.

One thought, the expected utility of letting our god-like figure run this Turing simulation might well be positive! S/He is essentially *creating* these 3^^^3 people and then killing them. And in fact, it's reasonable to assume that expected disutility of killing them is entirely dependent on (and thus exactly balanced by) the utility of their creation.

So, our mugger doesn't really hand us a dilemma unless the claim is that this simulation is already *running*, and those people have lives worth living, but if you don't pay the $5, the program will be altered (sun will stop in the sky, so tto speak) and they will all be killed). This last is more of a nitpick.

It does seem to me that the bayesian inference we draw from this person's statement must be *extraordinarily* low, with an uncertainty much larger than its absolute value. Because a being which is both capable of this and willing to offer such a wager (either in truth or as a test) is deeply beyond our moral or intellectual comprehension. Indeed, if the claim is true, that fact will have utility implications that completely dwarf the immediate decision. If they are willing to do this much over 5 dollars, what will they do for a billion? Or for some end that money cannot normally purchase? Or merely at whim? It seems that the information we receive by failing to pay may be of value commensurate with the disutility of them truthfully carrying out their threat.

Regarding the comments about exploding brains, it's a wonder to me that we *are* able to think about these issues and not lose our sanity. How is it that a brain evolved for hunting/gathering/socializing is able to consider these problems at all? Not only that, but we seem to have some useful intuitions about these problems. Where on Earth did they come from?

Nick> Does your proposal require that one accepts the SIA?

Yes, but using a complexity-based measure as the anthropic probability measure implies that the SIA's effect is limited. For example, consider two universes, the first with 1 observer, and the second with 2. If all of the observers have the same complexity you'd assign a higher prior probability (i.e., 2/3) to being in the second universe. But if the second universe has an infinite number of observers, the sum of their measures can't exceed the measure of the universe as a whole, so the "presumptuous philosopher" problem is not too bad.

Nick> If I understand your suggestion correctly, you propose that the same anthropic probability measure should also be used as a measure of moral importance.

Yes, in fact I think there are good arguments for this. If you have an anthropic probability measure, you can argue that it should be used as the measure of moral importance, since everyone would prefer that was the case from behind the veil of ignorance. On the other hand, if you have a measure of moral importance, you can argue that for decisions not involving externalities, the global best case can be obtained if people use that measure as the anthropic probability measure and just consider their self interests.

BTW, when using both anthropic reasoning and moral discounting, it's easy to accidentally apply the same measure twice. For example, suppose the two universes both have 1 observer each, but the observer in the second universe has twice the measure of the one in the first universe. If you're asked to guess which universe you're in with some payoff if you guess right, you don't want to think "There's 2/3 probability that I'm in the second universe, and the payoff is twice as important if I guess 'second', so the expected utility of guessing 'second' is 4 times as much as the EU of guessing 'first'."

I think that to avoid this kind of confusion and other anthropic reasoning paradoxes (see http://groups.google.com/group/everything-list/browse_frm/thread/dd21cbec7063215b/), it's best to consider all decisions and choices from a multiversal objective-deterministic point of view. That is, when you make a decision between choices A and B, you should think "would I prefer if everyone in my position (i.e., having the same perceptions and memories as me) in the entire multiverse chose A or B?" and ignore the temptation to ask "which universe am I likely to be in?".

But that may not work unless you believe in a Tegmarkian multiverse. If you don't, you may have to use both anthropic reasoning and moral discounting, being very careful not to double-count.

To be fair, humans are surrounded by thousands of other species that evolved under the same circumstances and can't consider them.

Before I get going, please let me make clear that I do not

understand the math here (even Eliezer's intuitive bayesian paper

defeated me on the first pass, and I haven't yet had the courage to

take a second pass), so if I'm Missing The Point(tm), please tell

me.

It seems to me that what's missing is talking about the probability

of given level of resourcefulness of the mugger. Let me 'splain.

If I ask the mugger for more detail, there are a wide variety of

different variables that determine how resourceful the mugger claims

to be. The mugger could, upon further questioning, reveal that all

the death events are the same entity being killed in the same way,

which I call one death; given the unlikelyhood of the mugger telling

the truth in the first place, I'd not pay. Similarily, the mugger

could reveal that the deaths, while of distinct entities, happen one

at a time, and may even include time for the entities to grow up and

become functioning adults (i.e. one death every 18 years), in which

case I can almost certainly put the money to better use by giving it

to SIAI.

On the other end of the scale, the mugger can claim infinite

resources, so that the can complete the deaths (of entirely distinct

entities, which have lives, grow up, and then are slaughtered) in an

infinitely small amount of time. If the mugger does so, they don't

get the money, because I assign an infinitely small value to

probability of the mugger having infinite resources. Yes, the

mugger may live in a magical universe where having infinite

resources is easy, but you don't get a

get-out-of-probability-assignment-free card because you say the word

"magic"; I still have to base my probability assignment of your

claims on the world around me, in which we don't yet have the

computing power to simulate even one human in real time (ignoring

the software problem entirely).

Between these two extremes is an entire range of possibilities. The

important part here is that the probability I assign to "the mugger

is lying" is going to increase exponentially as their claim of

resources increases. Until the claimed rate of birth, growing, and

dying exceeds the rate of deaths we already have here on Earth, I

don't care, because I can better spend the money here. After we

reach that point (~150K per day), I don't care, because my

probability is something like 1/O(2^n) (Computer Science big-O

there; sorry, that's my background) where n is the multiple of

computer resources claimed over "one mind in realtime", so n is,

umm, 150K deaths per day = 53400000 deaths per year, 18 years for

each person, so I think n is 961200000?. That's not even counting

the probability discount due to the ridiculousness of the whole

claim.

The point here is that I don't care about the 3^^^^3 number; I only

care about the claimed deaths per unit time, how that compares to

the number of people currently dying on Earth (on whom I *know* I

can well-spend the $5) and the claimed resourcefulness of the

mugger. By the time we get up to where the 3^^^^3 number matters,

i.e. "I can kill one-onemillionth of 3^^^^3 people every realtime

year", my probability assignment for their claimed resourcefulness

is so incredibly low (and so incredibly lower than the numbers they

are throwing at me) that I laugh and walk away.

There is not, as far as I can tell, a sweet spot where the number of

lives I *might* save by giving the mugger the $5 is enough more than

the number of people currently dying on Earth to offset the

ridiculously low probability I'd be assiging to the mugger's

resourcefulness. I'd rather give the $5 to SIAI.

-Robin

My apologies for the horrific formatting; I wrote that huge diatribe in w3m before discovering the captcha needed javascript, and then pasted it here. If an admin can fix it, please do so.

-Robin

One idea is to tell the AI not to expend a portion of its resources greater than the chance of the mugger's statement being true.

I think a large universe full of randomly scattered matter is much more probable than a small universe that consists of a working human mind and little else.

"But, small as this probability is, it isn't anywhere near as small as 3^^^^3 is large"

Eliezer, I contend your limit!

*1 point [-]I think this scenario is ingenious. Here are a few ideas, but I'm really not sure how far one can pursue them / how 'much work' they can do:

(1) Perhaps the agent needs some way of 'absolving itself of responsibility' for the evil/arbitrary/unreasonable actions of another being. The action to be performed is the one that yields highest expected utility but only along causal pathways that don't go through an adversary that has been labelled as 'unreasonable'.

(Except this approach doesn't defuse the variation that goes "You can never wipe your nose because you've computed that the probability of this action killing 3^^^^3 people in a parallel universe is ever so slightly greater than the probability of it saving that number of people".)

(2) We only have a fixed amount of 'moral concern', apportioned somehow or other to the beings we care about. Our utility function looks like: Sum(over beings X) ConcernFor(X)*HappinessOf(X). Allocation of 'moral concern' is a 'competitive' process. The only way we can gain some concern about Y is to lose a bit of concern about some X, but if we have regular and in some sense 'positive' interactions with X then our concern for X will be constantly 'replenishing itself'. When the magician appears and tells us his story, we may acquire a tiny bit of concern about him and the people he mentions, but the parts of us that care about the people we know (a) aren't 'told' the magician's story and thus (b) refuse to 'relinquish' very much.

The trouble with is that it sounds too reminiscent of the insanely stupid moral behaviour of human beings (where e.g. they give exactly as much money to save a hundred penguins as ten thousand.)

(3) We completely abandon the principle of using minimum description length as some kind of 'universal prior'. (For some reason. And replace it with something else. For some reason.)

Our best understanding of the nature of the "simulation" we call reality has this concept we call "cause and effect" in place. So when something happens it has non-zero (though nigh infinitely small) effects on everything else in existence (progressively smaller effect with each degree of separation).

The effect that affecting 3^^^3 things (regardless of type or classification) has on other things (even if the individual effects of affecting one thing would be extremely small) would be non-trivial (enormously large even after a positively ludicrous degree of separations).

Once you consider the level of effect that this would have on the whole "simulation" you are forced to consider basically all possible futures. You have nigh-infinite good (when these things are removed/effected you end up with utopia and a range of all possible net benefits for the whole of the simulation) and nigh-infinite penalty (when these things are removed/effected you end up with hell and a range of all possible net losses for the whole of the simulation). I cannot foresee how an AI can possibly have enough processing power to overcome the vagary being unable to predict all possible futures following the event.

Moreover, I personally balk at the assumption of that level of responsibility. It is for the same reason that I balk at time travel scenarios. I refuse to be responsible for whatever changes are wrought across all of reality (which in sum become quite large when you consider a Vast possibly infinite universe regardless of how "small" the initial event seems).

Also does the probability assignment take into account the likelihood of the actor in question approaching you? Assuming there are 3^^^3 people (minds), then surely the probability assignment of approaching you specifically must be adjusted accordingly. I understand that "somebody has to be approached," but surely no one here is willing to contend that any of us have traits which are so exceptional that they cannot be found inside of a population which is 3^^^3 in size?

Assume that the basic reasoning for this is true, but nobody actually does the mugging. Since the probability doesn't actually make a significant difference to the expected utility, I'll just simplify and say there equal.

The total expected marginal utility, assuming you're equally likely to save or kill the people, would be (3^^^3 - 3^^^3) + (3^^^^3 - 3^^^^3) + (3^^^^^3 - 3^^^^^3) + ... = 0. At least, it would be if you count it by alternating with saving and killing. You could also count it as 3^^^3 + 3^^^^3 - 3^^^3 + 3^^^^^3 - 3^^^^3 + ... = infinity. Or you could count it as -3^^^3 - 3^^^^3 + 3^^^3 - 3^^^^^3 + 3^^^^3 - ... = -infinity. Or you could even do 3^^^3 - 3^^^3 + 3^^^^3 - 3^^^^3 + 3^^^^^3 - 3^^^^^3 + ... (without parentheses) which doesn't even converge to anything.

You could also construct hypothetical possibility sets where you can set it to add to any given number by rearranging the possibilities.

It's one thing when order matters for talking about total utility of an infinitely long universe. It at least has an order, assuming you don't mind abandoning special relativity, but what order are you even supposed to count expected utility in?

I figure the only way out of this is to use a prior that decreases with expected utility faster than those formulations of Occam's razor would suggest. I don't like the idea of doing this, but not doing so just doesn't add up.

The probability of some action costing delta-utility x and resulting in delta-utility y, where y >> x, is low. The Anti Gratis Dining modifier is x/y. These things I conjecture, anyways.

The apple-salespeep who says, "Give me $0.50, and I will give you an apple" is quite believable, unlike the apple-salespeep who claims, "Give me $3.50, and I will give apples to all who walk the Earth". We understand how buying an apple gets us an apple, but we know far less about implementing global apple distribution.

Suppose I have a Holy Hand Grenade of FAI, which has been carefully proofed by all the best mathematicians, programmers, and philosophers, and I am (of course) amongst them. And am randomly selected to activate it! Sadly, there is an ant caught in the pin. I can not delay to extricate it, for that means more deaths left unprevented. I pull the pin and kill the ant anyways.

So, the more understanding you have about the situation at hand, the less the AGD factor applies to the situation.

*5 points [-][Late edit: I have since retracted this solution as wrong, see comments below; left here for completeness. The ACTUAL solution that really works I've written in a different comment :) ]

I do believe I've solved this. Don't know if anyone is still reading or not after all this time, but here goes.

Eliezer speaks of the symmetry of Pascal's wager; I'm going to use something very similar here to solve the issue. The number of things that could happen next - say, in the next nanosecond - is infinite, or at the very least incalculable. A lot of mundane things could happen, or a lot of unforeseen things could happen. It could happen that a car would go through my living room and kill me. Or it could happen that the laws of energy conservation were violated and the whole world would turn into bleu cheese. Each of these possibilities could, in theory, have a probability assigned to it, given our priors.

But! We only have enough computing power to calculate a finite number of outcomes at any given moment. That means that we CANNOT go around assigning probabilities by calculation. Rather, we're going to need some heuristic to deal with all the probabilities we do NOT calculate.

Suppose our AI is very good at predicting things. It manages to assign SOME probability to what will happen next about 99% of the time (Note: My solution works equally well for anything from 0% to 100% minus epsilon - and I shouldn't have to explain why a Bayesian AI should never be 100% certain that it got an answer right). That means that 1% of the time, something REALLY surprises it; it just did not assign any probability at all. Now, because the number of things that could be in that category is infinite, they cancel out. Sure, we could all turn to cheese if it says "abracadabra". Or we could turn to cheese UNLESS it says so. The utility functions will always end in 0 for the uncalculated mass of probabilities.

That means that the AI always works under the assumptions that "or something I didn't see coming will happen; but I must be neutral regarding such an outcome until I know more about it".

Now. Say the AI manages to consider 1 million possibilities per prediction it makes (how it still gets 1% of them wrong is beyond me but again, the exact number doesn't matter for my solution). So any outcome that has NOT been calculated could, in fact, be considered to have a probability of 1%/ 1 million - not because there are only a million possibilities the AI hasn't considered, but because that is how many it could TRY to consider.

This number is your cutoff. Before you multiply a probability with a utility function, you subtract this number from the probability, first. So now if someone comes up to you and says it'll kill 3^^^^3 people and you decide to actually spend the cycles to consider how likely that is, and you get 1/googol, that number is LESS than the background noise of everything you don't have time to calculate. You round it down to zero, not because it is

arbitrarilysmall enough, but because anything you have not considered for calculation must be considered to have higher probability - and like in Pascal's wager, those options' utility is infinite and can counter any number that Pascal's Mugger can throw at me. You subtract, not an arbitrary number, but rather a number depending on how long the AI is thinking about the problem; how many possibilities it takes into account.Does this solve the problem? I think it does.

(By the way: ChrisA's way also works against this problem, except that coding your AI so that it may disregard value and morality if certain conditions are met seems like a pretty risky proposition).

The problem is one of rational behavior, not of bounded-rational hacks.

Are you saying that it's a good thing that the AI uses this rounding system and goes against its values this particular time?

If so, how did you tell that it was a good thing?

Can you mathematically formalize that intuition?

If you cannot do so, there is probably some other conflict between your intuitions and your AI code.

Actually, I think I made a mistake there.

Don't get me wrong, in my suggestion the AI is NOT going against its values nor being irrational, and this was not meant as a hack. Rather I'm claiming that the basic method of doing rationality as described needs revision that accounts for practicality, and if you disagree with that then your next rational move should DEFINITELY be to send me 50$ RIGHT NOW because I TOTALLY have a button that kicks 4^^^^4 puppies if I press it RIGHT HERE.

Having said that, I do think I might have made an error of intuition in there, so let's rethink it. Just because we should rethink what constitutes rational behavior does not mean I got it right.

Suppose I am an omnipotent being and have created a button that does something, once, if pressed. I truthfully tell you that there are several possible outcomes: 1. You receive 10$. This has a chance of 45% chance of happening. 2. You lose 5$. This, too, has a chance of 45% chance of happening. 3. Something else happens.

You should be pretty interested in what this "something else" might be before you press the button, since I've put absolutely no bounds on it. You could win 1000$. Or you could die. The whole world could die. You would wake up in a protein bath outside the Matrix. etc. etc. Some of these things you might be able to prepare for, if you know about them in advance.

If you're rational and you get no further information, you should probably press the button. The overall gain is 5$; as in Pascal's Wager, the infinity of possibilities that stem from the third option cancel each other out.

Now, suppose before I tell you that you get 10 guesses as to what the third thing is. Every time you guess, I tell you the precise probability that this thing is possible. Furthermore, the third option could do at least 12 different things, so no matter what you guessed, you would not be able to tell exactly what the button might do.

So you start guessing. One of your guesses is "3^^^^3 people will die horribly". I rate that one as a 10^-100 chance.

You've reached the end of the guesses and still a full 5% of probability remain - half of the third option's share.

So. Now do we press the button?

My claim was that the you should ignore every outcome smaller than 1% chance in this case, regardless of its utility. This now seems to me like a mistake. In theory, when we add the utility of all known options, it comes out extremely negative. Because the remaining 5% unknowns still have effectively zero chance of happening each, and they STILL cancel each other out.

I think I even know where my mathematical error was: I was assuming that anything less than 1% is a waste of a guess and therefore we should have guessed something else, which quite possibly has a higher chance - this establishes a cutoff for "a calculation that was not worth doing". However in this new example there are at least 12 things the button can do; essentially the number is infinite as far as I know. I should count myself VERY lucky to get 1% or more for anything I guess. In fact I should expect to get an answer of zero or epsilon for pretty much everything. That means that no guess is truly wasted or trivial.

Of course, if we don't press the button the Pascal Muggers will have won...

Back to the drawing board, I guess? :-/

If the injured parties are humans, I should be very skeptical of the assertion because a very small fraction, (1/3^^3)*1/10^(something), of people have the power of life and death over 3^^^3 other people, whereas 1/10^(something smaller) hear the corresponding hoax.

That's the only answer that makes sense because it's the only answer that works on a scale of 3^^^3.

I think.

"If the injured parties are humans, I should be very skeptical of the assertion because a very small fraction, (1/3^^3)*1/10^(something)"

I'm trying to think up several avenues. One is that the higher the claimed utility, the lower the probability (somehow); another tries to use the implications that accepting the claim would have on other probabilities in order to cancel it out.

I'll post a new comment if I manage to come up with anything good.

I know because of anthropics. It is a logical impossibility for more than 1/3^^^3 individuals to have that power. You and I cannot both have power over the same thing, so the total amount of power is bounded, hopefully by the same population count we use to calculate anthropics.

*2 points [-]Not in the least convenient possible world. What if someone told you that 3^^^3 copies of you were made before you must make your decision and that their behaviour was highly correlated as applies to UDT? What if the beings who would suffer had no consciousness, but would have moral worth as judged by you(r extrapolated self)? What if there was one being who was able to experience 3^^^3 times as much eudaimonia as everyone else? What if the self-indication assumption is right?

<troll> If you're going to engage in motivated cognition at least consider the least convenient possible world. </troll>

Am I talking to Omega now, or just some random guy? I don't understand what is being discussed. Please elaborate?

Then my expected utility would not be defined. There would be relatively simple worlds with arbitrarily many of them. I honestly don't know what to do.

Then my expected utility would not be defined. There would be relatively simple agents with arbitrarily sensitive utilities.

Then I would certainly live in a world with infinitely many agents (or I would not live in any worlds with any probability), and the SIA would be meaningless.

My cognition is motivated by something else - by the desire to avoid infinities.

*0 points [-]1) Sorry, I confused this with another problem; I meant some random guy.

2/3) Isn't how you decision process handles infinities rather important? Is there any corresponding theorem to the Von NeumannâMorgenstern utility theorem but without using either version of axiom 3? I have been meaning to look into this and depending on what I find I may do a top-level post about it. Have you heard of one?

edit: I found Fishburn, 1971, A Study of Lexicographic Expected Utility, Management Science. It's behind a paywall at http://www.jstor.org/pss/2629309. Can anyone find a non-paywall version or email it to me?

4) Yeah, my fourth one doesn't work. I really should have known better.

Sometimes, infinities must be made rigourous rather than eliminated. I feel that, in this case, it's worth a shot.

What worries me about infinities is, I suppose, the infinite Pascal's mugging - whenever there's a single infinite broken symmetry, nothing that happens in any finite world matters to determine the outcome.

This implies that all are thought should be devoted to infinite rather than finite worlds. And if all worlds are infinite, it looks like we need to do some form of SSA dealing with utility again.

This is all very convenient and not very rigorous, I agree. I cannot see a better way, but I agree that we should look. I will use university library powers to read that article and send it to you, but not right now.

I think you've just perfectly illustrated how

someScope Insensitivity can be good thing.Because a mind with

perfectscope sensitivity, will be diverted into chasing impossibly tiny probabilities for impossibly large rewards. If a good rationalist must win, then a good rationalist should commit to avoiding supposed rationality that makes him lose like that.So, here's a solution. If a probability is too tiny to be reasonably likely to occur in your lifespan, treat its bait as actually impossible. If you don't, you'll inevitably crash into effective ineffectiveness.

This seems to suggest a fuzzily-defined hack.

If you don't have a mathematical descriptor for what you consider "reasonably likely", then I'm afraid this doesn't promote us anywhere.

*0 points [-]This comment thread has grown too large :). I have a thought that seems to me to be the right way to resolve this problem. On the one hand, the thought is obvious, so it probably has already been played out in this comment thread, where it presumably failed to convince everyone. On the other hand, the thread is too large for me to digest in the time that I can reasonably give it. So I'm hoping that someone more familiar with the conversation here will tell me where I can find the sub-thread that addresses my point. (I tried some obvious word-searches, and nothing came up.)

Anyway, here is my point. I can see that the hypothesis that 3^^^^3 people are being tortured might be simple enough so that the Solomonoff prior is high enough so that the AI would give in to the mugger,

if the AI were using an un-updated Solomonoff prior. But the AI is allowed to update, right? And, from what the AI knows about humans, it can see that the low complexity of 3^^^^3alsomakes it more probable that a "philosopher out for a fast buck" would choose that number.So, the simplicity of 3^^^^3 contributes to

boththe hypothesis of a real torturerandthe hypothesis of the liar.And if, after taking all this into account, the AI still computes a high expected utility for giving in to the mugger, well, then I guess that that is really what it ought to do (assuming that it shares my utility function). But is there any reason to think that this is likely? Does it follow just from Eliezer's observation that "the utility of a Turing machine can grow much faster than its prior probability shrinks"? After all, it's the

updatedprobability that really matters, isn't it?That assumption is wrong, I argue.

I missed your post when it first came out. I've just commented on it.

It does seem that the probability of someone being able to bring about the deaths of N people should scale as 1/N, or at least 1/f(N) for some monotonically increasing function f. 3^^^^3 may be a more simply specified number than 1697, but it seems "intuitively obvious" (as much as that means anything) that it's easier to kill 1697 people than 3^^^^3. Under this reasoning, the likely deaths caused by not giving the mugger $5 are something like N/f(N), which depends on what f is, but it seems likely that it converges to zero as N increases.

It is an awfully difficult question, though, because how do we know we don't live in a world where 3^^^^3 people could die at any moment? It seems unlikely, but then so do a lot of things that are real.

Perhaps the problem lies in the idea that a Turing machine can create entities that have the moral status of humans. If there's a machine out there that can create and destroy 3^^^^3 humans on a whim, then are human lives really worth that much? But, on the other hand, there are laws of physics out there that have been demonstrated to create almost 3^^3 humans, so what is one human life worth on that scale?

On another note, my girlfriend says that if someone tried this on her, she'd probably give them the $5 just for the laugh she got out of it. It would probably only work once, though.

Incidentally: How would it affect your intuition if you instead could participate in the Intergalactic Utilium Lottery, where probabilities and payoffs are the same but where you trust the organizers that they do what they promise?

If I actually trust the lottery officials, that means that I have certain knowledge of the utility probabilities and costs for each of my choices. Thus, I guess I'd choose whichever option generated the most utility, and it wouldn't be a matter of "intuition" any more.

Applying that logic to the initial Mugger problem, if I calculated, and was certain of, there being at least a 1 in 3^^^^3 chance that the mugger was telling the truth, then I'd pay him. In fact, I could mentally reformulate the problem to have the mugger saying "If you don't give me $5, I will use the powers vested in me by the Intergalactic Utilium Lottery Commission to generate a random number between 1 and N, and if it's a 7, then I kill K people." I then divide K by N to get an idea of the full moral force of what's going on. If K/N is even within several orders of magnitude of 1, I'd better pay up.

The problem is the uncertainty. Solomonoff induction gives the claim "I can kill 3^^^^3 people any time I want" a substantial probability, whereas "common sense" will usually give it literally zero. If we trust the lottery guys, questions of induction versus common sense become moot - we

knowthe probability, and must act on it.I think this is actually the core of the issue - not certainty of your probability, per se, but rather how it is derived. I think I may have finally solved this!

See if you can follow me on this... If Pascal Muggers were completely independent instances of each other - that is, every person attempting a Pascal's Mugging has their own unique story and motivation for initiating it, without it correlating to you or the other muggers, then you have no additional information to go on. You shut up and multiply, and if the utility calculation comes out right, you pay the mugger. Sure, you're almost certainly throwing money away, but the off-chance more than offsets this by definition. Note that the probability calculation itself is complicated and not linear: Claiming higher numbers increases the probability that they are lying. However it's still possible they would come up with a number high enough to override this function.

At which point we previously said: "Aha! So this is a losing strategy! The Mugger ought not be able to arbitrarily manipulate me in this manner!" Or: "So what's stopping the mugger from upping the number arbitrarily, or mugging me multiple times?" ...To which I answer, "check the assumptions we started with".

Note that the assumption was that the Mugger is not influenced by me, nor by other muggings. The mugger's reasons for making the claim are their own. So "not trying to manipulate me knowing my algorithm" was an

explicitassumption here.What if we get rid of the assumption? Why, then now an increasingly higher utility claim (or recurring muggings) don't just raise the probability that the mugger is wrong/lying for their own inscrutable reasons. It additionally raises the probability that they are lying to manipulate me, knowing (or guessing) my algorithm.

Basically, I add in the question "why did the mugger choose the number 3^^^3 and not 1967? This makes it more likely that they are trying to overwhelm my algorithm, (mistakenly) thinking that it can thus be overwhelmed". If the mugger chooses 4^^^4 instead, this further (and proportionally?) increases said suspicion. And so on.

I propose that the combined weight of these probabilities rises faster than the claimed utility. If that is the case, then for all claimed utilities x higher than N, where N is a number that prompts a negative expected utility result, x would likewise produce a negative expected utility result.

Presumably, an AI with good enough grasp of motives and manipulation, this would not pose a problem for very long. We can specifically test for this behavior, checking the AI's analysis for increasingly higher claims and seeing whether the expected utility function really has a downward slope under these conditions.

I can try to further mathematize this (is this even a real word?). Is this necessary? The answer seems superficially satisfactory. Have I actually solved it? I don't really have a lot of time to keep grappling with it (been thinking about this on and off for the past few months), so I would welcome criticism even more than usual.

This is a very good point - the higher the number chosen, the more likely it is that the mugger is lying - but I don't think it quite solves the problem.

The probability that a person, out to make some money, will attempt a Pascal's Mugging can be no greater than 1, so let's imagine that it is 1. Every time I step out of my front door, I get mobbed by Pascal's Muggers. My mail box is full of Pascal's Chain Letters. Whenever I go online, I get popups saying "Click this link or 3^^^^3 people will die!". Let's say I get one Pascal-style threat every couple of minutes, so the probability of getting one in any given minute is 0.5.

Then, let the probability of someone genuinely having the ability to kill 3^^^^3 people, and then choosing to threaten me with that, be x per minute - that is, over the course of one minute, there's an x chance that a genuine extra-Matrix being will contact me and make a Pascal Mugging style threat, on which they will actually deliver.

Naturally, x is tiny. But, if I receive a Pascal threat during a particular minute, the probability that it's genuine is x/(0.5+x), or basically 2x. If 2x * 3^^^^3 is at all close to 1, then what can I do but pay up? Like it or not, Pascal muggings would be more common in a world where people can carry out the threat, than in a world where they can't. No amount of analysis of the muggers' psychology can change the prior probability that a genuine threat will be made - it just increases the amount of noise that hides the genuine threat in a sea of opportunistic muggings.

*0 points [-]But that is precisely it - it's no longer a Pascal mugging if the threat is credible. That is, in order to be successful, the mugger needs to be able to up the utility claim arbitrarily! It is assumed that we already know how to handle a credible threat, what we didn't know how to deal with was a mugger who could always make up a bigger number, to a degree where the seeming impossibility of the claim no longer offsets the claimed utility. But as I showed, this only works if you don't enter the mugger's thought process into the calculation.

This actually brings up an important corollary to my earlier point: The higher the number, the less likely the coupling is between the mugger's claim and the mugger's intent.

A person who can kill another person might well want 5$, for whatever reason. In contrast, a person who can use power from beyond the Matrix to torture 3^^^3 people already has IMMENSE power. Clearly such a person has all the money they want, and even more than that in the influence that money represents. They can probably create the money out of nothing. So already their claims don't make sense if taken at face value.

Maybe the mugger just wants me to surrender to an arbitrary threat? But in that case, why me? If the mugger really has immense power, they could create a person they know would cave in to their demands.

Maybe I'm special for some reason. But if the mugger is REALLY that powerful, wouldn't they be able to predict my actions beforehand, a-la Omega?

Each rise in claimed utility brings with it a host of assumptions that need to be made for the action-claimed reaction link to be maintained. And remember, the mugger's ability is not the only thing dictating expected utility, but also the mugger's intentions. Each such assumption not only weakens the probability of the mugger carrying out their threat because they can't, it also raises the probability of the mugger rewarding refusal and/or punishing compliance. Just because the off-chance comes true and the mugger contacting me actually CAN carry out the threat, does not make them sincere; the mugger might be testing my rationality skills, for instance, and could severely punish me for failing the test.

As the claimed utility approaches infinity, so does the scenario approach Pascal's Wager: An unknowable, symmetrical situation, where an infinite number of possible outcomes cancel each other out. The one outcome that isn't canceled out is the loss of 5$. So the net utility is negative. So I don't comply with the mugger.

I'm still not sure I'm fully satisfied with the level of math my explanation has, even though I've tried to set the solution in terms of limits and attractors. But I think I can draw a graph that dips under zero utility fairly quickly (or maybe doesn't really ever go over it?), and never goes back up - asymptotic at -5$ utility. Am I wrong?

Ah, my mistake. You're arguing based on the intent of a

legitimatemugger, rather than the fakes. Yes, that makes sense. If we let f(N) be the probability that somebody has the power to kill N people on demand, and g(N) be the probability that somebody who has the power to kill N people on demand would threaten to do so if he doesn't get his $5, then it seems highly likely that Nf(N)g(N) approaches zero as N approaches infinity. What's even better news is that, while f(N) may only approach zero slowly for easily constructed values of N like 3^^^^3 and 4^^^^4 because of their low Kolmogorov complexity, g(N) should scale with 1/N or something similar, because the more power someone has, the less likely they are to execute such a miniscule, petty threat. You're also quite right in stating that the more power the mugger has, the more likely it is that they'll reward refusal, punish compliance or otherwise decouple the wording of the threat from their actual intentions, thus making g(N) go to zero even more quickly.So, yeah, I'm pretty satisfied that N

f(N)g(N) will asymptote to zero, taking all of the above into account.(In more unrelated news, my boyfriend claims that he'd pay the mugger, on account of him obviously being mentally ill. So that's two out of three in my household. I hope this doesn't catch on.)

That is backward. It is

onlya Pascal mugging if the threat is credible. Like one made by Omega, who you mention later on.No, then it's just a normal mugging.

*0 points [-]If the threat is not credible from the perspective of the target it may only be an

attemptedmugging - not a proper mugging at all.Huh? Isn't the whole point of Pascal's mugging that it isn't likely and the mugger makes up for the lack of credibility by making the threat massive? If the mugger is making a credible threat we just call that a mugging.

The threat has to be credible at the level of probability it is assigned. It doesn't have to be

likely.How are you defining credible? It may be that we are using different notions of what this means. I'm using it to mean something like "capable of being believed" or "could be plausibly believed by a somewhat rational individual" but these have meanings that are close to "likely".

*3 points [-]"The threat has to be credible at the level of probability it is assigned. "

And what, precisely, does THAT mean? If I try to taboo some words here, I get "we must evaluate the likelihood of something happening as the likelihood we assigned for it to happen". That's simply tautological.

No probability is exactly zero except for self-contradictory statements. So "credible" can't mean "of zero probability" or "impossible to believe". To me, "credible" means "something I would not have a hard time believing without requiring extraordinary evidence", which in itself translates pretty much to ">0.1% probability". If you have some reason for distinguishing between a threat that is not credible and a threat with exceedingly low probability of being carried out, please state it. Also please note that my use of the word makes sense within the original context of my reply to HopeFox, who was discussing the implications of a world where such threats were

notincredible.Pascal's mugging happens when the probability you would assign disregarding manipulation is very low (not a credible threat by normal standards), with the claimed utility being arbitrarily high to offset this. If that is not the case, it's a non-challenge and is not particularly relevant to our discussion. Does that clarify my original statement?

*0 points [-]That makes sense. Whereas my statement roughly meant "Pascal's wager isn't about someone writing BusyBeaver<8>(3^^^3)" - that's not even a decision problem worth mentioning.

Comment deleted08 May 2011 11:43:48PM*[-]*0 points [-]I'm afraid I don't follow. I don't quite see how this negates the point I was making.

While it is conceivable that I simply lack the math to understand what you're getting at, it seems to me that a simply-worded explanation of what you mean (or alternately a simple explanation of why you cannot give one) would be more suitable in this forum. Or if this has already been explained in such terms anywhere, a link or reference would likewise be helpful.

This is known as the "What does God need with a starship?" problem.

Indeed. I was going to write that as part of my original post, and apparently forgot... Thanks for the addition :)

Philosophers of religion argue quite a lot about Pascal's wager and very large utilities or infinite utilities. I haven't bothered to read any of those papers, though. As an example, here is Alexander Pruss.

As I see it, the mugger seems to have an extremely bad hand to play.

If you evaluate the probability of the statement 'I will kill one person if you don't give me five dollars,' as being something that stands in a relationship to the occurrence of such threat being carried through on, and simply multiply up from there until you get to 3^^^^3 people, then you're going to end up with problems.

However, that sort of simplification â treating all the evidence as locating the same thing, only works for low multiples. (Which I'd imagine is why it feels wrong when you start talking about large numbers.) If you evaluate the evidence for and against different

partsof the statement, then you can't simply scale it up as a whole without scaling up all the variables that evidence attaches to. The probability that the person will carry through on a threat to kill 3^^^3 people for five dollars is going to zero out fairly quickly. You need to scale up the dollars asked to get all the bits of evidence to scale in proportion to each other.To make the threat plausible the mugger would have to be asking for a ridiculously large benefit for themselves. And when you start asking for that huge benefit then the computer simply has to answer whether the resources can be put to better use elsewhere.

As it stands, however, the variables in the statement haven't been properly scaled to keep the evidence for and against the proposed murders in a constant relationship. And, while it's just about possible that someone will kill a few hundred people for five dollars (destroying a train or the like would be a low investment exercise) the probability rapidly approaches zero as you increase the number of people that you're proposing to kill for five dollars.

By the time you're talking about 3^^^^3 lives the probability would have long since been reduced to an absurdity. Which would then be compared against all the other things the FAI could do with five dollars, that have far higher probabilities, and simply be dismissed as a bad gamble. (Since over a great length of time the computer could reasonably expect to approach the predicted loss/benefit ratio, regardless of whether the mugger actually killed 3^^^^3 people.)

*2 points [-]I think the problem might lie in the almost laughable disparity between the price and the possible risk. A human mind is not capable of instinctively providing a reason why it would be worth killing 3^^^^3 people - or even, I think, a million people - as punishment for not getting $5. A mind who would value $5 as much or more than the lives of 3^^^^3 people is utterly alien to us, and so we leap to the much more likely assumption that the guy is crazy.

Is this a bias? I'd call it a heuristic. It calls to my mind the discussion in Neal Stephenson's

Anathemabout pink nerve-gas-farting dragons. (Mandatory warning: fictional example.) The crux of it is, our minds only bother to anticipate situations that we can conceive of as logical. Therefore, the manifest illogicality of the mugging (why is 3^^^^3 lives worth $5; if you're a Matrix Lord why can't you just generate $5 or better yet, modify my mind so that I'm inclined to give you $5, etc.) causes us to anti-anticipate its truth. Otherwise, what's to stop you from imagining, as stated by Tom_McCabe2 (and mitchell_porter2, &c.), that typing the string "QWERTYUIOP" leads to, for example, 3^^^^3 deaths? If you imagine it, and conceive of it as a logically possible outcome, then regardless of its improbability, by your argument (as I see it), a "mind that worked strictly by Solomonoff induction" should cease to type that string of letters ever again. By induction, such a mind could cause itself to cease to take any action, which would lead to... well, if the AI had access to itself, likely self-deletion.That's my top-of-the-head theory. It doesn't really answer the question at hand, but maybe I'm on the right track...?

*2 points [-]Maybe I'm missing the point here, but why do we care about any number of simulated "people" existing outside the matrix at all? Even assuming that such people exist, they'll never effect me, nor effect anyone in the world I'm in. I'll never speak to them, they'll never speak to anyone I know and I'll never have to deal with any consequences for their deaths. There's no expectation that I'll be punished or shunned for not caring about people from outside the matrix, nor is there any way that these people could ever break into our world and attempt to punish me for killing them. As far as I care, they're not real people and their deaths or non-deaths do not factor into my utility function at all. Unless Pascal's mugger claims he can use his powers from outside the matrix to create 3^^^3 people in our world (the only one I care about) and then kill them here, my judgement is based soley on that fact that U(me loosing 5$) < 0.

So, let's assume that we're asking about the more interesting case and say that Pascal's mugger is instead threatening to use his magic extra-matrix powers to create 3^^^3 people here on Earth one by one and that they'll each go on international television and denounce me for being a terrible person and ask me over and over why I didn't save them and then explode into chunks of gore where everyone can see it before fading back out of the matrix (to avoid black hole concerns) and that all of this can be avoided with a single one time payment of 5$. What then?

I honestly don't know. Even one person being created and killed that way definitely feels worse than imagining any number of people outside the matrix getting killed. I'd be tempted on an emotional level to say yes and give him the money, despite my more intellectual parts saying this is clearly a setup and that something that terrible isn't actually going to happen. 3^^^3 people, while I obviously can't really imagine that many, is only worse since it will keep happening over and over and over until after the stars burn out of the sky.

The only really convincing argument, aside from the argument from absurdity ("That's stupid, he's just a philosopher out trying to make a quick buck.") is Polymeron's argument here

*0 points [-]Replace "matrix" with "light cone" and see if you would still endorse that.

There's not enough time.

If I ever ascend into dietyhood I'll be sorely tempted to go around offering Pascal's wager in various forms and inverting the consequences from what i said, or having no other consequences no matter what they do.

"Accept Christianity, and have a chance of heaven."

(Create heaven only for those who decline my Wager.)"Give me five dollars, or I will torture some people"

(Only torture people if they give me five dollars.)Check the multiverse to see how many beings will threaten people with Pascal's wager. Create 3^^(a large number of up arrows)^^^3 unique beings for each philosopher con man. Ask each new being: "Give me five dollars, or I will torture some people"

(Do nothing. Let them live out normal lives with the benefit of their money. [Don't worry, for each such reply I will add five dollars worth of goods and services to their world, to avoid deflation and related issues.])Why everyone is assuming the probability they are confronting a trickster testing them is zero, or that it is in any case a smaller probability than something different that they can't get a handle on because it is too small, I have no idea.

Since people are so taken with

onlytaking beings at their word, wouldn't a being telling them it will trick them if it gets the power confound them?Changing "matrix" to "light cone changes little, since I still don't expect to ever interact with them. The light cone example is only different insofar as I expect more people in my light cone to (irrationally) care about people beyond it. That might cause me to make some token efforts to hide or excuse my apathy towards the 3^^^3 lives lost, but not to the same degree as even 1 life lost here inside my light cone.

If you accept that someone making a threat in the form of "I will do X unless you do Y" is evidence for "they will do X unless you do ~Y", then by the principle of conservation of evidence, you have evidence that everyone who ISN'T making a threat in the form of "I will do X unless you do Y" will do X unless you Y. For all values of X and Y that you accept this trickster hypothesis for. And that is absurd.

This might be overly simplistic, but it seems relevant to consider the probability per murder. I am feeling a bit of scope insensitivity on that particular probability, as it is far too small for me to compute, so I need to go through the steps.

If someone tells me that they are going to murder one person if I don't give them $5, I have to consider the probability of it: not every attempted murder is successful, after all, and I don't have nearly as much incentive to pay someone if I believe they won't be successful. Further, most people don't actually attempt murder, and the cost to that person of telling me they will murder someone if they don't get $5 is much, much smaller then the cost of actually murdering someone. Consequences usually follow from murder, after all. I also have to consider the probability that this person is insane and doesn't care about the consequences: only the $5.

Still, only .00496% of people are murdered in a year. (According to Wolfram Alpha, at least) And while I would assign a higher probability to a person claiming to murder someone, it wouldn't jump dramatically- they could be lying, they could try but fail, etc. Even if I let "I will kill someone" be a 90% accurate test with only a 10% false positive rate- which I think is generous in the case of $5 with no additional evidence- as only being .004%. Even if it was 99% sure and 1% false positive, EXTREMELY generous odds, there is only a .4% total probability of it occurring.

In reality, I think there would be some evidence in the case of one murder. At very least I could get strong sociological cues that the person was likely to be telling the truth. However, since I am moving to an end point where they will be killing 3^^^^3 people, I'll leave that aside as it is irrelevant to the end example.

If such a person claimed they would murder 2 people, it would depend on whether I thought the probabilities of the events occurring together were dependent or independent: if him killing one person made it more likely that he would kill two, given the event (the threat) in question.

Now, if he says he will kill two people, and he kills one, he is unlikely to stop before killing another. BUT, there are more chances for complication or failure, and the cost:benefit for him shrinks by half, making the probability that he manages to or tries to kill anyone smaller. These numbers in reality would be affected by circumstance: it is a lot easier to kill two people with a pistol or a bomb than it is with your bare hands. But since I see no bomb or pistol and he is claiming some mechanism I have no evidence for, we'll ignore that reality for now.

I had trouble finding information on the rate of double homicide:single homicide to use as a baseline, but it seems likely that it is neither totally dependent, nor totally independent. In order to believe the threat credible, I have to believe (after hearing the threat) that they will attempt to kill two people, successfully kill one, AND successfully kill another. And if I put the probability of A+B at .004%, I can't very well put A+B+C at any higher. Since I used a 90% false positive rate for my initial calculation, let's use it twice: 81% false positive. We'll assume that the false negative (he murders people even when he says he won't) stays constant.

This means that each murder is

slightlymore likely than 90% as likely to occur as the murder before it. Now, it isn't exact, and these numbers get really, really small, so I'm looking at 3^3 as a reference.At 3^3, the cost has gone up 27x if he kills people, but the probability of the event has gone down to .06 of what it was. So, something like 1.7x more costly, given what was said above.

But all this was dependent on several assumed figures. So at what points does it balance out?

I'm a little tired for doing all the math right now, but some quick work showed that being only 80% sure of the test, with a 10% false positive rate, would be enough to where it would go down continuously. So if I am less than 80% sure of the test of "he says he will murder one person if I don't give him 5 dollars" then I can be sure that the probability that he will kill 3^^^^3 is far, far less than the cost if I am wrong.

I'm assuming that I am getting my math right here, and I am quite tired, so if anyone wishes to correct me on some portion of this I would be happy for the criticism.

FirstÂ¸ I didn't read all of the above comments, though I read a large part of it.

Regarding the intuition that makes one question Pascals mugging: I think it would be likely that there was a strong survival value in the ancestral environment to being able to detect and disregard statements that would cause you to pay money to someone else without there being any way to detect if these statements were true. Anyone without that ability would have been mugged to extinction long ago. This makes more sense if we regard the origin of our builtin utility function as a /very/ coarse approximation of our genes' survival fitness.

Regarding what the FAI is to do, I think the mistake made is assuming that the prior utility of doing ritual X is exactly zero, so that a very small change in our probabilities would make the expected utility of X positive. (Where X is "give the Pascal mugger the money"). A sufficiently smart FAI would have thought about the possibility of being Pascal-mugged long before that actually happens, and would in fact consider it a likely event to sometimes happen. I am not saying that this actually happening is not a tiny sliver of evidence in favor of the mugger telling the truth, but it is very tiny. The FAI would (assuming it had enough resources) compute for every possible Matrix scenario the appropriate probabilities and utilities for every possible action, taking the scenario's complexity into account. There is no reason to assume the prior expected utility for any religious ritual (such as paying Pascal muggers, whose statements you can't check) is exactly zero. Maybe the FAI finds that there is a sufficiently simple scenario in which a god exists and in which it is extremely utillious to worship that god, more so than any alternative scenarios. Or in which one should give in to (specific forms of) Pascal mugging.

However, the problem as presented in this blogpost implicitly assumes that the prior probabilities the FAI holds are such that the tiny sliver of probability provided by one more instance of Pascal's mugging happening, is enough to push the probability of the scenario of 'Extra-Matrix deity killing lots of people if I don't pay' over that of 'Extra-Matrix deity killing lots of people if I do pay'. Since these two scenarios need not have the exact same Kolmogorov complexity this is unlikely.

In short, either the FAI is already religious, (which may include as a ritual 'give money to people who speak a certain passphrase') or it is not, but the event of a Pascal mugging happening is unlikely to change its beliefs.

Now, the question becomes if we should accept the FAI doing things that are expected to favor a huge number of extra-matrix people at a cost to a smaller number of inside-matrix people. If we actually count every human life as equal, and we accept what Solomonoff-inducted bayesian probability theory has to say about huge payoff-tiny probability events and dutch books, the FAI's choice of religion would be the rational thing to do. Else, we could add a term to the AI's utility function to favor inside-matrix people over outside-matrix people, or we could make it favor certainty (of benefitting people known to actually exist) over uncertainty (of outside-matrix people not known to actually exist).

*1 point [-]Looks like strategic thinking to me. If you are to organize yourself to be prone to be Pascal-mugged, you will get Pascal mugged, and thus it is irrational to organize yourself to be Pascal-muggable.

edit: It is as rational to introduce certain bounds on applications of own reasoning as it is to try to build reliable, non-crashing software, or to impose simple rule of thumb limits on the output of the software that controls positioning of control rods in the nuclear reactor.

If you properly consider a tiny probability of mistake to your reasoning, a mistake that may lead to consideration of a number generated by a random string - a lot of such numbers are extremely huge - and apply some meta-cognition with regards to appearance of such numbers, you'll find that such extremely huge numbers are also disproportionally represented in products of errors in reasoning.

With regards to the wager, there is my answer: If you see someone bend over backwards to make a nickel, it is probably not Warren Buffett you're seeing. Indeed the probability of that person who's bending over backwards to make a nickel, having N$, would sharply fall off with increase of N. Here you see a being that is mugging you, and he allegedly has the power to simulate 3^^^^3 beings that he can mug, have sexual relations with, torture, what ever. The larger is the claim, the less probable it is that this is a honest situation.

It is however exceedingly difficult to formalize such answer or to arrive at it in a formal fashion. And for me, there could exist other wagers that are beyond my capability to reason correctly about.

For this reason as matter of policy I assume that I have an error per each inference step - the error that can result in consideration of an extremely huge number - and have an upper cut off on the numbers i'd use for considerations as an optimization strategy; if there is a huge number of this sort, more verification steps are needed. In particular, this has very high impact on morality on me. Any sort of situation where you are killing fewer people to save more people - those situations are extremely uncommon and difficult to conjecture - the appearance of such situation however can easily result from faulty reasoning.

*0 points [-]The problem seems to vanish if you don't ask "What is the expectation value of utility for this decision, if I do X", but rather "If I changed my mental algorithms so that they do X in situations like this all the time, what utility would I plausibly accumulate over the course of my entire life?" ("How much utility do I get at the 50th percentile of the utility probability distribution?") This would have the following results:

For the limit case of decisions where all possible outcomes happen infinitely often during your lifetime, you would decide exactly as if you wanted to maximize expectation value in an individual case.

You would not decide to give money to Pascals' mugger, if you don't expect that there are many fundamentally different scenarios which a mugger could tell you about: If you give a 5 % chance to the scenario described by Pascals mugger and believe that this is the only scenario which, if true, would make you give 5 $ to some person, you would not give the money away.

In contrast, if you believe that there are 50 different mugging scenarios which people will tell you during your life to pascal-mug you, and you assign an independent 5 % chance to all of them, you would give money to a mugger (and expect this to pay off occasionally).

*1 point [-]"Pascal's" Mugging requires me to believe that the apparent universe that we occupy, with its very low information content, is in fact merely part of a much larger program (in a causally linked and so incompressible way) which admits calculation within it of a specially designed (high-information content) universe with 3^^^^3 people (and not, say, as a side-effect of a low-information simulation that also computes other possibilities like giving immense life and joy to comparable numbers of people). The odds of that, if we use the speed priors, would seem to be 2^-(bits describing our universe + number of instructions to compute it):2^-(bits describing that vastly larger universe + number of instructions to compute it). That's going to be a

minimumof 1:-2^(O(3^^^^3)), so by the speed prior this particular kind of probability falls away hugely faster than the utility grows.However, I have little doubt that some creative philosopher can find some way to rescue the mugging argument in slightly different form.

I've been arguing about this with a friend recently [well, a version of this - I don't have any problems with arbitrarily large number of people being created and killed, unless the manner of their death is unpleasant enough that the negative value I assign to it exceeds the positive value of life].

He says that he can believe the person we are talking to has Agent Smith powers, but thinks that the more the Agent Smith promises, the less likely it is to be true, and this decreases faster the more that is promised, so that the probability that Agent Smith has the powers to create and kill [in an unpleasant manner] Y people multiplied by Y tends to zero as Y tends to infinity. . So the net expectancy tends towards zero. I disagree with this: I believe that if you assign probability X to the claim that the person you are talking to is genuinely from outside the Matrix [and that you're in the Matrix], then the probability that Agent Smith has the powers to create and kill [in an unpleasant manner] Y people multiplied by Y tends to infinity as Y tends to infinity.

Now, I think we can break this down further to find the root cause of our disagreement [this doesn't feel like a fundamental belief]: does anyone have any suggestions for how to go about doing this? We began to argue about entropy and the chance for Agent Smith to have found a way [from outside the Matrix = all our physics doesn't apply to him] to reverse it, but I think we went downhill from there.