DRAFT:Ethical Zombies - A Post On Reality-Fluid
I came up with this after watching a science fiction film, which shall remain nameless due to spoilers, where the protagonist is briefly in a similar situation to the scenario at the end. I'm not sure how original it is, but I certainly don't recall seeing anything like it before.
Imagine, for simplicity, a purely selfish agent. Call it Alice. Alice is an expected utility maximizer, and she gains utility from eating cakes. Omega appears and offers her a deal - they will flip a fair coin, and give Alice three cakes if it comes up heads. If it comes up tails, they will take one cake away her stockpile. Alice runs the numbers, determines that the expected utility is positive, and accepts the deal. Just another day in the life of a perfectly truthful superintelligence offering inexplicable choices.
The next day, Omega returns. This time, they offer a slightly different deal - instead of flipping a coin, they will perfectly simulate Alice once. This copy will live out her life just as she would have done in reality - except that she will be given three cakes. The original Alice, however, receives nothing. She reasons that this is equivalent to the last deal, and accepts.
(If you disagree, consider the time between Omega starting the simulation and providing the cake. What subjective odds should she give for receiving cake?)
Imagine a second agent, Bob, who gets utility from Alice getting utility. One day, Omega show up and offers to flip a fair coin. If it comes up heads, they will give Alice - who knows nothing of this - three cakes. If it comes up tails, they will take one cake from her stockpile. He reasons as Alice did an accepts.
Guess what? The next day, Omega returns, offering to simulate Alice and give her you-know-what (hint: it's cakes.) Bob reasons just as Alice did in the second paragraph there and accepts the bargain.
Humans value each other's utility. Most notably, we value our lives, and we value each other not being tortured. If we simulate someone a billion times, and switch off one simulation, this is equivalent to risking their life at odds of 1:1,000,000,000. If we simulate someone and torture one of the simulations, this is equivalent to risking a one-in-a-billion chance of them being tortured. Such risks are often acceptable, if enough utility is gained by success. We often risk our own lives at worse odds.
If we simulate an entire society a trillion times, or 3^^^^^^3 times, or some similarly vast number, and then simulate something horrific - an individual's private harem or torture chamber or hunting ground - then the people in this simulation *are not real*. Their needs and desires are worth, not nothing, but far less then the merest whims of those who are Really Real. They are, in effect, zombies - not quite p-zombies, since they are conscious, but e-zombies - reasoning, intelligent beings that can talk and scream and beg for mercy but *do not matter*.
My mind rebels at the notion that such a thing might exist, even in theory, and yet ... if it were a similarly tiny *chance*, for similar reward, I would shut up and multiply and take it. This could be simply scope insensitivity, or some instinctual dislike of tribe members declaring themselves superior.
Well, there it is! The weirdest of Weirdtopias, I should think. Have I missed some obvious flaw? Have I made some sort of technical error? This is a draft, so criticisms will likely be encorporated into the final product (if indeed someone doesn't disprove it entirely.)
Loading…
Subscribe to RSS Feed
Comments (116)
I don't currently accept the validity of this kind of anthropic reasoning (actually I am confused about anthropic reasoning in general). Is there an LW post where it is thoroughly defended?
Anthropic reasoning not working or not making sense in many cases is closer to being a standard position on LW (for example). The standard trick for making anthropic problems less confusing is to pose them as decision problems instead of as problems about probabilities. This way, when there appears to be no natural way of assigning probabilities (to instances of an agent) that's useful for understanding the situation, we are not forced to endlessly debate which way of assigning them anyway is "the right one".
You keep using that word. I don't think it means what you think it means.
Seriously, though, what do you think the flaw in the argument is, as presented in your quote?
I think I'm using "anthropic" in a way consistent with the end of the first paragraph of Fundamentals of kicking anthropic butt (to refer to situations in which agents get duplicated and/or there is some uncertainty about what agent an agent is). If there's a more appropriate word then I'd appreciate knowing what it is.
My first objection is already contained in Vladimir_Nesov's comment: it seems like in general anthropic problems should be phrased entirely as decision problems and not as problems involving the assignment of odds. For example, Sleeping Beauty can be turned into two decision problems: one in which Sleeping Beauty is trying to maximize the expected number of times she is right about the coin flip, and one in which Sleeping Beauty is trying to maximize the probability that she is right about the coin flip. In the first case, Sleeping Beauty's optimal strategy is to guess tails, whereas in the second case it doesn't matter what she guesses. In a problem where there's no anthropic funniness, there's no difference between trying to maximize the expected number of times you're right and trying to maximize the probability that you're right, but with anthropic funniness there is.
My second objection is that I don't understand how an agent could be convinced of the truth of a sufficiently bizarre premise. (I have the same issue with Pascal's mugging, torture vs. dust specks, and Newcomb's problem.) In this particular case, I don't understand how I could be convinced that another agent really has the capacity to perfectly simulate me. This seems like exactly the kind of thing that agents would be incentivized to lie about in order to trick me.
You may eventually obtain the capacity to perfectly simulate yourself, in which case you'll run into similar issues. I used Omega in a scenario a couple of years ago that's somewhat similar to the OP's, but really Omega is just a shortcut for establishing a "clean" scenario that's relatively free of distractions so we can concentrate on one specific problem at a time. There is a danger of using Omega to construct scenarios that have no real-world relevance, and that's something that we should keep in mind, but I think it's not the case in the examples you gave.
How would you characterize your issue with Pascal's mugging? The dilemma is not supposed to require being convinced of the truth of the proposition, just assigning it a non-zero probability.
Hmm. You're right. Upon reflection, I don't have a coherent rejection of Pascal's mugging yet.
Gotcha. Your posts have seemed pretty thoughtful so far so I was surprised by / curious about that comment. :)
Regarding anthropic reasoning, I always understood the term to refer to situations in which you could have been killed/prevented from existing.
How then, do you assign the odds?
You believe Omega because it's Omega, who always tells the truth and has access to godlike power.
You don't.
How does any particular agent go about convincing me that it's Omega?
So ... you don't know what the odds are, but you know how to act anyway? I notice that I am confused.
Assuming that there is only one godlike agent known or predicted in the environment, that Omega is a known feature of the environment, and that you have no reason to believe that e.g. you are hallucinating, then presumably all Omega needs to do is demonstrate his godlike powers - by predicting your every action ahead of time, say, or turning the sky green with purple spots.
In an anthropic situation, I don't think it makes sense to assign odds to statements like "I will see X" because the meaning of the term "I" becomes unclear. (For example, I don't think it makes sense for Sleeping Beauty to assign odds to statements like "I will see heads.") I can still assign odds to statements like "exactly fifteen copies of me will see X" by reasoning about what I currently expect my copies to see, given what I know about how I'll be copied, and using those odds I can still make decisions.
Omega needs to both always tell the truth and have access to godlike power. How does Omega prove to me that it always tells the truth?
I don't understand this, TBH, but whatever.
What do you think Alice should choose?
It is a known feature of the environment that people are regularly ambushed by an agent, calling itself Omega, which has never yet been known to lie and has access to godlike power.
There's nothing to choose until you've specified a decision problem.
An agent with godlike power can manufacture evidence that it has any other traits it wants, so observing such evidence isn't actually evidence that it has those traits.
Um, I have.
What do you think Alice should choose?
An agent with godlike power can manufacture evidence of anything. This seems suspiciously like a Fully General Counterargument. Such an agent could, in any case, directly hack your brain so you don't realize it can falsify evidence.
I don't know, but Omega does. Probably by demonstrating the ability to do something such that you believe the chance that it could be faked are epsilon^2, where epsilon is your prior belief that a given agent could have godlike powers.
If it helps you avoid fighting the hypothetical, Omega already knows what her answer will be, and has already acted on it.
Situation A: There are 3^^^^3 simulations of me, as well as myself. You come up to me and say "I'm going to torture forever one of our or your simulations, chosen randomly." Do I shrug and say, "well, whatever it 'certainly' won't be me" or do I scream in horror at the thought of you torturing a copy of me forever?
Situation B: There are me and 3^^^^3 other people in a rather large universe. You come up to me and say "I'm going to torture forever one of the people in this universe, chosen randomly." Do I shrug and say, "well, whatever, it 'certainly' won't be me" or do I scream in horror at the thought of you torturing someone forever?
What's the difference between these situations?
This is why I added Bob.
The difference is that a tiny risk of killing one, specific person is different to a certainty of killing any person, but not knowing who.
There's a jump between the paragraph beginning "Humans value each other's utility" and the next. Up to and including that paragraph, simulations are treated as equivalent to "real" people, but in the next, "the people in this simulation are not real", and are worth less than the Really Real. How do you get from the first part of the essay to the second?
I think the idea is meant to be that "one of many simulations" = "low probability" = "unimportant".
If so, I think this is simply a mistake. MugaSofer: you say that being killed in one of N simulations is just like having a 1/N chance of death. I guess you really mean 1/(N+1). Anyway, now Omega comes to you and says: unless you give me $100k (replace this with some sum that you could raise if necessary, but would be a hell of an imposition), I will simulate one copy of you and then stop simulating it at around the point of it's life you're currently at. Would you pay up? Would you pay up in the same way if the threat were "I'll flip a coin and kill you if it comes up heads"?
The right way to think about this sort of problem is still contentious, but I'm pretty sure that "make another copy of me and kill it" is not at all the same sort of outcome as "kill me with probability 1/2".
Now, suppose there are a trillion simulations of you. If you really believe what it says at the start of this article, then I think the following positions are open to you. (1) All these simulations matter about as much as any other person does. (2) All these simulations matter only about 10^-12 as much as any other person -- and so do I, here in the "real" world. Only if you abandon your belief that there's no relevant difference between simulated-you and real-you, do you have the option of saying that your simulations matter less than you do. In that case, maybe you can say that each of N simulations matters 1/N as much, though to me this feels like a bad choice.
No. He doubles my "reality", then halves it. This leaves me just as real as I was in the first place.
However, if he ran the simulation anyway, even if I paid up, then I think it does work out equivalent, because it's equivalent to creating the sim and then threatening to delete one of me if I didn't pay.
Does this answer your question?
I suspect I didn't made this as clear as I thought I had, but the term Really Real does not refer to people outside of simulations, it refers to people with vastly more simulations.
Well, as a draft comment, I don't think a trillion times, and 3^^^^^^3 times are conflatable in this context. There are simply too many arguments that apply to one and not the other.
For instance, a programmer can define 1 trillion unique societies. You could do this for instance, by having each society seeded from 12 variables with 10 levels each. You could then say that society seeded from 1,2,3,4,5,6,7,8,9,0,1,2 was the only society that was 1,2,3,4,5,6,7,8,9,0,1,2. I could generate a computer program which wrote out a textual description of each one. 1 trillion just isn't that large. For instance, there are well more than 1 trillion possible saved games in a simplistic game.
I don't even know if there are even 3^^^^^^3 potential possible states in the observable universe that can be physically distinguished from one another under current physics, but I would suspect not.
So I keep getting distracted by "A particular society out of 1 trillion societies may matter, but a particular society out of 3^^^^^^3 societies doesn't seem like it would matter any more than one of my atoms being less than one Planck length to the right would." but I'm not sure if that relates back to your point.
I suspect from some of the comments I'm getting that I didn't make this clear: copies are identical in this scenario. They receive the same inputs, make the same choices, they think and feel as one. They are, in short, one person (or civilization.) But one with vastly more reality-fluid (sometimes known as "measure") and thus, as far as I can tell, moral weight.
This is very thought provoking. Can you add clarity on your views on this point?
For instance, should I imply a "vastly" in front of moral weight as well as if there is a 1:1 correspondence or should I not do that?
Is this the only moral consideration you are considering on this tier? (I.E, there may be other moral considerations, but if this is the only "vast" one, it will probably outweigh all others.)
Does the arrangement of the copies reality fluid matter? Omega is usually thought of as a computer, so I am considering the file system. He might have 3 copies in 1 file for resilience, such as in a RAID array. Or he can have 3 copies that link to 3 files, such as in just having Sim001.exe and Sim002.exe and Sim003.exe having the exact same contents and being in the same folder. In both cases, the copies are identical. And if they are being run simultaneously and updated simultaneously, then the copies might not be able to tell which structure Omega was using. Which of these are you envisioning (or would it not matter? [Or do I not understand what a RAID array is?])
Some of these questions may be irrelevant, and if so, I apologize, I'm really am not sure I understand enough about your point to reply to it appropriately, and again, it does sound thought provoking.
Pretty much, yeah.
Well, I'm considering the torture's disutility, and the torturers' utility.
I'm not entirely sure I understand this question, but I don't think it should matter.
Is it worse to torture a virtual person running on redundant hardware (say 3 computers in lock-step, like the Space Shuttle used) whose permanent state (or backups) is stored on a RAID1 of disks instead of a virtual person running on a single CPU with one disk? Or even simpler; is it worse to torture a more massive person than a less massive person? Personally, I would say no.
Just like there's only one electron, I think there's only one of any particular thing, at least in the map. The territory may actually be weird and strange, but I don't have any evidence that redundant exact copies have as much moral weight as a single entity. I think that it's worse to torture 1 non-redundant person than it is to torture n-1 out of n exact copies of that person, for any n. That only applies if it's exactly the same simulation n-1 times. If those simulations start to diverge into n different persons, it starts to become as bad as torturing n different unique people. Eventually even those n-1 exact copies would diverge enough from the original to be considered copies of a different person with its own moral weight. My reasoning is just probabilistic in expected utility: It's worse for an agent to expect p(torture)=1 than p(torture)=n-1/n, and an identical agent can't distinguish between identical copies (including its environment) of itself.
As soon as you start torturing one of those identical agents, it ceases to be identical.
I guess the question from there is, does this produce a cascade of utility, as small divergences in the simulated universe produce slightly different agents for the other 6 billion people in the simulation, whose utility then exists independently?
That it is true, if unintuitive, that people gain moral worth the more "real" they get, is a position I have seen on LW, and the arguments do seem reasonable. (It is also rather more coherent when used in a Big Universe.) This post assumes that position, and includes a short version of the most common argument for that position.
Incidentally, I used to hold the position you describe; how do you deal with the fact that a tortured copy is, by definition, no longer "part" of the original?
Let N=3^^^^^^3, surely N nice world + another nice world is better than N nice worlds + a torture world. Why? Because another nice world is better than a torture world, and the prior existence of the N previous worlds shouldn't matter to that decision.
What about the probability of actually being in the torture world which is tiny 1/(N+1), the expected negative utility from this must surely be so small it can be neglected? Sure, but equally the expected utility of being the master of a torture world with probability 1/(N+1) can be neglected.
What this post tells me is that I'm still very very confused about reality fluid.
The torture world, in this case, is being used to satisfy the whims of the Niceworld's residents. Lots of Niceworld copies = lots of Reality = lots of utility. So goes the logic.
Since they are all the same, they can share a torture world.
Could this be an example of the noncentral fallacy? One big reason humans try to avoid death is because there is only one of each individual & once they die they are gone forever. If a simulation is made of me and gets turned off, there's still one of me (the original). In this alternate reality there's also the chance that Omega could always just make another new copy. I think the two situations are dissimilar enough that our standard intuitions can't be applied.
Well, presumably you would care less about that death depending on how "real" it was. If there's only one of you, you care as much as you do now (obviously,) if there's two sims you care half as much and so on.
First, I like the term e-zombie. It highlights the issue of "sim rights" vs "human rights" for me.
Second, I don't quite get the point you are trying to illustrate with this convoluted example. Is it that sims are intrinsically valueless or what? I don't see how this follows. Maybe some calculation is in order.
Not by a long shot. Pratchett has weirder ones in every chapter. For example, Only You Can Save Mankind.
It's that if sims that have less copies - are less "real" - are worth less, for the reasons presented above, then the whims of the many are worth more than the lives of the few, or the one.
Speaking as a Pratchett fan who's read pretty much everything he ever wrote, although it's been a while since I read OYCSM, I don't understand this. Ankh-Morpork is a weirdtopia, and so are a few other things he's written, (eg the "tradition as dead voting rights" bit from Johnny And The Dead,) but I don't recall anything like this. Maybe I'm just being an idiot, though.
What does it mean to simulate someone, and why should I value manipulation of a simulation?
How good does the simulation have to be before I value it; should I value a book in which I get cakes more? What about a fairly good simulation of the world, but the contents of my pantry are randomized each time I open it–should I value that simulation more if the expected number of cakes the next time the simulated me opens the pantry are higher?
I was assuming perfect quantum-level modelling of you and everything you interact with, acquired and sustained via magic. It makes things much simpler
As for your actual question ... I'm not sure. The sim would have to conscious, obviously, but the point at which it becomes "you" is ... unclear. It seems trivially true that a magical perfect simulation as above is "you", but an AI programmed to believe it's you is not. Beyond those two extremes ... it's tricky to say.
Of course, if utilities are additive, two almost-yous should be worth as much as one you with twice as much reality-fluid. So I guess humans can get away with ignoring the distinction between me and you, at least as long as they're using TDT or similar.
How close is a model that has an arbitrary number of cakes added?
I also say that no simulation has value to me if I am in a frame that knows they are a simulation. Likewise for quantum states that I don't manipulate.
Perfectly so before the cakes are added.
To be clear, are you actually asserting this or merely suggesting a possible resolution to the dilemma?
So you believe that it is irrelevant whether or not Omega' (a resident of the universe running a simulation) can create things of value to you but chooses not to? You have no preference for living in a world with constant physical laws?
It's a solution, but for it to apply to others they would have to share my values. What I'm saying is that there is no intrinsic value to me to the orientations of electrons representing a number which has a transformation function which results in a number which is perfectly analogous to me, or to any other person. Other people are permitted to value the integrity of those electrical orientations representing bits as they see fit.
So you, in fact, do not value simulations of yourself? Or anyone else, for that matter?
With the caveat that I am not a simulation for the purposes if that judgement. I care only about my layer and the layers which are upstream of (simulating) me, if any.
Well, obviously this post is not aimed at you, but I must admit I am curious as to why you hold this belief. What makes "downstream" sims unworthy of ethical consideration?
Maybe I've got a different concept of 'simulation'. I consider a simulation to be fully analogous to a sufficiently well-written computer program, and I don't believe that representations of numbers are morally comparable to living creatures, even if those numbers undergo transformations completely analogous to those creatures.
Why should I care if you calculate f(x) or f'(x), where x is the representation of the current state of the universe, f() is the standard model, and f'() is the model with all the cake?
Does that stay true if those representations are implemented in a highly distributed computer made out of organic cells?
I think I see where you're going with this, but your presentation actually leads somewhere else entirely; you discuss your point in the introduction and the conclusion, but you invest the majority of the weight of your argument into a place where your point is nowhere to be found.
Your argument, as presented, seems to be along these lines: Suppose there is one person, and one person is tortured. That's really important. Suppose there are a billion people, and one of them is tortured. That's not very important.
What I think you're getting at is more along these lines: Suppose there is one person, and one person is tortured; that is extremely important to that person. Suppose there are a billion copies of that one person, and one of them is tortured. Even a slight benefit arising from a decision leading to that torture may outweigh, by virtue of the fact that the benefit has been reproduced a billion less one times, the badness of the torture.
In other words, your presentation conflates the number of people with the importance of something bad happening to one of them. You don't discuss potential rewards at all; it's just, this torture is happening. Torture is equally bad regardless of the percentage of the population that is being tortured (given a specific number of people that are being tortured, I mean); we shouldn't care less about torture merely because there are more people who aren't being tortured. Whereas your actual point, as hinted at, is that, for some group that gains utility per individual as a result of the decision that results in that torture, the relative badness of that decision is dependent on the size of the group.
Or, in other words, you seem to be aiming for a discussion of dustmotes in eyes compared to torture of one person, but you're forgetting to actually discuss the dustmotes.
No. My argument is as follows: suppose there is one person, duplicated a billion times. These copies are identical, they are the same person. Suppose one copy is deleted. This is equivalent to a-billion-to-one odds of killing all of them. Furthermore, this holds for torture. Assuming this argument holds (and I have yet to see a reason it shouldn't) then the scenario at the bottom is a Good Thing.
However, if you consider it in terms of rewards adding up, then surely the trillions etc. copies of the society at the end receive enough utility to outweigh the disutility of the few copies getting tortured?
Would you do me a favor and reread my comment?
The point wasn't that that was what you were trying to say, my point was that that was the most natural way to interpret what you were actually saying. Hence my comment at the end - you seem to be trying to raise a dustmotes in the eye versus torture argument, but you never actually discuss the dustmotes. Your comments invest a heavy amount of weight in the torture aspect of the argument, and remember that's already an emotionally charged concept, and then you never discuss the utility that actually comes out of it. Allow me to elaborate:
Suppose, we vivisect an entire universe full of simulated people. If there are enough people it might not matter, the utility might outweigh the costs.
That's what your thread is right now. The reader is left baffled as to what utility you could possible be referring to; are we referring to the utility some lunatic gets from knowing that there are people getting vivisected? And are we disregarding the disutility of the people getting vivisected? Why is their disutility lower because there are more people in the universe? Does the absolute important of a single person decrease relative to absolute number of people?
You don't discuss the medical knowledge, or whatever utility everybody else is getting, from these vivisections.
Are you familiar with the thought experiment I'm referring to with dustmotes versus torture?
Ah.
I though I made that clear:
I think I may have aid too much emphasis on the infinitesmally small Reality of the victims, as opposed to the Reality of the citizens.
I'm puzzled at to why they should matter less.
Because they are less.
Retracted last comment because I realized I was misreading what you were saying.
Let me approach this from another direction:
You're basically supposing that 1/N odds of being tortured is morally equivalent to 1/N odds of being tortured with an implicit guarantee that somebody is going to get tortured. I think it is consistent to regard that 1/N for some sufficiently large N odds of me being tortured is less important than 1/N people actually being tortured.
If you create a precise duplicate of the universe in a simulation, I don't regard that we have gained anything; I consider that two instances of indistinguishable utility aren't cumulative. If you create a precise duplicate of me in a simulation and then torture that duplicate, utility decreases.
This may seem to be favoring "average" utility, but I think the distinction is in the fact that torturing an entity represents, not lower utility, but disutility; because I regard a duplicate universe as adding no utility, the negative utility shows up as a net loss.
I'd be hard-pressed to argue about the "indistinguishability" part, though I can sketch where the argument would lay; because utility exists as a product of the mind, and duplicate minds are identical from an internal perspective, an additional indistinguishable mind doesn't add anything. Of course, this argument may require buying into the anthropic perspective.
I'm basically assuming this reality-fluid stuff is legit for the purposes of this post. I included the most common argument in it's favor (the probability argument) but I'm not setting out to defend it, I'm just exploring the consequences.
Why?
If you're in a simulation right now, how would you feel about those running the machine simulating you? Do you grant them moral sanction to do whatever they like with you, because you're less than them?
I mean, maybe you're here as a representative of the people running the machine simulating me. I'm not sure I like where your train of thought is going, in that case.
Honestly, I would have upvoted just for this bit.