= 27d567bc2d48d9f40e9ccc20ea370b15)
DRAFT:Ethical Zombies - A Post On Reality-Fluid
I came up with this after watching a science fiction film, which shall remain nameless due to spoilers, where the protagonist is briefly in a similar situation to the scenario at the end. I'm not sure how original it is, but I certainly don't recall seeing anything like it before.
Imagine, for simplicity, a purely selfish agent. Call it Alice. Alice is an expected utility maximizer, and she gains utility from eating cakes. Omega appears and offers her a deal - they will flip a fair coin, and give Alice three cakes if it comes up heads. If it comes up tails, they will take one cake away her stockpile. Alice runs the numbers, determines that the expected utility is positive, and accepts the deal. Just another day in the life of a perfectly truthful superintelligence offering inexplicable choices.
The next day, Omega returns. This time, they offer a slightly different deal - instead of flipping a coin, they will perfectly simulate Alice once. This copy will live out her life just as she would have done in reality - except that she will be given three cakes. The original Alice, however, receives nothing. She reasons that this is equivalent to the last deal, and accepts.
(If you disagree, consider the time between Omega starting the simulation and providing the cake. What subjective odds should she give for receiving cake?)
Imagine a second agent, Bob, who gets utility from Alice getting utility. One day, Omega show up and offers to flip a fair coin. If it comes up heads, they will give Alice - who knows nothing of this - three cakes. If it comes up tails, they will take one cake from her stockpile. He reasons as Alice did an accepts.
Guess what? The next day, Omega returns, offering to simulate Alice and give her you-know-what (hint: it's cakes.) Bob reasons just as Alice did in the second paragraph there and accepts the bargain.
Humans value each other's utility. Most notably, we value our lives, and we value each other not being tortured. If we simulate someone a billion times, and switch off one simulation, this is equivalent to risking their life at odds of 1:1,000,000,000. If we simulate someone and torture one of the simulations, this is equivalent to risking a one-in-a-billion chance of them being tortured. Such risks are often acceptable, if enough utility is gained by success. We often risk our own lives at worse odds.
If we simulate an entire society a trillion times, or 3^^^^^^3 times, or some similarly vast number, and then simulate something horrific - an individual's private harem or torture chamber or hunting ground - then the people in this simulation *are not real*. Their needs and desires are worth, not nothing, but far less then the merest whims of those who are Really Real. They are, in effect, zombies - not quite p-zombies, since they are conscious, but e-zombies - reasoning, intelligent beings that can talk and scream and beg for mercy but *do not matter*.
My mind rebels at the notion that such a thing might exist, even in theory, and yet ... if it were a similarly tiny *chance*, for similar reward, I would shut up and multiply and take it. This could be simply scope insensitivity, or some instinctual dislike of tribe members declaring themselves superior.
Well, there it is! The weirdest of Weirdtopias, I should think. Have I missed some obvious flaw? Have I made some sort of technical error? This is a draft, so criticisms will likely be encorporated into the final product (if indeed someone doesn't disprove it entirely.)
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
= f037147d6e6c911a85753b9abdedda8d){kind=logo}
Comments (116)
I don't currently accept the validity of this kind of anthropic reasoning (actually I am confused about anthropic reasoning in general). Is there an LW post where it is thoroughly defended?
Anthropic reasoning not working or not making sense in many cases is closer to being a standard position on LW (for example). The standard trick for making anthropic problems less confusing is to pose them as decision problems instead of as problems about probabilities. This way, when there appears to be no natural way of assigning probabilities (to instances of an agent) that's useful for understanding the situation, we are not forced to endlessly debate which way of assigning them anyway is "the right one".
You keep using that word. I don't think it means what you think it means.
Seriously, though, what do you think the flaw in the argument is, as presented in your quote?
I think I'm using "anthropic" in a way consistent with the end of the first paragraph of Fundamentals of kicking anthropic butt (to refer to situations in which agents get duplicated and/or there is some uncertainty about what agent an agent is). If there's a more appropriate word then I'd appreciate knowing what it is.
My first objection is already contained in Vladimir_Nesov's comment: it seems like in general anthropic problems should be phrased entirely as decision problems and not as problems involving the assignment of odds. For example, Sleeping Beauty can be turned into two decision problems: one in which Sleeping Beauty is trying to maximize the expected number of times she is right about the coin flip, and one in which Sleeping Beauty is trying to maximize the probability that she is right about the coin flip. In the first case, Sleeping Beauty's optimal strategy is to guess tails, whereas in the second case it doesn't matter what she guesses. In a problem where there's no anthropic funniness, there's no difference between trying to maximize the expected number of times you're right and trying to maximize the probability that you're right, but with anthropic funniness there is.
My second objection is that I don't understand how an agent could be convinced of the truth of a sufficiently bizarre premise. (I have the same issue with Pascal's mugging, torture vs. dust specks, and Newcomb's problem.) In this particular case, I don't understand how I could be convinced that another agent really has the capacity to perfectly simulate me. This seems like exactly the kind of thing that agents would be incentivized to lie about in order to trick me.
You may eventually obtain the capacity to perfectly simulate yourself, in which case you'll run into similar issues. I used Omega in a scenario a couple of years ago that's somewhat similar to the OP's, but really Omega is just a shortcut for establishing a "clean" scenario that's relatively free of distractions so we can concentrate on one specific problem at a time. There is a danger of using Omega to construct scenarios that have no real-world relevance, and that's something that we should keep in mind, but I think it's not the case in the examples you gave.
How would you characterize your issue with Pascal's mugging? The dilemma is not supposed to require being convinced of the truth of the proposition, just assigning it a non-zero probability.
Hmm. You're right. Upon reflection, I don't have a coherent rejection of Pascal's mugging yet.
Gotcha. Your posts have seemed pretty thoughtful so far so I was surprised by / curious about that comment. :)
If it helps you avoid fighting the hypothetical, Omega already knows what her answer will be, and has already acted on it.
Situation A: There are 3^^^^3 simulations of me, as well as myself. You come up to me and say "I'm going to torture forever one of our or your simulations, chosen randomly." Do I shrug and say, "well, whatever it 'certainly' won't be me" or do I scream in horror at the thought of you torturing a copy of me forever?
Situation B: There are me and 3^^^^3 other people in a rather large universe. You come up to me and say "I'm going to torture forever one of the people in this universe, chosen randomly." Do I shrug and say, "well, whatever, it 'certainly' won't be me" or do I scream in horror at the thought of you torturing someone forever?
What's the difference between these situations?
This is why I added Bob.
The difference is that a tiny risk of killing one, specific person is different to a certainty of killing any person, but not knowing who.
There's a jump between the paragraph beginning "Humans value each other's utility" and the next. Up to and including that paragraph, simulations are treated as equivalent to "real" people, but in the next, "the people in this simulation are not real", and are worth less than the Really Real. How do you get from the first part of the essay to the second?
I think the idea is meant to be that "one of many simulations" = "low probability" = "unimportant".
If so, I think this is simply a mistake. MugaSofer: you say that being killed in one of N simulations is just like having a 1/N chance of death. I guess you really mean 1/(N+1). Anyway, now Omega comes to you and says: unless you give me $100k (replace this with some sum that you could raise if necessary, but would be a hell of an imposition), I will simulate one copy of you and then stop simulating it at around the point of it's life you're currently at. Would you pay up? Would you pay up in the same way if the threat were "I'll flip a coin and kill you if it comes up heads"?
The right way to think about this sort of problem is still contentious, but I'm pretty sure that "make another copy of me and kill it" is not at all the same sort of outcome as "kill me with probability 1/2".
Now, suppose there are a trillion simulations of you. If you really believe what it says at the start of this article, then I think the following positions are open to you. (1) All these simulations matter about as much as any other person does. (2) All these simulations matter only about 10^-12 as much as any other person -- and so do I, here in the "real" world. Only if you abandon your belief that there's no relevant difference between simulated-you and real-you, do you have the option of saying that your simulations matter less than you do. In that case, maybe you can say that each of N simulations matters 1/N as much, though to me this feels like a bad choice.
No. He doubles my "reality", then halves it. This leaves me just as real as I was in the first place.
However, if he ran the simulation anyway, even if I paid up, then I think it does work out equivalent, because it's equivalent to creating the sim and then threatening to delete one of me if I didn't pay.
Does this answer your question?
I suspect I didn't made this as clear as I thought I had, but the term Really Real does not refer to people outside of simulations, it refers to people with vastly more simulations.
Well, as a draft comment, I don't think a trillion times, and 3^^^^^^3 times are conflatable in this context. There are simply too many arguments that apply to one and not the other.
For instance, a programmer can define 1 trillion unique societies. You could do this for instance, by having each society seeded from 12 variables with 10 levels each. You could then say that society seeded from 1,2,3,4,5,6,7,8,9,0,1,2 was the only society that was 1,2,3,4,5,6,7,8,9,0,1,2. I could generate a computer program which wrote out a textual description of each one. 1 trillion just isn't that large. For instance, there are well more than 1 trillion possible saved games in a simplistic game.
I don't even know if there are even 3^^^^^^3 potential possible states in the observable universe that can be physically distinguished from one another under current physics, but I would suspect not.
So I keep getting distracted by "A particular society out of 1 trillion societies may matter, but a particular society out of 3^^^^^^3 societies doesn't seem like it would matter any more than one of my atoms being less than one Planck length to the right would." but I'm not sure if that relates back to your point.
I suspect from some of the comments I'm getting that I didn't make this clear: copies are identical in this scenario. They receive the same inputs, make the same choices, they think and feel as one. They are, in short, one person (or civilization.) But one with vastly more reality-fluid (sometimes known as "measure") and thus, as far as I can tell, moral weight.
This is very thought provoking. Can you add clarity on your views on this point?
For instance, should I imply a "vastly" in front of moral weight as well as if there is a 1:1 correspondence or should I not do that?
Is this the only moral consideration you are considering on this tier? (I.E, there may be other moral considerations, but if this is the only "vast" one, it will probably outweigh all others.)
Does the arrangement of the copies reality fluid matter? Omega is usually thought of as a computer, so I am considering the file system. He might have 3 copies in 1 file for resilience, such as in a RAID array. Or he can have 3 copies that link to 3 files, such as in just having Sim001.exe and Sim002.exe and Sim003.exe having the exact same contents and being in the same folder. In both cases, the copies are identical. And if they are being run simultaneously and updated simultaneously, then the copies might not be able to tell which structure Omega was using. Which of these are you envisioning (or would it not matter? [Or do I not understand what a RAID array is?])
Some of these questions may be irrelevant, and if so, I apologize, I'm really am not sure I understand enough about your point to reply to it appropriately, and again, it does sound thought provoking.
Pretty much, yeah.
Well, I'm considering the torture's disutility, and the torturers' utility.
I'm not entirely sure I understand this question, but I don't think it should matter.
Is it worse to torture a virtual person running on redundant hardware (say 3 computers in lock-step, like the Space Shuttle used) whose permanent state (or backups) is stored on a RAID1 of disks instead of a virtual person running on a single CPU with one disk? Or even simpler; is it worse to torture a more massive person than a less massive person? Personally, I would say no.
Just like there's only one electron, I think there's only one of any particular thing, at least in the map. The territory may actually be weird and strange, but I don't have any evidence that redundant exact copies have as much moral weight as a single entity. I think that it's worse to torture 1 non-redundant person than it is to torture n-1 out of n exact copies of that person, for any n. That only applies if it's exactly the same simulation n-1 times. If those simulations start to diverge into n different persons, it starts to become as bad as torturing n different unique people. Eventually even those n-1 exact copies would diverge enough from the original to be considered copies of a different person with its own moral weight. My reasoning is just probabilistic in expected utility: It's worse for an agent to expect p(torture)=1 than p(torture)=n-1/n, and an identical agent can't distinguish between identical copies (including its environment) of itself.
As soon as you start torturing one of those identical agents, it ceases to be identical.
I guess the question from there is, does this produce a cascade of utility, as small divergences in the simulated universe produce slightly different agents for the other 6 billion people in the simulation, whose utility then exists independently?
That it is true, if unintuitive, that people gain moral worth the more "real" they get, is a position I have seen on LW, and the arguments do seem reasonable. (It is also rather more coherent when used in a Big Universe.) This post assumes that position, and includes a short version of the most common argument for that position.
Incidentally, I used to hold the position you describe; how do you deal with the fact that a tortured copy is, by definition, no longer "part" of the original?
Let N=3^^^^^^3, surely N nice world + another nice world is better than N nice worlds + a torture world. Why? Because another nice world is better than a torture world, and the prior existence of the N previous worlds shouldn't matter to that decision.
What about the probability of actually being in the torture world which is tiny 1/(N+1), the expected negative utility from this must surely be so small it can be neglected? Sure, but equally the expected utility of being the master of a torture world with probability 1/(N+1) can be neglected.
What this post tells me is that I'm still very very confused about reality fluid.
The torture world, in this case, is being used to satisfy the whims of the Niceworld's residents. Lots of Niceworld copies = lots of Reality = lots of utility. So goes the logic.
Since they are all the same, they can share a torture world.
Could this be an example of the noncentral fallacy? One big reason humans try to avoid death is because there is only one of each individual & once they die they are gone forever. If a simulation is made of me and gets turned off, there's still one of me (the original). In this alternate reality there's also the chance that Omega could always just make another new copy. I think the two situations are dissimilar enough that our standard intuitions can't be applied.
Well, presumably you would care less about that death depending on how "real" it was. If there's only one of you, you care as much as you do now (obviously,) if there's two sims you care half as much and so on.
First, I like the term e-zombie. It highlights the issue of "sim rights" vs "human rights" for me.
Second, I don't quite get the point you are trying to illustrate with this convoluted example. Is it that sims are intrinsically valueless or what? I don't see how this follows. Maybe some calculation is in order.
Not by a long shot. Pratchett has weirder ones in every chapter. For example, Only You Can Save Mankind.
It's that if sims that have less copies - are less "real" - are worth less, for the reasons presented above, then the whims of the many are worth more than the lives of the few, or the one.
Speaking as a Pratchett fan who's read pretty much everything he ever wrote, although it's been a while since I read OYCSM, I don't understand this. Ankh-Morpork is a weirdtopia, and so are a few other things he's written, (eg the "tradition as dead voting rights" bit from Johnny And The Dead,) but I don't recall anything like this. Maybe I'm just being an idiot, though.
What does it mean to simulate someone, and why should I value manipulation of a simulation?
How good does the simulation have to be before I value it; should I value a book in which I get cakes more? What about a fairly good simulation of the world, but the contents of my pantry are randomized each time I open it–should I value that simulation more if the expected number of cakes the next time the simulated me opens the pantry are higher?
I was assuming perfect quantum-level modelling of you and everything you interact with, acquired and sustained via magic. It makes things much simpler
As for your actual question ... I'm not sure. The sim would have to conscious, obviously, but the point at which it becomes "you" is ... unclear. It seems trivially true that a magical perfect simulation as above is "you", but an AI programmed to believe it's you is not. Beyond those two extremes ... it's tricky to say.
Of course, if utilities are additive, two almost-yous should be worth as much as one you with twice as much reality-fluid. So I guess humans can get away with ignoring the distinction between me and you, at least as long as they're using TDT or similar.
How close is a model that has an arbitrary number of cakes added?
I also say that no simulation has value to me if I am in a frame that knows they are a simulation. Likewise for quantum states that I don't manipulate.
Perfectly so before the cakes are added.
To be clear, are you actually asserting this or merely suggesting a possible resolution to the dilemma?
So you believe that it is irrelevant whether or not Omega' (a resident of the universe running a simulation) can create things of value to you but chooses not to? You have no preference for living in a world with constant physical laws?
It's a solution, but for it to apply to others they would have to share my values. What I'm saying is that there is no intrinsic value to me to the orientations of electrons representing a number which has a transformation function which results in a number which is perfectly analogous to me, or to any other person. Other people are permitted to value the integrity of those electrical orientations representing bits as they see fit.
So you, in fact, do not value simulations of yourself? Or anyone else, for that matter?
With the caveat that I am not a simulation for the purposes if that judgement. I care only about my layer and the layers which are upstream of (simulating) me, if any.
Well, obviously this post is not aimed at you, but I must admit I am curious as to why you hold this belief. What makes "downstream" sims unworthy of ethical consideration?
Maybe I've got a different concept of 'simulation'. I consider a simulation to be fully analogous to a sufficiently well-written computer program, and I don't believe that representations of numbers are morally comparable to living creatures, even if those numbers undergo transformations completely analogous to those creatures.
Why should I care if you calculate f(x) or f'(x), where x is the representation of the current state of the universe, f() is the standard model, and f'() is the model with all the cake?
Does that stay true if those representations are implemented in a highly distributed computer made out of organic cells?
I think I see where you're going with this, but your presentation actually leads somewhere else entirely; you discuss your point in the introduction and the conclusion, but you invest the majority of the weight of your argument into a place where your point is nowhere to be found.
Your argument, as presented, seems to be along these lines: Suppose there is one person, and one person is tortured. That's really important. Suppose there are a billion people, and one of them is tortured. That's not very important.
What I think you're getting at is more along these lines: Suppose there is one person, and one person is tortured; that is extremely important to that person. Suppose there are a billion copies of that one person, and one of them is tortured. Even a slight benefit arising from a decision leading to that torture may outweigh, by virtue of the fact that the benefit has been reproduced a billion less one times, the badness of the torture.
In other words, your presentation conflates the number of people with the importance of something bad happening to one of them. You don't discuss potential rewards at all; it's just, this torture is happening. Torture is equally bad regardless of the percentage of the population that is being tortured (given a specific number of people that are being tortured, I mean); we shouldn't care less about torture merely because there are more people who aren't being tortured. Whereas your actual point, as hinted at, is that, for some group that gains utility per individual as a result of the decision that results in that torture, the relative badness of that decision is dependent on the size of the group.
Or, in other words, you seem to be aiming for a discussion of dustmotes in eyes compared to torture of one person, but you're forgetting to actually discuss the dustmotes.