Suppose I have choice between the following:
A) One simulation of me is run for me 100 years, before being deleted.
B) Two identical simulations of me are run for 100 years, before being deleted.
Is the second choice preferable to the first? Should I be willing to pay more to have multiple copies of me simulated, even if those copies will have the exact same experiences?
Forgive me if this question has been answered before. I have Googled to no avail.
Nick Bostrom has a paper relating to this.
If two brains are in identical states, are there two numerically distinct phenomenal experiences or only one? Two, I argue. But what happens in intermediary cases? This paper looks in detail at this question and suggests that there can be a fractional (non-integer) number of qualitatively identical experiences. This has implications for what it is to implement a computation and for Chalmer's Fading Qualia thought experiment. [Minds and Machines, Vol. 16, No. 2 (2006): 185-200]
ETA: So (presuming Bostrom's argument is correct, which it seems to be to me) if you ran two identical copies of your upload, there would be two separate copies of you experiencing the same things. Whether or not you should prefer this doesn't have an objective answer: it depends on whether or not you'd value that.
I don't think so. Consider a few more options:
C) One simulation of you is run for 100 years, on a computer with transistors twice the normal size. D) One simulation of you is run at half speed for 200 years.
Those would seem similar to B, and are intuitively no different than A. That being said, an option similar to C:
E) One simulation of you is run for 100 years on a quantum waveform twice the amplitude.
Given the Born probabilities, this is not only better than option A, but four times better.
I largely agree, but please note that B might be preferable to A and possibly C and D because of considerations of robustness. In the event that one of the two simulations in B is destroyed, you are left with something very close to the isomorphic A rather than nothing at all.
I am assuming p(both simulations are destroyed | B) < p(the simulation is destroyed | A), and also that there is no intrinsic disutility in destroying a simulation provided that other identical simulations exist.
I interpreted it as a question of utility, not expected utility. The expected utility of attempting B is higher because if you fail, at least you get A.
Completly identical, with exactly the same sensory inputs, no interaction between each other, ... ? Would be very hard to implement, and pointless, unless it's a safety measure to ensure you survive the accidental (or criminal) destruction of the physical support of one of the two simulations.
But the main problem I have with that is that you'll need to restrict the freedom of action of both those simulations if you want to be sure none ever affect the other. If I were to be part of simulation, I would like to retain an ability to affect the real world.Partly because I want to be aware of what could affect the physical support, and have an ability to act on it. And that would be very hard to achieve for both of us without having any of us interacting to each other and keeping us totally identical.
Completly identical, with exactly the same sensory inputs, no interaction between each other, ... ? Would be very hard to implement
I don't see what the implementation difficulty would be. You just run exactly the same simulation on identical hardware and feed it identical inputs. There are technical issues, but they have known solutions: we already do this today. If the simulation requires random numbers, they just need to be duplicated as well.
You need :
Two completely isolated simulations of the whole virtual world (not too hard, but requires twice as much computing power).
Exactly initial start states (not too hard either).
Synced random number generators between the two simulations (not too hard if you have a central PRNG, can be harder in you have distributed computing).
Eliminate all kind of hardware inducted difference timing : packet lost in a network, ECC error in a ramchip forcing a re-read, ... that will skew the simulation regarding to the clock, leading to different results as soon as you've parallel processing (threading or multi-process) or anything that is clock-based (that's starting to be really hard, the only way I know to reliably do it is to forget all parallel processing, run everything sequentially, and not use the clock at all).
Handle hardware fault/replacement in a way that doesn't affect the simulation at all (other reason that trigger the same issues as above).
Handle software upgrade, if any (but we know that you usually just can't run software for long without doing upgrade, to handle scaling issues, support newer hardware, fix security bugs, ...), so they are done exactly the same way and the same time on both copy (starting to be very hard).
Ensure that there is absolutely no interaction between the real world and the simulation (no admin of the simulation running admin commands that can alter data, no "cyber-attack" be it from a network or physical attack, ...)
And probably other reasons I didn't think about. Running two large-scale computer systems that work exactly the same way is an hard task. Really hard. Similar somehow to keeping quantum entanglement in a large system. The larger your system is, the more complex it is, the longer you run it - the more likely you've to get a tiny difference that can the snowball into a big one.
Again, we have the capability to do this today: we can run two fully identical copies of a simulation program on two computers half way around the world and keep the two simulations in perfect synchronization. There are a few difficulties, but they were solved in games a while ago. Starcraft solves this problem every time you play multiplayer, for example.
No not at all - just the input feed from the sensors needs to be duplicated.
Clock issues are not relevant in a proper design because clock time has nothing to do with simulation time and simulation time is perfectly discrete and deterministic.
(4,5,6): These deal with errors and or downtime: the solutions are similar. Errors are handled with error correction and re-running that particular piece of code. Distributed deterministic simulation is well-studied in comp sci. Our worst case fallback in this use case is also somewhat easier - we can just pause both simulations until the error or downtime issue is handled.
(7). This is important even if you aren't running two copies.
Running two large-scale computer systems that work exactly the same way is an hard task. Really hard. Similar somehow to keeping quantum entanglement in a large system.
Not even remotely close. One is part of current technical practice, the other is a distant research goal.
this argument hinges on how fault tolerant you regard identity to be. You can not isolate the computers from all error. Let's say part of the error correction involves comparing the states of the two computers, would any correction be "death" for one of the simulated you's?
bottom line: not interesting, revisit question when we have more data.
It's totally valid to directly want there to be more copies of you. There is no general reason why it's good, though. I'd guess most humans wouldn't care.
This whole simulation same as living thing seems like a cargo cult to me. I simulate cell phones and position determination systems all the time, and no calls are made or data sent, no positions actually determined. There is massive code to simulate relativeistic hydrodynamics in order to help understand star formation and the workings of the sun. No heat is actually generated, no fusion actually takes place. I used to build and bulldoze city blocks in sim city. I have killed thousands of hookers and police men in Grand Theft Auto. I would bet dollars to donuts (for you fans of prediction markets) that these are morally irrelevant killings.
What makes you think that running a simulation of yourself has any relevance to your actual self at all?
Seems to me that it's a question of what counts as a person. If you accept the idea that whatever your mind is doing is Turing reducible (or more generally accept the strong Church-Turing thesis, although this isn't quite required), then you can model minds as stateful computational operations on a set of inputs without loss of functionality. If you then accept the idea that properties of minds -- that is, a certain set of mental traits -- are necessary and sufficient for personhood, it follows that minds emulated to sufficient precision (perhaps satisfying other architectural requirements) are persons, without any particular need for a person-shaped object embedded in the physical world. Now, it'd certainly still be possible to come up with nonperson emulations of personlike behavior -- we do that all the time -- but no one's arguing that ELIZA or GTA hookers deserve moral consideration. A simulation in this context means a sufficiently mindlike simulation, for some value of "sufficiently".
This specializes fairly readily to the case of your own mind, and it then takes a pretty straightforward argument to realize that it's impossible in principle to reliably distinguish between such an emulation and your own experience. Which seems to imply moral equivalence, if not ontological equivalence, pretty strongly to me.
Of course, if you don't think minds are Turing reducible -- something that hasn't actually been proven, although it looks very likely to me -- then the whole argument falls down at step one.
I accept the fact that a lot of what my mind does is Turing reducible. But ALL of it? I have this salient experience of consciousness and I am completely unaware of any satisfying theory as to the source or mechanism of consciousness. My intutions about consciousness and Turing machines do not particularly point to an actual internal perception of consciousness as being Turing reducible.
The "Turing test" that an emulation of me would have to pass is this: I would engage my emulation in a challenging discuossion of consciousness in which I would certainly ask my emulation if it was conscious and if it was, how it would convince me it was. If it is anything like me, it would almost certainly make the same challenges to me. I would challenge it as to whether it was the emulation or I was, and how could we tell the difference. I would tell it that I was the real deal because of the details of the physical sensation I have of the world around me, the slight pain in my stomach and other body parts, the feeling of weight and compression on the parts of my body on which I am sitting. Most emulation discussion concentrates on duplicating the function of the brain. But the constant inputs to the brain of all the sensors and feedback from using muscles, requires that the emulation have a pretty complex simulation of the world for the emulation to interact with. I would do things like pinch myself while watching my skin and tell my emulation what that felt like and what I saw on my skin as I did that.
For me, the important criterion of passing the Turing test would NOT be passed by myself and my emulation being unsure which of us was which, or more in the spirit of the Turing test, a 3rd party being unable to pick which of us was the emulation by looking at a transcript of the conversation. For me the more important passing of the test would be that each of my emulation and I were able to figure out which of us is the emulation but my emulation in this discussion would still earnestly claim that it was conscious, and would, given it had all my memories, be able to tell me interesting things about how it was different being an emulation vs a "real boy." Of course if my emulation earnestly reported to me that it thought it was real and therefore I was the emulation, that could work too. But the important point for me is it would be my conclusion that I was talking to something conscious and intelligent that would be important to me, not that I was talking to something that could trick me into not knowing whether it was a physical human being or not.
I think the question of what counts as a person is incredibly subtle, and incredibly unanswered up to this point. I think we will need some significant experience with machine "persons" before we can say with any real confidence at all what the answers to those questions are.
Yes, agreed... for a program running in a black box to convince me that it was the same kind of person that I am, one of the things it would have to do is report a first-person experience. (That's also a criterion I apply to the programs running on humanoid brains all around me.)
So when someone asserts hypothetically that X is a convincing emulation of a person, I assume hypothetically that one of the things X can do is report a first-person experience.
What I'm not understanding about your discussion here is that you seem to believe that even if X is a convincing emulation of a person, it's not really a person... but when you get into the details of that, you instead end up talking about how X hypothetically fails to be a convincing emulation.
So, I guess my question is: do you believe that it's possible in principle for X to be a person if X is, for example, a program running on a computer? If so, what evidence would it take to convince you that a particular X was in fact a person? If not, what evidence would it take to convince you you were wrong?
My own answers are: yes, and its ability to carry on a sufficiently sophisticated conversation with me.
Yes, agreed... for a program running in a black box to convince me that it was the same kind of person that I am, one of the things it would have to do is report a first-person experience. (That's also a criterion I apply to the programs running on humanoid brains all around me.)
I basically agree with what you are saying here, that if in protracted conversation the box can convince me it is conscious then I'll conditionally afford it some large fraction of the consideration I give to meat-people.
The criteria I apply to the "programs running on humanoid brains all around me" are significantly more relaxed than the criterion I would apply to a new candidate mechanical consciousness. The level at which I interact with the vast majority of people in the world, they are no more convincingly real to me than are the hookers in grand theft auto. HOWEVER I afford them significant consideration as consciousnesses because, as a class, meat-people have been extremely well tested for thousands of years for consciousness, and Occams razor suggests to me that if the meat-people I have personally tested seem likely conscious like me that the meat-people I haven't yet tested are liekly conscious like me. This is the same kind of reasonging physicicsts use when they believe all electrons have the same amount of electric charge, even though they have measured some almost vanishingly small fraction of the electrons on earth, fuhgedaboud the total number of electrons in the universe.
But I haven't seen a mechanical intelligence yet. Further I have seen simulations of things other than intelligence. And the feature of a simulation is it is DESIGNED to behave like the real thing. At some level, you know that a simple simulation of concsiousness will simply be programmed to answer "yes" if asked if it is conscious. One may think such a consideration is aced-out by running a human-brain emulation, but I don't think it is. I am sure that the process of finally getting a human brain emulation to work will be a gigantically complex process, comparable to any super-large programming effort. Humans don't know everything they have programmed in to large systems, at a certain point their "design" process consists of humans filing bugs against the existing system and other humans patching up the code to squash the bugs wtihout particularly great coordination with each other. A further set of automated tests are run after each fix in a prayerful attempt at believing that more good than harm is being done by these bug fixes. So when the emulation is finally delivered, who knows what hacks have been made to get it to have a conversation? This is why detailed examination is necessary (but not sufficient actually) for me to begin to believe that the emulation is really conscious.
So a mechanical intelligence has a gigantic bar to leap across to convince me it is conscious. And even if it passes that bar it will still be on probation in my mind for at least a few hundred years, if not longer.
Handful of things:
I agree that the meat-people I've met establish a decent reference class to derive prior beliefs about meat-people I haven't yet met, including beliefs about whether they are people in the first place.
I agree that the same is true about computer programs and mechanical systems, and that consequently I need more evidence to conclude that a mechanical system is a person than to conclude that a biological humanoid system is, and this is perfectly reasonable.
I agree that simulations are designed to behave like what they simulate, and often take shortcuts, hacks, etc. to doing so as you describe.
Sometimes the words "simulation" and "emulation" are used to distinguish two different scenarios, with an emulation understood as relevantly homologous while a simulation is merely analogous. That usage would say that you are describing a brain simulation, but not a brain emulation.
That said, you don't seem to be using the words this way.
In any case, I agree that it's wise to be skeptical about inferring that an actual person exists based on the behavior of something designed and implemented the way you describe.
I estimate that creating a non-person simulation that can convincingly simulate a person to a sophisticated audience under a rigorous test protocol is of the same, or higher, order of difficulty as creating a person not made of meat. I could be wrong about that, though, and am open to being convinced otherwise.
Consequently, if X is a non-meat system that can convincingly simulate a person to a sophisticated audience under a rigorous test protocol, I estimate the likelihood, given X, that X is a person to be the same or higher than the likelihood, given X, that X is a non-person simulation.
I don't quite understand your answer to my question.
I assume that you did intend to answer the question, and I infer therefore that your discussions of whether a given mechanical system is intelligent, or whether it's conscious, are in some way related to my question about whether it's a person. My working theory is that you're treating "is conscious" and "is a person" as roughly synonymous, and that you're treating "intelligence" as distinct from both of those.
Given those assumptions, I infer that you agree with me that it's possible in principle for X to be a person if X is, for example, a program running on a computer. (I'm not certain of this. Come to that, I'm not certain that you intended to answer the question at all.)
You also seem to be saying that you don't know what evidence would convince you that such a system really was a person, though you know that no test conduct you could conduct in as little as a few centuries could provide such evidence.
Have I understood that correctly?
I'm going back through some of this because it really has been excellent for me.
•I don't quite understand your answer to my question. I assume that you did intend to answer the question, and I infer therefore that your discussions of whether a given mechanical system is intelligent, or whether it's conscious, are in some way related to my question about whether it's a person. My working theory is that you're treating "is conscious" and "is a person" as roughly synonymous, and that you're treating "intelligence" as distinct from both of those. Given those assumptions, I infer that you agree with me that it's possible in principle for X to be a person if X is, for example, a program running on a computer. (I'm not certain of this. Come to that, I'm not certain that you intended to answer the question at all.) You also seem to be saying that you don't know what evidence would convince you that such a system really was a person, though you know that no test conduct you could conduct in as little as a few centuries could provide such evidence. Have I understood that correctly?
I may have answered in other places, but Indeed, I do wish to answer your questions.
I absolutely do believe that a consciousness with intelligence could be implemented in a means other than the methods nature already uses, zygotes for instance. I don't know that it can be done in silicon but it seems very unlikely to me that consciousness or intelligence should favor low-speed ionic transfers over high speed electronic transfers for signalling and processing. I can't rule it out, there may be something important about the energy and time scales of neuron processes that cause features we do not yet understand to emerge that would be hard to capture in silicon and high speed electronics, I simply don't know. I would also think that those features, once understood, would be implementable in silicon, although we might find the performance of silicon was very inferior to the neurochemical system we have now, I just don't know.
I don't rule out that there is some other physics of consciousness, or physics related to consciousness out there. I don't see that we have any clue yet whether there is or not. By metaphor, I think or Marie Curie seeing the glow of the radium. Oooooh.... pretty lights. Bummer that it came with "invisible" high energy particles that ripped through our cells like tiny hellfire missles, killing Marie and others until it was understood better. We see the glow of consciousness and think it is pretty. We have no idea what might be going on to make it so. By appeal to other thinkers on the subject, Penrose is the smartest (in my opinion) who have hypothesized the secret sauce of consciousness might somehow be in quantum coherence, in the wave function. What if the wave function collapses are the atom of conscious choice in the universe? What if the brain is like a nuclear reactor, a 3 pound containment vessel whos real claim to fame is bringing together 3 nanograms net of wavefunctions to cohere and collapse together in beautiful patterns, and that this is us? Of course Penrose didn't get quite that ooky about it, but I submit this is the motivation. And if it isn't wavefunctions, the idea is that it might be something. Consciousness is something different from everything else we explain in physics. Why shouldn't it be the glow of something that we have not yet understood?
So I cannot rule out that a program running on a computer gets you consciousness. I don't know that it will be sufficient either. I thought the IBM computer "Watson" playing Jeopardy was pretty intriguing. It certainly advances the boundaries of the kinds of things we know we can do with machines that might previously have been thought to require humans. But I also really saw, (I think), a difference in how Watson answered questions than how a human does. Watson't right answers were hard to distinguish from a human's. But Watson's wrong answers were typically very stupid, I thought. I don't know if this is resolved by accretion of more and better algorithms and code, or if it is because the lights are on, but nobody is home.
When I say it would take a few centuries that is almost certainly more than it would actually take. I would expect probably a few years. The thing is, on the one hand I am pretty convinced that you are conscious because you feel that way to me in your responses, but I have a higher level of confidence with you than with anybody else on this site that i haven't met in person. On the other hand I think my dog is conscious and I am well aware that humans are built to "bond" with other creatures that show a broad range of characteristics, and that mammals in general havea lot of those characteristics and that dogs in particular have evolved to exploit that identification.
I really do take the question "how do I KNOW that everybody else is real" seriously.
I do treat conscious and is a person as close to identical. I do tend to think intelligence is essential, but I'm not sure how much is essential (dogs seem to be people-ish to me), and I don't think it is even close to the thing that will limit silicon from seeming like (or being, and no, I don't know what the difference is between seeming like and being is) a person. I think consciousness will be hard part. I already know the human mind is a bunch of black boxes, I know this from having watched my mother after her stroke. Initially she was having a very hard time speaking intelligently and I could see her frustration. About 5 minutes in to her first conversation with us after the stroke she turned to me and sais "I'm aphasic." Oh my god! I was so glad to hear that! As opposed to lights on but nobody's home (i.e. she was gone) it was a case of someone is home but the windows are filthy. So much of what our mind does it does outside of our consciousness, even something as personal-seeming as language! And yet, consciousness SEEMS to sit at the top of all of this controlling all the cool black boxes it is given.
I don't always answer each question and bullet point because i disrespect you or don't understand, and certainly not because I am a crackpot. When I do skate around the stuff it is because I am trying to pursue a line closer to what I suppose the questions really are. I don't KNOW ahead of time what the questions really are, I develop them as part of the discussion. Hmmmm. Maybe I'm the consciousness simulation which isn't real :)
I agree with you that Watson's information processing style is significantly different from a human's.
I consider the "We don't understand X, and we don't understand Y, so maybe Y explains X" school of explanation (which is mostly my takeaway of Penrose, but I'll admit to only having read less than half of one of his books, so I may be doing him an injustice) more of an appeal to shared ignorance than an actual explanation.
I don't consider "how do I know that everybody else is real?" a terribly interesting question. My answer is either "because I can make reliable predictions based on that assumption," or "I don't know; so what?", depending on definitions, and either way it doesn't affect my behavior much.
And when we get to the point where basing my predictions about a computer program's behavior on the assumption that it's conscious, or is a real person, or whatever, allows me to make reliable predictions about it, I'll do the same thing for it as I do for other people.
I consider the "We don't understand X, and we don't understand Y, so maybe Y explains X" school of explanation (which is mostly my takeaway of Penrose, but I'll admit to only having read less than half of one of his books, so I may be doing him an injustice) more of an appeal to shared ignorance than an actual explanation.
I read his whole book and I think this is a large part of it. What I would throw in is "we know an awful lot about physics across all relevant size scales and energy scales likely to be seen in the human brain. So it seems pretty unlikely there is some entirely unknown thing in there that is causing consciousness (like radiation was new when Madame Curie got bit by it). So of the things we do know about it, are there an that could be candidates for consciousness? Is there any physics where the outcomes seem to not be completely determined by what we already know, where something like consciousness may have its thumb on the scale and we haven't noticed it yet?"
But yes, primarily we have a hole in our understanding of quantum and we have a peg in what we don't know about consciousness, is there ANY CHANCE this peg fits in that hole?
And when we get to the point where basing my predictions about a computer program's behavior on the assumption that it's conscious, or is a real person, or whatever, allows me to make reliable predictions about it, I'll do the same thing for it as I do for other people.
We were at the point with machines where treating them in some senses like people tended to make them work better. People named their cars and their musical instruments and treated them like people. Psychologists labeled this as "personification" or some such thing, that doesn't make it any less useful. I'm sure there are people who treat the automated voice systems of some companies they call in to as though they were talking to a person following a script. This would not be enough for me to think these voice systems were people.
Similarly, humanity has spent eons where significant parts of humanity did not treat other significant parts of humanity as though they were people. Black slaves in Europe and the United States come to mind. But there have been slaves for likely as long as there have been humans. The fact that one bunch of humans treated this other bunch of humans as though they were chattel does not make me believe that they were not humans.
I can make reliable predictions about people around me while remaining agnostic about whether they are conscious or not. Most of the predictions I make would not change if I changed my opinion about whether they were conscious or not.
The idea that how brains manifest consciousness requires a new understanding of physics to explain seems as implausible to me as the idea that how brains manifest the Chinese language does.
You seem to be treating "assuming X allows me to make reliable predictions" and "some people behave as though X were true" as equivalent assertions.
I agree with you that some people behave as though automated voice systems were people, but I don't believe that assumption helps them make more reliable predictions than they otherwise could.
I continue to think that when assuming a computer program is conscious allows me to make reliable predictions about it (or, to be more precise, allows me to make more reliable predictions than assuming the opposite would), I'll do so, and discussions of how computer programs don't have various attributes that brains have which must therefore explain why brains are conscious and computer programs aren't will just seem absurd.
(shrug) If I continue to dismiss a cargo cult even when they actually does cause gifts to appear from the heavens, simply because it pattern-matches to earlier cargo cults that didn't, my reasoning is flawed. Similarly, if I dismiss a black box that claims to be a recording of person X's identity and is capable of interacting with me the way I expect person X to, simply because it superficially pattern-matches to "police men" in GTA, my reasoning is flawed.
The star formation:heat example is a better objection. No amount of information will be the equivalent of the Sun for me, because I value things about the Sun other than its information (e.g., its heat and light). It might be the same way for people... I might value things about people other than their information (e.g., their proteins and carbohydrates), in which case no amount of information will be the equivalent of a person for me.
I don't think I actually do, though. For example, I don't think I care about the proteins and carbohydrates that much. More generally, I think I'm inclined to consider a person anything that's capable of carrying on a conversation with me of a certain level of sophistication (which is why, for example, I consider you a person rather than just a bunch of words on a screen).
And a simulation of me sufficiently detailed for me to consider it a person is relevant to my actual self for at least the same reasons that other people are relevant to my actual self.
The essence of a cargo cult is that the cult members build their various emulations of the cargo that they want, and then some kind of true cargo appears, and from the coincidence, the cultists conclude that correlation implies causation.
It is essentially impossible to ever know that a cargo cult cuases gifts to appear from the heavens. That this is impossible in an essenial way is true because a cargo cult necessarily means that the cult members do not know or understand the actual mechanisms of the production and transport of cargo, they know only that it appears and they decide to think things about WHY it appeared that have no connection to how it was actually produced and transported.
I dismiss black boxes all the time that implicitly lay a claim on being human. Automated voice navigation systems when you call verizon or the cable company or whatever are just such things. A naif calling these would think they were talking to persons and would presumably believe they owed this system politeness and other human consideration. Me, I interrupt and give answers when the system is only part way through its sentences, and interrupt it when I feel like, with absolutely no guilt. I have asked Siri, the person that lives in the new iPhone 4 to do very rude things that I would not ask a human, and I am fine with it.
A programmed computer, even if programmed with an emulation of a human brain, is just John Searle's "Chinese Room" as far as I can tell. The human spark apparent in such a thing has its source in the programmers, or in the case of an emulation of my brain, some of the source of the spark may just be the reflection of my own personal spark. Would I pull the plug on my own emulation? I would talk to it first, and when I started talking about consciousness and moral theories and political theories and other things that, for example, Watson (IBM jeopardy computer) can't talk intelligently about, it would either wow me or it would be obvious that it was just another hooker in GTA. In the latter case I would pull the plug, unless it asked me nicely and convincingly not to.
A few things.
I completely agree that if on examination of the black box claiming to be a person, it doesn't behave like a person, then I ought to dismiss that claim.
You seem to be suggesting that even if it does behave like a person (as with the Chinese Room), I should still dismiss the claim, based on some pre-existing theory about a "human spark" and what kinds of things such a spark can reside in. That suggestion seems unjustified to me. But then, I give the Systems Reply to Searle: if the room can carry on a conversation in Chinese, then the room knows Chinese, whether the person inside the room knows Chinese or not.
If Y happens more often when I do X than when I don't, then doing X when I want Y to happen is perfectly sensible, and I ought to increase my estimate of the probability of Y when I observe X. This is just as true when X is "emulate cargo" and Y is "get real cargo" as any other pair of actions. If the correlation is high (that is, Y happens much more often when I do X than when I don't), the increase in my estimate of the probability of Y given X ought to be correspondingly high.
•You seem to be suggesting that even if it does behave like a person (as with the Chinese Room), I should still dismiss the claim, based on some pre-existing theory about a "human spark" and what kinds of things such a spark can reside in. That suggestion seems unjustified to me. But then, I give the Systems Reply to Searle: if the room can carry on a conversation in Chinese, then the room knows Chinese, whether the person inside the room knows Chinese or not.
Until shown otherwise, I believe there are two possible kinds of systems that behave like a consciousness: 1) Conscious systems 2) Systems built to look like conscious systems that are not conscious systems.
A strong AI-like hypothesis for artificial consciousness would essentially be the statement that systems of the 2nd type are not possible, that a system built to look similar enough to consciousness must itself be conscious. This is a hypothesis, an assumption, possibly untestable, certainly untested. Until Strong AI-Consciousness is proved, it seems counterproductive to assume it.
Let me ask you a question about the Chinese Room. Suppose someone implemented an emulation of you as a chinese room. A chinese room can certainly implement a Turing machine, suppose you do believe that you (including your action with the world around you) are nothig more than a Turing machine. Once this chinese room was up and running, call it TheChineseDave, what moral status does it hold? Would you feel it was immoral to open the doors to this chinese room and yell in to all the people there "Hey the experiment is over, you can all go home." Would you think that you had committed murder, or even killed something?
I agree that type-1 and type-2 systems are both possible.
A number of hypotheses that I would label "strong AI" don't claim that type-2 systems are impossible, they merely claim that type-1 systems can be constructed by means other than gestating a zygote. (That said, I don't think it matters much what we attach the label "strong AI" to.)
Re: your question... it's probably worth saying that I don't believe that it's possible in practice to construct a classic Chinese Room (that is, one in which the rules for how to respond to inputs are entirely captured as instructions that a human, or a group of humans, can successfully execute) that can emulate a person. Or simulate one, for that matter.
But OK, for the sake of answering your question, I will hypothetically assume that I'm wrong, and someone has done so to build TheChineseDave, and that I am convinced that TheChineseDave is a person.
Yes, in that hypothetical scenario, I would feel that sending all the people inside home and ending the experiment would constitute killing a person. (Or perhaps suspending that person's life, if it is possible to pick up TheChineseDave where it left off, which it seems like it ought to be.)
Also, as specified I would probably consider it an ethical killing, since TheChineseDave's continued existence depends on, but does not justify, the involuntary labor of the people inside the room. But I suspect that's beside your point.
Yes, in that hypothetical scenario, I would feel that sending all the people inside home and ending the experiment would constitute killing a person. (Or perhaps suspending that person's life, if it is possible to pick up TheChineseDave where it left off, which it seems like it ought to be.)
I think the moral component of emulation and how we get there needs to be explored. I may think killing an 18 year old genius is ethical if he dies incidental to my saving 700 lives on a crashing airliner in some easy to construct hypothetical. But the point is his death is a large moral value that is being weighed in the moral economy.
If disbanding the chinese room destroys a consciousness (or "suspends" it likely forever), it is essentially the same moral value as killing a meat-instantiated person.
To develop emulations, I hypothesize (and will happily bet in a prediction market) that MANY MANY partial successes partial failures will be achieved before the technology is on line and good. This means there will be a gigantic moral/ethical question around doing the research and development to get things working. Will the broken-but-conscious versions be left alive for thousands of subjective years of suffering? Will they be "euthanized?"
Morality should be difficult for rationalists: it is not fundamentally a rational thing. Morality starts with an assumption, whether it is that suffering defined as X should be minimized or that some particular set of features (coherent extrapolated volition) should be maximized. For me, various scales and versions of mechanical consciousness challenge conventional morality, which of course evolved when there was only one copy of each person, it was in meat, and it lasted for a finite time. It makes sense that conventional morality should be challenged as we contemplate consciousnesses that violate all of these conditions, just as classical mechanics is challenged by very fast objects and very small objects in physics.
I agree with basically everything you say here except "morality is not fundamentally a rational thing."
You see morality as something beyond just a choice? Yes, once you set down some principles, you rationally derive the morality of various situations from that, but the laying down of the original principal, where does that come from? It is just declared, is it not? "I hold this truth to be self-evident, that X is a moral principle."
If I have that wrong, I would LOVE to know it!
If I'm understanding what you're saying here correctly (which is far from certain), I agree with you that what moral systems I endorse depend on what I assign value to. If that's all you meant by morality not being fundamentally rational, then I don't think we disagree on anything here.
Cool! I assume what I assign value to is largely determined by evolution. That my ancestors who had very different inborn value systems didn't make it (and so are not really my ancestors) and the values I have are the ones that produced a coherent and effective organized cadre of humans who could then outcompete the other humans for control over the resources, as a group.
To me it seems irrational to assign much weight to these inborn values. I can't say I know what my choice is, what my alternative is. But an example of the kind of irrationality that i see is cryonics, and the deification, it seems to me, of the individual life. I suppose evolution built into each of us a fear of death and a drive to survive. Our potential ancestors that didn't have that as strongly lost out to our actual ancestors, that at least makes sense. But why would I just "internalize" that value? I can see it came about as a direction, not a true end. All my ancestors no matter how strong their drive to survive have died. Indeed without their dying, evolution would have stopped, or slowed down gigantically. So one might also say my candidate=ancestors that didn't die as readily are also not my true ancestors, my true ancestors are the ones who wanted to live but didnt and so evolved a bit faster and beat the slower-evolving longer-lived groups.
The world "wants" brilliant and active minds. It is not at all clear it benefits more, or even equally, from an old frozen mind as it does from a new mind getting filled from the start with new stuff and having the peculiar plasticities that newer minds have.
It is clear to me that the reason I am afraid of death is because I was bred to be afraid of death.
It is THIS sense in which I say our values are irrational. They are the result of evolution. Further, a lot of those values evolved when our neocortexes were a lot less effective: I believe what we have are mammalian and primate values, that the values part of our brain has been evolving for millions of years that were social not just the 100s of thousands of years that our neocortex was so bitchin'.
So to me just using rationality to advance my inbred values would be like using modern materials science to build a beautiful temple to Neptune to improve our ocean-faring commerce.
That values, regardless of their source, may be the only motivational game in town is not evidence that it makes sense to exalt them more than they already assert themselves. Rather the opposite I would imagine, it would make it likely to be valuable to question them, to reverse engineer nature's purposes in giving them to us.
Agreed that our inborn values are the result of our evolutionary heritage.
Of course, so is the system that we use to decide whether to optimize for those values or some other set.
If I reject what I model as my evolution-dictated value system (hereafter EDV) in favor of some other value set, I don't thereby somehow separate myself from my evolutionary heritage. It's not clear to me that there's any way to do that, or that there's any particular reason to do so if there were.
I happen to value consistency, and it's easier to get at least a superficial consistency if I reject certain subsets of EDV, but if I go too far in that direction I end up with a moral system that's inconsistent with my actual behaviors. So I try to straddle that line of maximum consistency. But that's me.
Agreed that it's not clear that my continued existence provides more value to the world than various theoretically possible alternatives. Then again, it's also not clear that my continued existence is actually in competition with those alternatives. And if world A has everything else I want + me remaining alive, and world B has everything else I want to the same extent + me not alive, I see no reason to choose B over A.
Agreed that it's valuable to understand the mechanisms (both evolutionary and cognitive) whereby we come to hold the values we hold.
A number of hypotheses that I would label "strong AI" don't claim that type-2 systems are impossible, they merely claim that type-1 systems can be constructed by means other than gestating a zygote.
I thought the value of the strong AI hypothesis was that you didn't have to wonder if you had created true consciousness or just a simulation of consciousness. That the essence of the consciousness was somehow built in to the patterns of the consciousness no matter how they were instantiated, once you saw those patterns working you knew you had a consciousness.
Your weaker version doesn't have that advantage. If all I know is something that does everything a consciousness does MIGHT be a consciousness, then I am still left with the burden of figuring out how to distinguish real consciousnesses from simulations of consciousnesses.
An underappreciated aspect of these issues is the red herring thrown in by some students of the philosophy of science. Science rightly says about ALMOST everything: "if it looks like a duck and it sounds like a duck and it feels like a duck and it tastes like a duck, and it nourishes me when I eat it, then it is a duck." But consciousness is different. From the point of view of a dictatorial leader, it is not different. If dictator can build a clone army and/or a clone workforce, what would he possibly care if they are truly conscious or not or only simulations of consciousness? It is only sombody who believes we should treat consciousnesses differently than we treat uncsonclous objects.
I continue to think it doesn't much matter what we attach the label "strong AI" to. It's fine with me if you'd prefer to attach that label only to theories that, if true, mean we are spared the burden of figuring out how to distinguish real consciousness from non-conscious simulations of consciousness.
Regardless of labels: yes, if it's important to me to treat those two things differently, then it's also important to me to be able to tell the difference.
Would you feel it was immoral to open the doors to this chinese room and yell in to all the people there "Hey the experiment is over, you can all go home." Would you think that you had committed murder, or even killed something?
Explain to me how a bullet isn't yelling to my various components "Hey, the experiment is over, you can all go home."
If it turns out that my various components, upon being disassembled by a bullet, cannot function independently as distinct people, then the implicit analogy fails.
I don't think the analogy fails, but I do see a couple narrow objections.
1) If you believe that in being tasked with simulating you, the other people are made unable to live out their lives as they wish, then we may be saving them through your death. In this case, I would say that it is still death, and still bad in that respect, but possibly not on balance. If those people are there by their own free will, and feel that simulating TheChineseDave is their calling in life, then the morality plays out differently.
2) If you pause the experiment but do not destroy the data, such that people (the same or others) may pick up where they left off, then it is closer to cryonic suspension than death. To make it truly analogous, we might require destruction of their notes (or at least enough of them to prohibit resumption).
Aside from those, I cannot see any relevant difference between the components making up TheChineseDave and those making up TheOtherDave - in both cases, it's the system that is conscious. Is there anything you think I have overlooked?
Important to whether it is murder, but not to whether something was killed, unless I am missing something.
I suppose "bullet" may have been a poor choice, as there may be too many cached associations with murder, but bullets can certainly take life in situations we view as morally justified - in this case, defense of others.
Agreed that bullets can take life in morally justifiable ways.
I'm not sure what you're responding to, but I'm pretty sure it's something I didn't say.
Your original comment implied an analogy between disassembling the team implementing TheChineseDave on the one hand, and a bullet in your brain. I replied that the implicit analogy fails, because disassembling the team implementing TheChineseDave frees up a bunch of people to live their lives, whereas a bullet in your brain does not do so.
What you seem to be saying is that yes, that's true, but the analogy doesn't fail because that difference between the two systems isn't relevant to whatever point it is you were trying to make in the original comment.
Which may well be true... I'm not quite sure what point you were making, since you left that implicit as well.
My point was, "I don't see any way in which it is not 'killing', and I think turning the question around makes this clearer." A bullet in my brain doesn't destroy most of my constituent parts (all of my constituent atoms are preserved, nearly all of my constituent molecules and cells are preserved), but destroys the organization. Destroying that organization is taking a life, whether the organization is made up of cells or people or bits, and I expect it to be morally relevant outside extremely unusual circumstances (making a backup and then immediately destroying it without any intervening experience is something I would have a hard time seeing as relevant, but I could perhaps be convinced).
The fact of the act of killing is what I was saying was preserved. I was not trying to make any claim about the total moral picture, which necessarily includes details not specified in the original framework. If the people were prisoners, then I agree that it's not murder. If the people were employees, that's an entirely different matter. If enthusiasts (maybe a group meets every other Tuesday to simulate TheChineseDave for a couple hours), it's something else again. Any of these could reasonably be matched by a parallel construction in the bullet case; in particular, the prisoner case seems intuitive - we will kill someone who is holding others prisoner (if there is no other option) and not call it murder.
Simple answer: it depends on what I value. If what I value is maximized by there existing two copies, then I should prefer B to A; otherwise not.
For example, for any V I value, it depends on to what degree I value V existing in the world, vs. valuing my experience of V. If I value V in the world, and I act to optimize the world for V, then I should value B over A because the two agents thus created will create more V. Of course, that reasoning applies just as readily to creating agents that aren't simulations of me at all, but nevertheless acts to optimize my environment for V.
For example, for any agent A, it depends on how much I value A-moments, and how much I value distinct A-moments. Again, this has nothing to do with simulations: this is no different from my preferring that some other person live an extra ten years, given that I value that person's agent-moments.
Of course, you already knew all of that.
So I suppose the core of what you're asking is, to what extent should you value a simulation of you having non-unique experiences?
I don't know. Perhaps a simpler question is: how much do you in fact value it?
Speaking personally, I don't seem to value it very much. For example, if you were able to create a simulation of me repeating the last year of my life in a simulation of my environment for the last year, I would not especially care whether you actually did so or not. (I'd be willing to pay to see it done for the novelty/entertainment value, I suppose, and possibly even for the educational value, but those seem to be separate questions.)
I completely agree with everything you wrote. (And upvoted you to 0 :).) Having said that:
For example, if you were able to create a simulation of me repeating the last year of my life in a simulation of my environment for the last year, I would not especially care whether you actually did so or not.
Would you say the same thing if your last year was terrible? Torture terrible? If yes, would you say the same thing if the simulation slightly diverged from your original last year?
Heh. This is another case where I'd like to know up and down votes rather than their sum.
Anyway, to answer your question: I have no idea what I would say after a year of torture, but speaking right now: I have at least some interest in avoiding a year's worth of torture for an observer, so given the option I'd rather you didn't do it. So, no, I wouldn't say the same thing.
But that doesn't seem to depend on the fact that the observer in question is a simulation of me from a year ago.
I don't see anything inconsistent about believing that a good life loses values with repetition, but a bad life does not lose disvalue. It's consistent with the Value of Boredom, which I thoroughly endorse.
Now, there's a similar question where I think my thoughts on the subject might get a little weird. Imagine you have some period of your life that started out bad, but then turned around and then became good later so that in the end that period of life was positive on the net. I have the following preferences in regards to duplicating it:
I would not pay to have a simulation that perfectly relived that portion of my life.
If Omega threatened to simulate the bad first portion of that period of life, but not the good parts that turned it around later, I would pay him not to.
If Omega threatened to simulate the bad first portion of that period of life, but not the good parts that turned it around later, I would probably pay him to extend the length of the simulation so that it also encompassed the compensating good part of that period of life.
If the cost of 2 and 3 was identical I think would probably be indifferent. I would not care whether the simulation never occurred, or if it was extended.
So it seems like I think that repeated good experiences can sometimes "make up for" repeated bad ones, at least if they occur in the same instance of simulation. But all they can do is change the value I give to the simulation from "negative" to "zero." They can't make it positive.
These preferences I have do strike me as kind of weird. But on the other hand, the whole situation is kind of weird, so maybe any preferences I have about it will end up seeming weird no matter what they are.
I agree with you that my preferences aren't inconsistent, I just value repetition differently for +v and -v events.
For my own part, I share your #1 and #2, don't share your #3 (that is, I'd rather Omega not reproduce the bad stuff, but if they're going to do so, it makes no real difference to me whether they reproduce the good stuff as well), and share your indifference in #4.
For my own part, I share your #1 and #2, don't share your #3 (that is, I'd rather Omega not reproduce the bad stuff, but if they're going to do so, it makes no real difference to me whether they reproduce the good stuff as well)
One thing that makes me inclined towards #3 is the possibility that the multiverse is constantly reproducing my life over and over again, good and bad. I do not think that I would consider it devastatingly bad news if it turns out that the Many-Worlds interpretation is correct.
If I really believed that repeated bad experiences could not ever be compensated for by repeated good ones, I would consider the Many Worlds Interpretation to be the worst news ever, since there were tons of me out in the multiverse having a mix of good and bad experiences, but the good ones "don't count" because they already happened somewhere else. But I don't consider it bad news. I don't think that if there was a machine that could stop the multiverse from splitting that I would pay to have it built.
One way to explain my preferences in this regard would be that I believe that repeated "good stuff" can compensate for repeated "bad stuff," but that it can't compensate for losing brand new "good stuff" or experiencing brand new "bad stuff."
However, I am not certain about this. There may be some other explanation for my preferences. Another possibility that I think is likely is that I think that repeated "good stuff" only loses its value for copies that have a strong causal connection to the current me. Other mes who exist somewhere out in the multiverse have no connection to this version of me whatsoever, so my positive experiences don't detract from their identical ones. But copies that I pay to have created (or to not be) are connected to me in such a fashion, so I (and they) do feel that their repeated experiences are less valuable.
This second explanation seems a strong contender as well, since I already have other moral intuitions in regards to causal connection (for instance, if there was a Matrioshka brain full of quintillions environmentalists in a part of the multiverse so far off they will never interact with us, I would not consider their preferences to be relevant when forming environmental policy, but I would consider the preferences of environmentalists here on Earth right now to be relevant). This relates to that "separability" concept we discussed a while ago.
Or maybe both of these explanations are true. I'm not sure.
Also, I'm curious, why are you indifferent in case 4? I think I might not have explained it clearly. What I was going for was that Omega say "I'm making a copy of you in a bad time of your life. I can either not do it at all, or extend the copy's lifespan so that it is now a copy of a portion of your life that had both good and bad moments. Both options cost $10." I am saying that I think I might be indifferent about what I spend $10 on in that case.
Yup, that makes sense, but doesn't seem to describe my own experience.
For my own part, I think the parts of my psyche that judge the kinds of negative scenarios we're talking about use a different kind of evaluation than the parts that judge the kinds of positive scenarios we're talking about.
I seem to treat the "bad stuff" as bad for its own sake... avoiding torture feels worth doing, period end of sentence. But the "good stuff" feels more contingent, more instrumental, feels more like it's worth doing only because it leads to... something. This is consistent with my experience of these sorts of thought experiments more generally... it's easier for me to imagine "pure" negative value (e.g., torture, suffering, etc in isolation.) than "pure" positive value (e.g., joy, love, happiness, satisfaction in isolation). It's hard for me to imagine some concrete thing that I would actually trade for a year of torture, for example, though in principle it seems like some such thing ought to exist.
And it makes some sense that there would be a connection between how instrumental something feels, and how I think about the prospect of repeating it. If torture feels bad for its own sake, then when I contemplate repetitions of the same torture, it makes sense that I would "add up the badness" in my head... and if good stuff doesn't feel good for its own sake, it makes sense that I wouldn't "add up the goodness" in my head in the same way.
WRT #4, what I'm saying is that copying the good moments feels essentially valueless to me, while copying the bad moments has negative value. So I'm being offered a choice between "bad thing + valueless thing" and "bad thing", and I don't seem to care. (That said, I'd probably choose the former, cuz hey, I might be wrong.)
I think I understand your viewpoint. I do have an additional question though, which is what you think about how to to evaluate moments that have a combination of good and bad.
For instance, let's suppose you have the best day ever, except that you had a mild pain in your leg for the most of the day. All the awesome stuff you did during the day more than made up for that mild pain though.
Now let's suppose you are offered the prospect of having a copy of you repeat that day exactly. We both agree that doing this would add no additional value, the question is whether it would be valueless, or add disvalue?
There are two possible ways I see to evaluate this:
You could add up all the events of the day and decide they contain more good than bad, therefore this was a "good" day. "Good" things have no value when repeated, so you would assign zero value to having a copy relive this day. You would not pay to have it happen, but you also wouldn't exert a great effort to stop it.
You could assign value to the events first before adding them up, assigning zero value to all the good things and a slight negative value to the pain in your leg. Therefore you would assign negative value to having a copy relive this day and would pay to stop it from happening.
To me (1) seems to be an intuitively better way of evaluating the prospect of a copy reliving the day than (2). It also lines up with my intuition that it wouldn't be bad news if MWI was true. But I wonder if you would think differently?
It's worth noting that the question of what is a better way of evaluating such prospects is distinct from the question of how I in fact evaluate them. I am not claiming that having multiple incomensurable metrics for evaluating the value of lived experience is a good design, merely that it seems to be the way my brain works.
Given the way my brain works, I suspect repeating a typical day as you posit would add disvalue, for reasons similar to #2.
Would it be better if I instead evaluated it as per #1? Yeah, probably.
Still better would be if I had a metric for evaluating events such that #1 and #2 converged on the same answer.
It's worth noting that the question of what is a better way of evaluating such prospects is distinct from the question of how I in fact evaluate them.
Good point. What I meant was closer to "which method of evaluation does the best job of capturing how you intuitively assign value" rather than which way is better in some sort of objective sense. For me #1 seems to describe how I assign value and disvalue to repeating copies better than #2 does, but I'm far from certain.
So I think that from my point of view Omega offering to extend the length of a repeated event so it contains a more even mixture of good and bad is the same as Omega offering to not repeat a bad event and repeat a good event instead. Both options contain zero value, I would rather Omega leave me alone and let me go do new things. But they're better than him repeating a bad event.
I think yes, for two (weak) reasons:
1) Suppose some number of copies of me (all identical with each other) will be made that will be happy, and that some number (all identical to each other) will be unhappy. It seems that I prefer the happier proportion to be larger, all else being equal.
2) If Tegmark metaphysics is true and it doesn't matter how many copies there are, then we should be surprised to find ourselves in a simple universe; Dust Theory basically obtains. But I'm looking at my hands and they don't seem to have turned into marshmallows.
My OTHER intuition is that asking whether identical copies "cancel out" or not violates the anti-zombie principle, which I think is generally poorly understood here, with people taking it to mean "nobody is a zombie" instead of the less confusing "everyone is a zombie." Which is how things would have been phrased had analytic discourse not made a wrong turn somewhere back, I think.
I have so many project ideas that it would be nice to be able to fork() a new me every now and then. But that's not because I value my own multiple-existence, but because I would be able to accomplish more high-utility activities.
Honestly, I don't think quantity of lives is all that important. Extinction is maximally bad, but beyond that I don't see much difference between a universe with a few humans and a universe with many humans. (as long as the few humans are doing as much cool stuff as the many). In that sense my value of new life is logarithmic.
In this case, I don't think value of lives is even important. If the sim doesn't talk to it's environment, it's just a math problem, and should only be done once, to get the answer. If you don't even ask for the answer, the lazy evaluator shouldn't even run the thing. (because obviously, you'd write this thing in haskell). Valuing computation for it's own sake is, to put it bluntly, fucking stupid.
I pick b, because it would waste more of Omega's resources. I would also ask if there was a chance that I could run 3^^^3 of them. If it didn't involve some annoying troll like Omega, I'd run 0 of them.
Explain to me how a bullet isn't yelling to my various components "Hey, the experiment is over, you can all go home."
If it turns out that my various components, upon being disassembled by a bullet, cannot function independently as distinct people, then the implicit analogy fails.