Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

L-zombies! (L-zombies?)

20 Post author: Benja 07 February 2014 06:30PM

Reply to: Benja2010's Self-modification is the correct justification for updateless decision theory; Wei Dai's Late great filter is not bad news

"P-zombie" is short for "philosophical zombie", but here I'm going to re-interpret it as standing for "physical philosophical zombie", and contrast it to what I call an "l-zombie", for "logical philosophical zombie".

A p-zombie is an ordinary human body with an ordinary human brain that does all the usual things that human brains do, such as the things that cause us to move our mouths and say "I think, therefore I am", but that isn't conscious. (The usual consensus on LW is that p-zombies can't exist, but some philosophers disagree.) The notion of p-zombie accepts that human behavior is produced by physical, computable processes, but imagines that these physical processes don't produce conscious experience without some additional epiphenomenal factor.

An l-zombie is a human being that could have existed, but doesn't: a Turing machine which, if anybody ever ran it, would compute that human's thought processes (and its interactions with a simulated environment); that would, if anybody ever ran it, compute the human saying "I think, therefore I am"; but that never gets run, and therefore isn't conscious. (If it's conscious anyway, it's not an l-zombie by this definition.) The notion of l-zombie accepts that human behavior is produced by computable processes, but supposes that these computational processes don't produce conscious experience without being physically instantiated.

Actually, there probably aren't any l-zombies: The way the evidence is pointing, it seems like we probably live in a spatially infinite universe where every physically possible human brain is instantiated somewhere, although some are instantiated less frequently than others; and if that's not true, there are the "bubble universes" arising from cosmological inflation, the branches of many-worlds quantum mechanics, and Tegmark's "level IV" multiverse of all mathematical structures, all suggesting again that all possible human brains are in fact instantiated. But (a) I don't think that even with all that evidence, we can be overwhelmingly certain that all brains are instantiated; and, more importantly actually, (b) I think that thinking about l-zombies can yield some useful insights into how to think about worlds where all humans exist, but some of them have more measure ("magical reality fluid") than others.

So I ask: Suppose that we do indeed live in a world with l-zombies, where only some of all mathematically possible humans exist physically, and only those that do have conscious experiences. How should someone living in such a world reason about their experiences, and how should they make decisions — keeping in mind that if they were an l-zombie, they would still say "I have conscious experiences, so clearly I can't be an l-zombie"?

If we can't update on our experiences to conclude that someone having these experiences must exist in the physical world, then we must of course conclude that we are almost certainly l-zombies: After all, if the physical universe isn't combinatorially large, the vast majority of mathematically possible conscious human experiences are not instantiated. You might argue that the universe you live in seems to run on relatively simple physical rules, so it should have high prior probability; but we haven't really figured out the exact rules of our universe, and although what we understand seems compatible with the hypothesis that there are simple underlying rules, that's not really proof that there are such underlying rules, if "the real universe has simple rules, but we are l-zombies living in some random simulation with a hodgepodge of rules (that isn't actually ran)" has the same prior probability; and worse, if you don't have all we do know about these rules loaded into your brain right now, you can't really verify that they make sense, since there is some mathematically possible simulation whose initial state has you remember seeing evidence that such simple rules exist, even if they don't; and much worse still, even if there are such simple rules, what evidence do you have that if these rules were actually executed, they would produce you? Only the fact that you, like, exist, but we're asking what happens if we don't let you update on that.

I find myself quite unwilling to accept this conclusion that I shouldn't update, in the world we're talking about. I mean, I actually have conscious experiences. I, like, feel them and stuff! Yes, true, my slightly altered alter ego would reason the same way, and it would be wrong; but I'm right...

...and that actually seems to offer a way out of the conundrum: Suppose that I decide to update on my experience. Then so will my alter ego, the l-zombie. This leads to a lot of l-zombies concluding "I think, therefore I am", and being wrong, and a lot of actual people concluding "I think, therefore I am", and being right. All the thoughts that are actually consciously experienced are, in fact, correct. This doesn't seem like such a terrible outcome. Therefore, I'm willing to provisionally endorse the reasoning "I think, therefore I am", and to endorse updating on the fact that I have conscious experiences to draw inferences about physical reality — taking into account the simulation argument, of course, and conditioning on living in a small universe, which is all I'm discussing in this post.

NB. There's still something quite uncomfortable about the idea that all of my behavior, including the fact that I say "I think therefore I am", is explained by the mathematical process, but actually being conscious requires some extra magical reality fluid. So I still feel confused, and using the word l-zombie in analogy to p-zombie is a way of highlighting that. But this line of reasoning still feels like progress. FWIW.

But if that's how we justify believing that we physically exist, that has some implications for how we should decide what to do. The argument is that nothing very bad happens if the l-zombies wrongly conclude that they actually exist. Mostly, that also seems to be true if they act on that belief: mostly, what l-zombies do doesn't seem to influence what happens in the real world, so if only things that actually happen are morally important, it doesn't seem to matter what the l-zombies decide to do. But there are exceptions.

Consider the counterfactual mugging: Accurate and trustworthy Omega appears to you and explains that it just has thrown a very biased coin that had only a 1/1000 chance of landing heads. As it turns out, this coin has in fact landed heads, and now Omega is offering you a choice: It can either (A) create a Friendly AI or (B) destroy humanity. Which would you like? There is a catch, though: Before it threw the coin, Omega made a prediction about what you would do if the coin fell heads (and it was able to make a confident prediction about what you would choose). If the coin had fallen tails, it would have created an FAI if it has predicted that you'd choose (B), and it would have destroyed humanity if it has predicted that you would choose (A). (If it hadn't been able to make a confident prediction about what you would choose, it would just have destroyed humanity outright.)

There is a clear argument that, if you expect to find yourself in a situation like this in the future, you would want to self-modify into somebody who would choose (B), since this gives humanity a much larger chance of survival. Thus, a decision theory stable under self-modification would answer (B). But if you update on the fact that you consciously experience Omega telling you that the coin landed heads, (A) would seem to be the better choice!

One way of looking at this is that if the coin falls tails, the l-zombie that is told the coin landed heads still exists mathematically, and this l-zombie now has the power to influence what happens in the real world. If the argument for updating was that nothing bad happens even though the l-zombies get it wrong, well, that argument breaks here. The mathematical process that is your mind doesn't have any evidence about whether the coin landed heads or tails, because as a mathematical object it exists in both possible worlds, and it has to make a decision in both worlds, and that decision affects humanity's future in both worlds.

Back in 2010, I wrote a post arguing that yes, you would want to self-modify into something that would choose (B), but that that was the only reason why you'd want to choose (B). Here's a variation on the above scenario that illustrates the point I was trying to make back then: Suppose that Omega tells you that it actually threw its coin a million years ago, and if it had fallen tails, it would have turned Alpha Centauri purple. Now throughout your history, the argument goes, you would never have had any motive to self-modify into something that chooses (B) in this particular scenario, because you've always known that Alpha Centauri isn't, in fact, purple.

But this argument assumes that you know you're not a l-zombie; if the coin had in fact fallen tails, you wouldn't exist as a conscious being, but you'd still exist as a mathematical decision-making process, and that process would be able to influence the real world, so you-the-decision-process can't reason that "I think, therefore I am, therefore the coin must have fallen heads, therefore I should choose (A)." Partly because of this, I now accept choosing (B) as the (most likely to be) correct choice even in that case. (The rest of my change in opinion has to do with all ways of making my earlier intuition formal getting into trouble in decision problems where you can influence whether you're brought into existence, but that's a topic for another post.)

However, should you feel cheerful while you're announcing your choice of (B), since with high (prior) probability, you've just saved humanity? That would lead to an actual conscious being feeling cheerful if the coin has landed heads and humanity is going to be destroyed, and an l-zombie computing, but not actually experiencing, cheerfulness if the coin has landed heads and humanity is going to be saved. Nothing good comes out of feeling cheerful, not even alignment of a conscious' being's map with the physical territory. So I think the correct thing is to choose (B), and to be deeply sad about it.

You may be asking why I should care what the right probabilities to assign or the right feelings to have are, since these don't seem to play any role in making decisions; sometimes you make your decisions as if updating on your conscious experience, but sometimes you don't, and you always get the right answer if you don't update in the first place. Indeed, I expect that the "correct" design for an AI is to fundamentally use (more precisely: approximate) updateless decision theory (though I also expect that probabilities updated on the AI's sensory input will be useful for many intermediate computations), and "I compute, therefore I am"-style reasoning will play no fundamental role in the AI. And I think the same is true for humans' decisions — the correct way to act is given by updateless reasoning. But as a human, I find myself unsatisfied by not being able to have a picture of what the physical world probably looks like. I may not need one to figure out how I should act; I still want one, not for instrumental reasons, but because I want one. In a small universe where most mathematically possible humans are l-zombies, the argument in this post seems to give me a justification to say "I think, therefore I am, therefore probably I either live in a simulation or what I've learned about the laws of physics describes how the real world works (even though there are many l-zombies who are thinking similar thoughts but are wrong about them)."

And because of this, even though I disagree with my 2010 post, I also still disagree with Wei Dai's 2010 post arguing that a late Great Filter is good news, which my own 2010 post was trying to argue against. Wei argued that if Omega gave you a choice between (A) destroying the world now and (B) having Omega destroy the world a million years ago (so that you are never instantiated as a conscious being, though your choice as an l-zombie still influences the real world), then you would choose (A), to give humanity at least the time it's had so far. Wei concluded that this means that if you learned that the Great Filter is in our future, rather than our past, that must be good news, since if you could choose where to place the filter, you should place it in the future. I now agree with Wei that (A) is the right choice, but I don't think that you should be happy about it. And similarly, I don't think you should be happy about news that tells you that the Great Filter is later than you might have expected.

Comments (71)

Comment author: paulfchristiano 08 February 2014 06:26:06AM *  6 points [-]

At this point I would be mildly surprised (though pleased) to discover that there was any metaphysically significant "reality fluid," mostly because it seems that even in a universe without any everyone would have all of the same evidence for its existence that we do. "What the physical world actually looks like" seems likely to be a matter of preferences (perhaps combined with considerations of accessibility, though if you are willing to grant that inaccessible regions of the universe really exist, you probably are reduced to preferences).

If your preferences are sensitive to your own physical existence (for example if you don't care what happens whenever you don't actually exist) then there is a straightforward conflict of values between the two copies in the mugging (which can be resolved by bargaining in the usual way). If not then as you say its not clear why whether the decision-maker physically exists or not ought to have any significance to the problem.

My bet as to the best approach (and even my best bet as to what we will retrospectively regard as the best approach) is to accept a probable or at least plausible dependence of "really exist" on preferences, and to live full and satisfying lives while dealing with things honestly. Doing otherwise seems too likely to create problems down the line.

That said, it looks quite likely that these considerations (fortunately?) don't affect what we ought to do. If you are able, and comfortable enough with that "likely," to avoid thinking about them then that might be an even better response, due to opportunity costs. And if we are in a world where these strange considerations do importantly affect what we ought to do, then it seems particularly hazardous to entertain philosophically unwarranted positions for psychological comfort . So probably I'd go with one of "avoid thinking about it" or "move away from ascribing metaphysical signifcance to existing," or at least moving towards a state of ignorance.

Comment author: ESRogs 09 February 2014 04:06:14AM 2 points [-]

At this point I would be mildly surprised (though pleased) to discover that there was any metaphysically significant "reality fluid," mostly because it seems that even in a universe without any everyone would have all of the same evidence for its existence that we do.

Is it a trivial mistake that I am thinking of quantum amplitudes as a sort of "magical reality fluid"? Isn't something like that needed for an explanation of why we live in a world where all our measurements seem to have followed the modulus squared law with respect to this number?

Comment author: ESRogs 09 February 2014 03:53:53AM *  0 points [-]

And if we are in a world where these strange considerations do importantly affect what we ought to do, then it seems particularly hazardous to entertain philosophically unwarranted positions for psychological comfort .

I find myself confused by this sentence. If the strange considerations affect what we ought to do, why are they unwarranted? Or do you mean that considering them will lead us to change what we think we ought to do, but the change will be unwarranted because so were the considerations?

(And just to make sure I'm following you, are the considerations we are talking about in this paragraph those related to the question of whether reality depends on preferences?)

Comment author: learnmethis 08 February 2014 02:34:10PM *  5 points [-]

It seems at best fairly confused to say that an L-zombie is wrong because of something it would do if it were run, simply because we evaluated what it would say or do against the situation where it didn't. Where you keep saying "is" and "concludes" and "being" you should be saying "would", "would conclude", and "would be", all of which is a gloss for "would X if it were run", and in the (counter-factual) world where the L-zombie "would" do those things it "would be running" and therefore "would be right". Being careful with your tenses here will go a long way.

Nonetheless I think the concept of an L-zombie is useful, if only to point out that computation matters. I can write a simple program that encapsulates all possible L-zombies (or rather would express them all, if it were run), yet we wouldn't consider that program to be those consciousnesses--a point well worth remembering in numerous examinations of the topic.

Comment author: ESRogs 07 February 2014 09:54:49PM 4 points [-]

Actually, there probably aren't any p-zombies

Should that (from the first line of the third paragraph) be l-zombies?

Comment author: Benja 07 February 2014 10:24:27PM 1 point [-]

Fixed, thanks!

Comment author: Coscott 07 February 2014 07:22:08PM 3 points [-]

Thank you for this post.

The concept of l-zombie is something that will be very helpful in allowing me to communicate my own beliefs about anthropics.

Comment author: trist 07 February 2014 09:19:32PM *  10 points [-]

Are cryopreserved humans l-zombies?

keeping in mind that if they were an l-zombie, they would still say "I have conscious experiences, so clearly I can't be an l-zombie"?

As well they should. For l-zombies to do anything they need to be run, whereupon they stop being l-zombies.

Comment author: Benja 07 February 2014 10:30:15PM 2 points [-]

For l-zombies to do anything they need to be run, whereupon they stop being l-zombies.

Omega doesn't necessarily need to run a conscious copy of Eliezer to be pretty sure that Eliezer would pay up in the counterfactual mugging; it could use other information about Eliezer, like Eliezer's comments on LW, the way that I just did. It should be possible to achieve pretty high confidence that way about what Eliezer-being-asked-about-a-counterfactual-mugging would do, even if that version of Eliezer should happen to be an l-zombie.

Comment author: ThisSpaceAvailable 08 February 2014 07:39:23AM 3 points [-]

But you see Eliezer's comments because a conscious copy of Eliezer has been run. If I'm figuring out what output a program "would" give "if" it were run, in what sense am I not running it? Suppose I have a program MaybeZombie, and I run a Turing Test with it as the Testee and you as the Tester. Every time you send a question to MaybeZombie, I figure out what MaybeZombie would say if it were run, and send that response back to you. Can I get MaybeZombie to pass a Turing Test, without ever running it?

Comment author: Benja 10 February 2014 11:05:04PM 2 points [-]

But you see Eliezer's comments because a conscious copy of Eliezer has been run.

A conscious copy of Eliezer that thought about what Eliezer would do when faced with that situation, not a conscious copy of Eliezer actually faced with that situation -- the latter Eliezer is still an l-zombie, if we live in a world with l-zombies.

Comment author: alexey 08 February 2014 06:59:37PM 1 point [-]

If I'm figuring out what output a program "would" give "if" it were run, in what sense am I not running it?

In the sense of not producing effects on the outside world actually running it would produce. E.g. given this program

int goodbye_world() {
launch_nuclear_missiles();
return 0;
}

I can conclude running it would launch missiles (assuming suitable implementation of the launch_nuclear_missiles function) and output 0 without actually launching the missiles.

Comment author: FeepingCreature 08 February 2014 08:01:33PM 5 points [-]

Within the domain that the program has run (your imagination) missiles have been launched.

Comment author: ThisSpaceAvailable 09 February 2014 06:08:18AM 2 points [-]

Benja defines an l-zombie as "a Turing machine which, if anybody ever ran it..." A Turing Machine can't launch nuclear missiles. A nuclear missile launcher can be hooked up to a Turing Machine, and launch nuclear missile on the condition that the Turing Machine reach some state, but the Turing Machine isn't launching the missiles, the nuclear missile launcher is.

Comment author: alexey 10 February 2014 01:13:30PM 0 points [-]

But I can still do static analysis of a Turing machine without running it. E.g. I can determine a T.M. would never terminate on given input in finite time.

Comment author: ThisSpaceAvailable 16 February 2014 07:02:24AM *  1 point [-]

But my point is that at some point, a "static analysis" becomes functionally equivalent to running it. If I do a "static analysis" to find out what the state of the Turing machine will be at each step, I will get exactly the same result (a sequence of states) that I would have gotten if I had run it for "real", and I will have to engage in computation that is, in some sense, equivalent to the computation that the program asks for.

Suppose I write a program that is short and simple enough that you can go through it and figure out in your head exactly what the computer will do at each line of code. In what sense has your mind not run the program, but a computer that executes the program has?

Imagine the following dialog:

Alice: "So, you've installed a javascript interpreter on your machine?"

Bob: "Nope."

Alice: "But I clicked on this javascript program, and I got exactly what I was supposed to get."

Bob: "Oh, that's because I've associated javascript source code files with a program that looks at javascript code, determines what the output would be if the program had been run, and outputs the result."

Alice: "So... you've a installed a javascript interpreter."

Bob: "No. I told you, it doesn't run the program, it just computes what the result of the program would be."

Alice: "But that's what a javascript interpreter is. It's a program that looks at source code, determines what the proper output is, and gives that output."

Bob: "Yes, but an interpreter does that by running the program. My program does it by doing a static analysis."

Alice: "So, what is the difference? For instance, if I write a program that adds two plus two, what is the difference?"

Bob: "An interpreter would calculate what 2+2 is. My program calculates what 2+2 would be, if my computer had calculated the sum. But it doesn't actually calculate the sum. It just does a static analysis of a program that would have calculated the sum."

I don't see how, outside of a rarefied philosophical context, Bob wouldn't be found to be stark raving mad. It seems to me that you're arguing for p-zombies (at least, behavioral zombies): one could build a machine that, given any input, tells you what the output would be if it were a conscious being. Such a machine would be indistinguishable from an actually conscious being, without actually being conscious.

Comment author: alexey 16 February 2014 07:11:17AM *  2 points [-]

But my point is that at some point, a "static analysis" becomes functionally equivalent to running it. If I do a "static analysis" to find out what the state of the Turing machine will be at each step, I will get exactly the same result (a sequence of states) that I would have gotten if I had run it for "real", and I will have to engage in computation that is, in some sense, equivalent to the computation that the program asks for.

Crucial words here are "at some point". And Benja's original comment (as I understand it) says precisely that Omega doesn't need to get to that point in order to find out with high confidence what Eliezer's reaction to counterfactual mugging would be.

Comment author: alexey 10 February 2014 01:33:03PM 0 points [-]

Suppose I've seen records of some inputs and outputs to a program: 1->2, 5->10, 100->200. In every case I am aware of it was given a number as input, it output the doubled number. I don't have the program's source and or ability to access the computer it's actually running on. I form a hypothesis: if this program received input 10000, it would output 20000. Am I running the program?

In this case: doubling program<->Eliezer, inputs<->comments and threads he is answering, outputs<->his replies.

Comment author: Lumifer 10 February 2014 07:00:46PM 0 points [-]

Am I running the program?

No, you've built your model of the program and you're running your own model.

Comment author: Coscott 07 February 2014 09:48:40PM *  1 point [-]

I do not think your complaint is valid.

I can ask questions like what is the googolth digit of pi, without calculating it.

Similarly, you can ask questions whether a Turing Machine mind would believe it has conscious experience without actually running it.

Comment author: satt 08 February 2014 05:48:22PM 2 points [-]

I can certainly ask, "were this Turing machine mind to be run, would it believe it were conscious?" But that doesn't give me licence to assert that "because this Turing machine would be conscious if it were run, it is conscious even though it has not run, is not running, and never will run".

A handheld calculator that's never switched on will never tell us the sum of 6 and 9, even if we're dead certain that there's nothing wrong with the calculator.

Comment author: Coscott 08 February 2014 05:59:14PM 1 point [-]

I am not trying to say it would be conscious without being run. (Although I believe it would)

I am trying to say that the computation as an abstract function has an output which is the sentence "I believe I am conscious."

Comment author: satt 08 February 2014 06:07:20PM 3 points [-]

I am not trying to say it would be conscious without being run. [...] I am trying to say that the computation as an abstract function has an output which is the sentence "I believe I am conscious."

Now I think I agree with you. (Because I think you're using "output" here in the counterfactual sense.)

But now these claims are too weak to invalidate what trist is saying. If we all agree that l-zombies, being the analogue of the calculator that's never switched on, never actually say anything (just as the calculator never actually calculates anything), then someone who's speaking can't be an l-zombie (just as a calculator that's telling us 6 + 9 = 15 must be switched on).

Comment author: Coscott 08 February 2014 06:13:36PM 1 point [-]

Okay, I guess I interpreted trist incorrectly.

I agree that in order for the L-zombie to do anything in this world, it must be run. (Although I am very open to the possibility that I am wrong about that and prediction without simulation is possible)

Comment author: somervta 07 February 2014 10:07:29PM 0 points [-]

Well, yes, it would if you ran it :D

Comment author: Coscott 07 February 2014 10:15:01PM 4 points [-]

Pi has a googolth digit even if we don't run the calculation. A decision procedure has an output even if we do not run it. We just do not know what it is. I do not see the problem.

Comment author: ThisSpaceAvailable 08 February 2014 07:34:08AM 2 points [-]

No, a decision procedure doesn't have an output if you don't run it. There is something that would be the output if you ran it. But if you ran it, it would not be an l-zombie.

Let's give this program a name. Call it MaybeZombie. Benja is saying "If MaybeZombie is an l-zombie, then MaybeZombie would say 'I have conscious experiences, so clearly I can't be an l-zombie' ". Benja did not say "If MaybeZombie is an l-zombie, then if MaybeZombie were run, MaybeZombie would say 'I have conscious experiences, so clearly I can't be an l-zombie' ".

There is no case in which a program can think "I have conscious experiences, so clearly I can't be an l-zombie" and be wrong. You're trying to argue that based on a mixed counterfactual. You're saying "I'm sitting here in Universe A, and I'm imagining Universe B where there is this program MaybeZombie that isn't run, and the me-in-Universe-A is imagining the me-in-Universe-B imagining a Universe C in which MaybeZombie is run. And now the me-in-Universe-A is observing that the me-in-Universe-B would conclude that MaybeZombie-in-Universe-C would say 'I have conscious experiences, so clearly I can't be an l-zombie'."

You're evaluating "Does MaybeZombie say 'I have conscious experiences, so clearly I can't be an l-zombie'?" in Universe C, but evaluating "Is MaybeZombie conscious?" in Universe B. You're concluding that MaybeZombie is "wrong" by mixing two different levels of counterfactuals. The analogy to pi is not appropriate, because the properties of pi don't change depending on whether we calculate it. The properties of MaybeZombie do depend on whether MaybeZombie is run.

It is perfectly valid for any mind to say "I have conscious experiences, so clearly I can't be an l-zombie". The statement "I can't be an l-zombie" clearly means "I can't be an l-zombie in this universe".

Comment author: wedrifid 08 February 2014 11:09:24AM 2 points [-]

No, a decision procedure doesn't have an output if you don't run it. There is something that would be the output if you ran it.

I'm not sure that is a particularly useful way to carve reality. At best it means that we need another word for the thing that Coscott is referring to as 'output' that we can use instead of the word output. The thing Coscott is talking about is a much more useful thing when analysing decision procedures than the thing you have defined 'output' to mean.

Comment author: IlyaShpitser 08 February 2014 11:49:58AM *  3 points [-]

That's just a potential outcome, pretty standard stuff:

http://www.stat.cmu.edu/~fienberg/Rubin/Rubin-JASA-05.pdf

"What would happen if hypothetically X were done" is one of the most common targets in statistical inference. That's a huge chunk of what Fisher/Neyman had done (originally in the context of agriculture: "what if we had given this fertilizer to this plot of land?") This is almost a hundred years ago.

Comment author: Coscott 08 February 2014 07:49:56AM 1 point [-]

I do not understand how the properties of MaybeZombie depend on whether or not MaybeZombie is run.

Comment author: Baughn 08 February 2014 04:28:07PM 3 points [-]

Because consciousness isn't a property of MaybeZombie, it's a property of the process of running it?

Comment author: Coscott 08 February 2014 06:07:36PM 1 point [-]

No, a decision procedure doesn't have an output if you don't run it.

This made me think that he was talking about the property of the output, so my misunderstanding was relative to that interpretation.

I personally think that consciousness is a property of MaybeZombie, and that L-zombies do not make sense, but the position I was trying to defend in this thread was that we can talk about the theoretical output of a function without actually running that funciton. (We might not be able to talk very much, since perhaps we can't know the output without running it)

Comment author: ThisSpaceAvailable 09 February 2014 06:27:59AM 0 points [-]

We are getting in a situation similar to the Ontological Argument for God, in which an argument gets bogged down in equivocation. The question becomes: what is a valid predicate of MaybeZombie? One could argue that there is a distinction to be made between such predicates as "the program has a Kolmogorov complexity of less than 3^^^3 bits" on the one hand, versus such predicates as "the program has been run" on the other. The former is an inherent property, while the latter is extrinsic to the program, and in some sense is not a property of the program itself. And yet, grammatically at least, "has been run" is the predicate of "the program" in the sentence "the program has been run". If "has said 'I must not be a zombie' " is not a valid predicate of MaybeZombie, then talking about whether MaybeZombie has said 'I must not be a zombie' is invalid. If one can meaningfully talk about whether MaybeZombie has said 'I must not be a zombie', then "has said 'I must not be a zombie' " is a valid predicate of MaybeZombie. Since this predicate is obviously false if MaybeZombie isn't run, and could be true if MaybeZombie is run, then this is a property of MaybeZombie that depends on whether MaybeZombie is run.

Comment author: DefectiveAlgorithm 08 February 2014 08:25:09AM 2 points [-]

If L-zombies have conscious experience (even when not being 'run'), does the concept even mean anything? Is there any difference, even in principle, between such an L-zombie and a 'real' person?

Comment author: CronoDAS 07 February 2014 08:27:11PM 2 points [-]

So, an L-zombie is a person that could exist, but doesn't?

Comment author: Coscott 07 February 2014 08:37:29PM 9 points [-]

That seems right to me, but you have to be careful with the word "could" especially, when you are talking outside the realm of choices and probability.

I think a good way to think about it is through an AI. An AI that is currently running is like a person. If we have code for an AI that we decide never to run, that AI is an L-zombie.

Comment author: Benja 07 February 2014 09:06:14PM 1 point [-]

(Agree with Coscott's comment.)

Comment author: lmm 07 February 2014 07:23:52PM 2 points [-]

If I've made a good moral choice and saved humanity I absolutely should feel happy about it - it means my feelings are aligned with my preferences (and even if the two were independent, other things being equal I'd rather be happy than sad).

I think the correct response to l-zombies is the same as that to p-zombies - that they really are impossible, and thus, contemplating their consequences is likely to lead us astray rather than yield valuable insights.

Comment author: Coscott 07 February 2014 07:33:25PM 2 points [-]

Yes, but I think most people think l-zombies and p-zombies are impossible for different reasons.

If you think "All minds exist in Tegmark 4 with different amounts of reality fluid, and therefore logical zombies do not exist," that seems categorically different from saying, "Philosophical zombies are a meaningless concept."

My current position is that L-zombies are just as meaningless as P-zombies, because there is no "existence" we are all just computations that would be thinking what we are thinking if run, but none of us are ever run.

Comment author: jaibot 07 February 2014 07:46:43PM 0 points [-]

I'm either extremely confused or parsing your post wrong: You appear to be saying that everyone is an L-zombie ("none of us are ever run").

Comment author: Coscott 07 February 2014 07:53:31PM *  3 points [-]

I am saying that.

I tried not to say everyone is an L-zombie, because Benja's definition said that L-zombies are not conscious.

I believe that L-zombie is meaningless like P-zombie, because something with the traits of L-zombie gets consciousness for free (just like with P-zombie).

Everyone is whatever the conscious version of an L-zombie is.

Comment author: Coscott 08 February 2014 01:57:14AM 1 point [-]

I wrote a post to try to put my neck out on this claim. It looks like you consider this an extreme claim, so please help me out and let me know what you think.

http://lesswrong.com/r/discussion/lw/jn2/preferences_without_existence/

Comment author: jaibot 08 February 2014 05:08:04AM 0 points [-]

I'm extremely confused in this whole area. My intuition doesn't like your claim, but I don't trust my intuition here.

Comment author: hairyfigment 07 February 2014 11:10:09PM -1 points [-]

I think most people think l-zombies and p-zombies are impossible

I doubt this holds even for LW regulars. (I feel confident it does not hold for all visitors.)

To my way of thinking, we have no reason to believe in a mathematical macrocosm. Naive Tegmark 4 gives false predictions. Now this could reflect our confusion about anthropic probabilities or "reality fluid", but the same confusion could make Tegmark 4 seem more attractive than it actually is. (I don't think that last point applies to MWI, which I more-or-less believe because it seems like the natural reading of a successful theory.)

Comment author: torekp 08 February 2014 03:35:02PM 0 points [-]

Why would l-zombies be impossible? Can you provide links to some arguments?

Comment author: lmm 09 February 2014 03:43:36PM 1 point [-]

Because the concept is incoherent. In particular there's no (empirical) way to distinguish between an l-zombie and a normal person - by definition, an l-zombie acts exactly like a normal person. Or put another way, I assert that you're already an l-zombie.

Comment author: torekp 09 February 2014 08:52:46PM *  0 points [-]

Thanks for explaining. However, an l-zombie would act exactly like a normal person (would actually be a normal person, even) if it physically existed somewhere and did stuff. But it doesn't, by definition of "l-zombie". I think trist and satt and learnmethis are making similar points to what I am trying to say here, in their comments. If I have failed to explain my objection adequately, those comments might help.

Comment author: Locaha 08 February 2014 02:44:16PM -3 points [-]

Angels. Pins. Sigh...

Comment author: CronoDAS 08 February 2014 04:01:15AM 1 point [-]

One can get kind of weird about whether or not a program is being run. After all, there exists a way to interpret the random motion of the molecules of a cup of hot tea as a succession of states corresponding to the execution of whatever program you want. Taking this line of argument to its ridiculous conclusion leads to Greg Egan's Dust Theory...

Comment author: Emile 08 February 2014 10:42:44AM 0 points [-]

Taking this line of argument to its ridiculous conclusion leads to Greg Egan's Dust Theory...

Is it ridiculous? As far as I can tell, Dust Theory (or something like it, I don't know the details offhand) is the cosmology that makes the most sense.

Comment author: CronoDAS 08 February 2014 09:40:07PM *  1 point [-]

Short answer: Any cosmology that suggests that we're Boltzmann brains probably has something wrong with it.

Comment author: Emile 08 February 2014 11:17:17PM 3 points [-]

Even if there's a 99% chance any instant that I'm a Boltzmann brain, my decisions only matter for the remaining 1%, so I may as well take the same decisions anyway. In other words, I don't behave any differently if the Boltzmann Brain thing is correct, so I don't have any problem with it.

Comment author: CronoDAS 09 February 2014 10:47:13PM 2 points [-]

I'm not entirely sure that's true. It's possible for a Boltzmann brain to exist for longer than an instant, so it should be possible for it to make decisions that affect its experiences in, say, the next few seconds. That might be enough to skew your expected value calculations in favor of, say, taking a vial of cocaine that happens to be in front of you...

Comment author: Emile 10 February 2014 04:24:18PM 1 point [-]

I have preferences over what happens to Emile on planet Earth, not on what happens to a Boltmann Brain that exists for a few seconds in the Minor Squableen Nebula of Galaxy XJB-819-delta. A Boltzmann Brain would also have preferences on what happens to Emile on planet Earth, so it would want to make the same decisions.

Comment author: 9eB1 07 February 2014 10:51:35PM 1 point [-]

Perhaps if you are an L-zombie and your decisions can influence the real world, then your decision doesn't matter. Omega has selected you from the space of all possible minds, which is unimaginably large, and asked for 1 bit of information, there will be a 50% chance that the bit will be either 0 or 1, since the space of all possible L-zombies has maximum entropy(?). Or perhaps not.

More generally, I think the idea that if you are an L-zombie your decisions might effect the real world is a violation of the thought experiment. It's presupposing something that doesn't make sense internally. If logical zombies can affect the universe, in what sense can they be said to exist only logically?

The difference between an L-zombie who only exists logically and a physical simulation is that simulations have some measure, but that measure must be smaller than the measure of actual reality because greater "resources of the universe" are required to provide the inputs to the simulation than are required to run an actually real person on the universe providing its own inputs.

If we ever developed a friendly AI, I'm pretty sure that I would want it to answer (A) to this question if it had reason to believe that simulations of its processing were physically impossible, technologically improbable, or (B) to this question if it had reason to believe that simulations were possible, and that there were people/aliens/gods likely to use these simulations and carry out the threats contained therein, which is exactly the sort of things humans would also do. I'm not sure what we are actually supposed to take away from this thought experiment?

I'm not standing by anything said above with high certainty, as I find this very confusing to think about.

Comment author: Coscott 08 February 2014 01:02:39AM 1 point [-]

I completely agree that L-zombies, if the concept made sense, would not be able to impact our world. The simulation of the L-zombie used to make it change the world brings the L-zombie to life.

Comment author: kokotajlod 08 February 2014 05:10:34PM 0 points [-]

Can you explain to me this notion of "resources of the universe?" I would have thought that simulated brains would have the same measure as actual brains.

Comment author: 9eB1 08 February 2014 07:29:55PM 0 points [-]

Thinking about it more carefully, I think the statement that they don't have the same measure is broken (not even wrong, incoherent).

As far as resources, I think the argument can be made in terms of entropy or energy, but I will make it in terms of energy because it's easier to understand. Suppose for the sake of argument that a perfect simulation of a brain requires the same amount of energy to run as a real brain (in actuality it would require more, because you either have to create a duplicate of the brain, which requires the same energy, or solve the field equations for the brain explicitly, which requires greater energy). In order to provide the simulated inputs to the brain, you have to spend energy to make sure that they are internally consistent, react properly to the outputs of the brain, etc. So it's impossible for a perfect simulation to require less energy than a real brain. If we are somewhere in a tower of simulated universes, either our simulation is being run imperfectly, or each universe in the tower must be smaller than the last, and probably dramatically smaller.

Now, imagine that you have a solar system worth of energy, and that running a simulation incurs an overhead of 3 orders of magnitude to calculate the simulation and consistent inputs to it. Using that solar system's energy, you can either support (warning: completely made up numbers ahead) a trillion people accepting inputs from the universe at large, or a billion people running on efficient perfect simulations (perfect with respect to their brain activity, but obviously not perfect with respect to the universe, because that's not possible without a greater than universe sized source of energy).

Measure with respect to minds is related to probability, so it really relates an existing consciousness to its futures. If I step into a duplicator, the measure of each of my future selves is 1/2 with respect to my current self, because I have a 50% probability of ending up in either of those future bodies, but from the perspective of my duplicates post-duplication, their measure is once again one. Bearing this in mind, the measure of a simulation currently running is 1, from its own perspective, and the measure of any given individual is also 1 if they currently exist.

Comment author: kokotajlod 09 February 2014 04:06:39PM 0 points [-]

Thinking about it more carefully, I think the statement that they don't have the same measure is broken (not even wrong, incoherent).

So you agree with me then, that they have the same measure?

As for resources: I really don't think that the amount of energy and matter used to compute a mind has any bearing on the measure of that mind. What matters is whether or not the energy and matter instantiates the correct program; if it does, then the mind exists there, if it doesn't, then it doesn't.

True, the quantity of minds matters (probably) for measure. So a mind with a trillion copies has greater measure than a mind with a billion copies. If we think that the relevant level of detail for implementation is exactly the fundamental level for our brains, then yes this would mean we should expect ourselves, other things equal, to be brains rather than simulations. But I'd say it is highly likely that the relevant level of detail for implementation is much higher--the neuron level, say--and thus simulations quite possibly outnumber brains by a great deal.

Of course, either way, it comes down to more than just the resource requirements--it also comes down to e.g. how likely it is that a posthuman society would create large numbers of ancestor simulations.

Comment author: mwengler 11 February 2014 01:21:12AM 0 points [-]

Why would somebody care about the properties of things that do not exist?

I think this is a real question, an interesting question, so don't downvote it because you think I am being flippant. Instead think about why you care about the properties of things that don't exist. I think anybody who cares about the properties of something that doesn't exist is equivocating on the meaning of doesn't exist: they are thinking something that doesn't exist is actually something, but with its "I exist" checkbox left unchecked. As opposed to the idea that something that doesn't exist really just doesn't exist: it has no checkboxes. Existence is not a property of something that it can either have or not have. If it doesn't exist it doesn't have properties.

I guess I do care a little, just a LOT less than I do about the properties of things that do exist.

Comment author: Eugine_Nier 12 February 2014 05:48:49AM 0 points [-]

See counter factual mugging and related situations in decision theory.

Comment author: Lumifer 11 February 2014 01:27:20AM 1 point [-]

Why would somebody care about the properties of things that do not exist?

Can you be more specific? There are few more or less trivial answers -- e.g. a model of some real process does not exist (it's just a model in my head) but I'm interested in its properties because the model is useful for, say, forecasting reality. Or, for another example, I might be imagining a thing that I will build and, again, the reason to be interested in its properties is obvious.

Comment author: Pentashagon 09 February 2014 06:23:32AM 0 points [-]

Consider a Turing Machine whose input is the encoded state of the world as a binary integer, S, of maximum value 2^N-1, and which seeks SN positions into its binary tape and outputs the next N bits from the tape. Does that Turing Machine cause the same experience as an equivalent Turing Machine that simulates 10 seconds of the laws of physics for the world state S and outputs the resulting world state? I posit that the former TM actually causes no experience at all when run, despite the equivalence. So there probably exist l-zombies that would act as if they have experience when they are run but are *wrong.

Your last paragraph brings up an interesting question. I assumed that transitions between l-zombie and person are one-way. Once run, how do you un-run something? It seems to imply that you could construct a TM that is and is not an l-zombie.

Comment author: notsonewuser 08 February 2014 03:34:51PM 0 points [-]

Good post, but...

I imagined myself as those L-zombies as I was reading through and trying to understand. Thus they're not L-zombies anymore. Did you do the same as you were writing?

Comment author: Ishaan 11 February 2014 10:51:09AM *  0 points [-]

In general for counter-factual muggings, the way I like to think of it is that you would like to choose A, but as a rational agent you pre-committed to choosing B at a previous time point and are now physically incapable of making a different choice, thanks to an iron-clad pre-commitement originally intended to convince the omnipotent Omega that you were going to behave a certain way, by actually binding you to behave thusly.

But if you find that you can choose A, then by all means, choose A. It's evidence that you failed as a rational agent at a previous time point, but at the current time point it means you're lucky.

(Of course, this is all following the premise that you realize this is the end now, and that you aren't in a simulation.)

Comment author: Kutta 09 February 2014 04:44:21PM *  0 points [-]

The unsimulated life is not worth living.

-- Socrates.

Comment author: mwengler 11 February 2014 01:22:46AM 1 point [-]

I've always admired the rhetorical dishonesty of the original quote: the unexamined life is not worth living.

In fact, this statement says NOTHING about whether the examined life is worth living or not. But virtually every usage of this original quote I have ever heard was intended to exhort us to examine our lives in order to make them worthwhile.

Comment author: Trevor_Blake 08 February 2014 12:47:14AM -3 points [-]

The thought patterns of infants, people with traumatic brain injuries, people with some mental illnesses, people who are asleep and people on drugs can be as alien as you describe here. Non-human animal's thought patterns are also as alien as you like. Right here on this Earth are likely examples of what you describe. And in an infinite universe, yes for sure.

Comment author: shminux 12 February 2014 12:15:34AM -1 points [-]

Why stop at the written program level? What if you are about to type the final semi-colon in the description of a simulated human? When does it become an L-zombie or, alternatively, conscious? What about the day before you go to your office to finish the program? Maybe at the moment you made the decision to write this program? Where is the magical boundary? Is "finished program" just a convenient Schelling point?

Comment author: Benja 28 February 2014 08:31:08AM 2 points [-]

I'm not sure which of the following two questions you meant to ask (though I guess probably the second one), so I'll answer both:

(a) "Under what circumstances is something (either an l-zombie or conscious)?" I am not saying that something is an l-zombie only if someone has actually written out the code of the program; for the purposes of this post, I assume that all natural numbers exist as platonical objects, and therefore all observers in programs that someone could in principle write and run exist at least as l-zombies.

(b) "When is a program an l-zombie, and when is it conscious?" The naive view would be that the program has to be actually run in the physical world; if you've written a program and then deleted the source without running it, it wouldn't be conscious. But as to what exactly the rule is that you can use to look at say a cellular automaton (as a model of physical reality) and ask whether the conscious experience inside a given Turing machine is "instantiated inside" that automaton, I don't have one to propose. I do think that's a weak point of the l-zombies view, and one reason that I'd assign measureless Tegmark IV higher a priori probability.

Comment author: shminux 28 February 2014 03:38:42PM 1 point [-]

Thanks for clarifying!