Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Comment author: Vladimir_Nesov 07 February 2017 03:18:18AM *  1 point [-]

A way around this would be if you’re not completely updateless, but if you instead have already updated on the fact that you do exist.

It's not a given that you can easily observe your existence. From updateless point of view, all possible worlds, or theories of worlds, or maybe finite fragments of reasoning about them, in principle "exist" to some degree, in the sense of being data potentially relevant for estimating the value of everything, which is something to be done for the strategies under agent's consideration. So in case of worlds, or instances of the agent in worlds, the useful sense of "existence" is relevance for estimating the value of everything (or of change in value depending on agent's strategy, which is the sense in which worlds that couldn't contain or think about the agent, don't exist). Since in this case we are talking about possible worlds, they do or don't exist in the sense of having no measure (probability) in the updateless prior (to the extent that it makes sense to talk about the decision algorithm using a prior). In this sense, observing one's existence means observing an argument about the a priori probability of the world you inhabit. In a world that has relatively tiny a priori probability, you should be able to observe your own (or rather the world's) non-existence, in the same sense.

This also follows the principle of reducing concepts like existence or probability (where they make sense) to components of the decision algorithm, and abandoning them in sufficiently unusual thought experiments (where they may fail to make sense, but where it's still possible to talk about decisions). See also this post of Vadim's and the idea of cognitive reductions (looking for the role a concept plays in your thinking, not just for what it could match in the world).

Comment author: Johannes_Treutlein 25 February 2017 09:13:10PM *  0 points [-]

Thanks for the reply and all the useful links!

It's not a given that you can easily observe your existence.

It took me a while to understand this. Would you say that for example in the Evidential Blackmail, you can never tell whether your decision algorithm is just being simulated or whether you're actually in the world where you received the letter, because both times, the decision algorithms receive exactly the same evidence? So in this sense, after updating on receiving the letter, both worlds are still equally likely, and only via your decision do you find out which of those worlds are the simulated ones and which are the real ones. One can probably generalize this principle: you can never differentiate between different instantiations of your decision algorithm that have the same evidence. So when you decide what action to output conditional on receiving some sense data, you always have to decide based on your prior probabilities. Normally, this works exactly as if you would first update on this sense data and then decide. But sometimes, e.g. if your actions in one world make a difference to the other world via a simulation, then it makes a difference. Maybe if you assign anthropic probabilities to either being a "logical zombie" or the real you, then the result would be like UDT even with updating?

What I still don't understand is how this motivates updatelessness with regard to anthropic probabilities (e.g. if I know that I have a low index number, or in Psy Kosh's problem, if I already know I'm the decider). I totally get how it makes sense to precommit yourself and how one should talk about decision problems instead of probabilities, how you should reason as if you're all instantiations of your decision algorithm at once, etc. Also, intuitively I agree with sticking with the priors. But somehow I can't get my head around what exactly is wrong about the update. Why is it wrong to assign more "caring energy" to the world in which some kind of observation that I make would have been more probable? Is it somehow wrong that it "would have been more probable"? Did I choose the wrong reference classes? Is it because in these problems, too, the worlds influence each other, so that you have to consider the impact that your decision would have on the other world as well?

Edit: Never mind, I think http://lesswrong.com/lw/jpr/sudt_a_toy_decision_theory_for_updateless/ kind of answers my question :)

Comment author: ProofOfLogic 02 February 2017 10:52:28PM 1 point [-]

I find this and the smoker's lesion to have the same flaw, namely: it does not make sense to me to both suppose that the agent is using EDT, and suppose some biases in the agent's decision-making. We can perhaps suppose that (in both cases) the agent's preferences are what is affected (by the genes, or by the physics). But then, shouldn't the agent be able to observe this (the "tickle defense"), at least indirectly through behavior? And won't this make it act as CDT would act?

But: I find the blackmail letter to be a totally compelling case against EDT.

Comment author: Johannes_Treutlein 24 February 2017 10:08:57AM *  1 point [-]

I agree with all of this, and I can't understand why the Smoking Lesion is still seen as the standard counterexample to EDT.

Regarding the blackmail letter: I think that in principle, it should be possible to use a version of EDT that also chooses policies based on a prior instead of actions based on your current probability distribution. That would be "updateless EDT", and I think it wouldn't give in to Evidential Blackmail. So I think rather than an argument against EDT, it's an argument in favor of updatelessness.

Comment author: cousin_it 09 February 2017 07:27:37PM *  1 point [-]

UDT takes bet 2.

Can you put your flavor of EDT in clear conflict with UDT? Or are they equivalent?

If you need a rigorous formulation of proof-based UDT, this old post of mine might be helpful. Feel free to ask if anything isn't clear.

Comment author: Johannes_Treutlein 24 February 2017 09:44:16AM *  0 points [-]

Thanks for the link! What I don't understand is how this works in the context of empirical and logical uncertainty. Also, it's unclear to me how this approach relates to Bayesian conditioning. E.g. if the sentence "if a holds, than o holds" is true, doesn't this also mean that P(o|a)=1? In that sense, proof-based UDT would just be an elaborate specification of how to assign these conditional probabilities "from the viewpoint of the original position", so with updatelessness, and in the context of full logical inference and knowledge of the world, including knowledge about one's own decision algorithm. I see how this is useful, but don't understand how it would at any point contradict normal Bayesian conditioning.

As to your first question: if we ignore problems that involve updatelessness (or if we just stipulate that EDT always had the opportunity to precommit), I haven't been able to find any formally specified problems where EDT and UDT diverge.

I think Caspar Oesterheld's and my flavor of EDT would be ordinary EDT with some version of updatelessness. I'm not sure if this works, but if it turns out to be identical to UDT, then I'm not sure which of the two is the better specified or easier to formalize one. According to the language in Arbital's LDT article, my EDT would differ from UDT only insofar as instead of some logical conditioning, we use ordinary Bayesian conditioning. So (staying in the Arbital framework), it could look something like this (P stands for whatever prior probability distribution you care about):

Comment author: Vladimir_Nesov 08 February 2017 10:00:04AM *  2 points [-]

I'll cite the thought experiment for the reference:

Betting on the Past: In my pocket (says Bob) I have a slip of paper on which is written a proposition P. You must choose between two bets. Bet 1 is a bet on P at 10:1 for a stake of one dollar. Bet 2 is a bet on P at 1:10 for a stake of ten dollars. So your pay-offs are as follows: Bet 1, P is true: 10; Bet 1, P is false: -1; Bet 2, P is true: 1; Bet 2, P is false: -10. Before you choose whether to take Bet 1 or Bet 2 I should tell you what P is. It is the proposition that the past state of the world was such as to cause you now to take Bet 2. [Ahmed 2014, p. 120]

Some comments on your post:

Alice is betting on a past state of the world. She can’t causally influence the past, and she’s uncertain whether the proposition is true or not.

More precisely, Alice is betting on implications of the past state of the world, on what it means about the future, or perhaps on what it causes the future to be. Specifically Alice's action, an implication of the past state of the world. If we say that Alice can causally influence her own action, it's fair to say that Alice can causally influence the truth of the proposition, even if she can't causally influence the state of the past. So she can't influence the state of the past, but can influence implications of the state of the past, such as her own action. Similarly, a decision algorithm can't influence its own code, but can influence the result it computes. (So I'm not even sure what CDT is supposed to do here, since it's not clear that the bet is really on the past state of the world and not on truth of a proposition about the future state of the world.)

Perhaps if the bet was about the state of the world yesterday, LDT would still take Bet 2. Clearly, LDT’s algorithm already existed yesterday, and it can influence this algorithm’s output; so if it chooses Bet 2, it can change yesterday’s world and make the proposition true.

It's better to avoid the idea of "change" in this context. Change always compares alternatives, but for UDT, there is no default state of the world before-decision-is-made, there are only alternative states of the world following the alternative decisions. So a decision doesn't change things from the way they were before it's made to the way after it's made, at most you can compare how things are after one possible decision to how things are after the other possible decision.

Given that, I don't see what role "LDT’s algorithm already existed yesterday" plays here, and I think it's misleading to state that "it can change yesterday’s world and make the proposition true". Instead it can make the proposition true without changing yesterday’s world, by ensuring that yesterday’s world was always such that the proposition is true. There is no change, yesterday’s world was never different and the proposition was never false. What changed (in our observation of the decision making process) is the state of knowledge about yesterday’s world, from uncertainty about the truth of the proposition to knowledge that it's true.

If we choose a more distant point in the past as a reference for Alice’s bet – maybe as far back as the birth of our universe – she’ll eventually be unable to exert any possible influence via logical counterfactuals.

Following from the preceding point, it doesn't matter when the past state of the world is, since we are not trying to influence it, we are instead trying to influence its consequences, which are in the future. There is something unusual about influencing consequences of a construction without influencing the construction itself, but it helps to recall that it's exactly what any program does, when it influences its actions without influencing its code. It's what a human emulation in a computer does, by making decisions without changing the initial image of their brain from which the running emulation was loaded. And it's also what a regular human running inside physics without any emulation does.

Comment author: Johannes_Treutlein 22 February 2017 01:50:30PM *  0 points [-]

Thanks a lot for your elaborate reply!

(So I'm not even sure what CDT is supposed to do here, since it's not clear that the bet is really on the past state of the world and not on truth of a proposition about the future state of the world.)

Hmm, good point. The truth of the proposition is evaluated on basis of Alice's action, which she can causally influence. But we could think of a Newcomblike scenario in which someone made a perfect prediction a 100 years ago and put down a note about what state the world was in at that time. Now instead of checking Alice's action, we just check this note to evaluate whether the proposition is true. I think then it's clear that CDT would "two-box".

Given that, I don't see what role "LDT’s algorithm already existed yesterday" plays here, and I think it's misleading to state that "it can change yesterday’s world and make the proposition true". Instead it can make the proposition true without changing yesterday’s world, by ensuring that yesterday’s world was always such that the proposition is true. There is no change, yesterday’s world was never different and the proposition was never false.

Sorry for the fuzzy wording! I agree that "change" is not a good terminology. I was thinking about TDT and a causal graph. In that context, it might have made sense to say that TDT can "determine the output" of the decision nodes, but not that of the nature nodes that have a causal influence on the decision nodes?

Following from the preceding point, it doesn't matter when the past state of the world is, since we are not trying to influence it, we are instead trying to influence its consequences, which are in the future.

OK, if I interpret that correctly, you would say that our proposition is also a program that references Alice's decision algorithm, and hence we can just determine that program's output the same way we can determine our own decision. I am totally fine with that. If we can expand this principle to all the programs that somehow reference our decision algorithms, I would be curious whether there are still differences left between this and evidential counterfactuals.

Take the thought experiment in this post, for instance: Imagine you're an agent that always chooses the action "take the red box". Now there is a program that checks whether there will be cosmic rays, and if so, then it changes your decision algorithm to one that outputs "take the green box". Of course, you can still "influence" your output like all regular humans, and you can thus in some sense also influence the output of the program that changed you. By extension, you can even influence whether or not the output of the program "outer space" is "gamma rays" or "no gamma rays". If I understand your answers to my Coin Flip Creation post correctly, this formulation would make the problem into a kind of anthropic problem again, where the algorithm would at one point "choose to output red" in order to be instantiated into the world without gamma rays. Would you agree with this, or did I get something wrong?

[Link] “Betting on the Past” – a decision problem by Arif Ahmed

2 Johannes_Treutlein 07 February 2017 09:14PM
Comment author: Johannes_Treutlein 03 February 2017 10:53:46AM 0 points [-]

CDT, TDT, and UDT would not give away the money because there is no causal (or acausal) influence on the number of universes.

I'm not so sure about UDT's response. From what I've heard, depending on the exact formal implementation of the problem, UDT might also pay the money? If your thought experiment works via a correlation between the type of universe you live in and the decision theory you employ, then it might be a similar problem to the Coin Flip Creation. I introduced the latter decision problem in an attempt to make a less ambiguous version of the Smoking Lesion. In a comment in response to my post, cousin_it writes:

Here's why I think egoistic UDT would one-box. From the problem setup it's provable that one-boxing implies finding money in box A. That's exactly the information that UDT requires for decision making ("logical counterfactual"). It doesn't need to deduce unconditionally that there's money in box A or that it will one-box.

One possible confounder in your thought experiment is the agent’s altruism. The agent doesn’t care about which world he lives in, but only about which worlds exist. If you reason from an “updateless”, outside perspective (like Anthropic Decision Theory), it then becomes irrelevant what you choose. This is because if you act in a way that’s only logically compatible with world A, you know you just wouldn’t have existed in the other world. A way around this would be if you’re not completely updateless, but if you instead have already updated on the fact that you do exist. In this case you’d have more power with your decision. “One-boxing” might also make sense if you're just a copy-egoist and prefer to live in world A.

Comment author: cousin_it 26 January 2017 05:39:30PM 1 point [-]

I can only give a clear-cut answer if you reformulate the smoking lesion problem in terms of Omega and specify the UDT agent's egoism or altruism :-)

Comment author: Johannes_Treutlein 30 January 2017 01:50:44PM 0 points [-]

That's what I was trying to do with the Coin Flip Creation :) My guess: once you specify the Smoking Lesion and make it unambiguous, it ceases to be an argument against EDT.

Comment author: Vaniver 26 January 2017 07:27:31PM *  1 point [-]

So causal thinking in some way seems to violate the deterministic way the world works.

I agree there's a point here that lots of decision theories / models of agents / etc. are dualistic instead of naturalistic, but I think that's orthogonal to EDT vs. CDT vs. LDT; all of them assume that you could decide to take any of the actions that are available to you.

My point is that if we assume that we can have a causal influence on the future, then this is already a kind of violation of determinism

I suspect this is a confusion about free will. To be concrete, I think that a thermostat has a causal influence on the future, and does not violate determinism. It deterministically observes a sensor, and either turns on a heater or a cooler based on that sensor, in a way that does not flow backwards--turning on the heater manually will not affect the thermostat's attempted actions except indirectly through the eventual effect on the sensor.

One could maybe even object to Newcomb's original problem on similar grounds. Imagine the prediction has already been made 10 years ago. You learned about decision theories and went to one of the gurus in the meantime, and are now confronted with the problem. Are you now free to choose or does the prediction mess with your new, intended action, so that you can't choose the way you want?

This depends on the formulation of Newcomb's problem. If it says "Omega predicts you with 99% accuracy" or "Omega always predicts you correctly" (because, say, Omega is Laplace's Demon), then Omega knew that you would learn about decision theory in the way that you did, and there's still a logical dependence between the you looking at the boxes in reality and the you looking at the boxes in Omega's imagination. (This assumes that the 99% fact is known of you in particular, rather than 99% accuracy being something true of humans in general; this gets rid of the case that 99% of the time people's decision theories don't change, but 1% of the time they do, and you might be in that camp.)

If instead the formulation is "Omega observed the you of 10 years ago, and was able to determine whether or not you then would have one-boxed or two-boxed on traditional Newcomb's with perfect accuracy. The boxes just showed up now, and you have to decide whether to take one or both," then the logical dependence is shattered, and two-boxing becomes the correct move.

If instead the formulation is "Omega observed the you of 10 years ago, and was able to determine whether or not you then would have one-boxed or two-boxed on this version of Newcomb's with perfect accuracy. The boxes just showed up now, and you have to decide whether to take one or both," then the logical dependence is still there, and one-boxing is the correct move.

(Why? Because how can you tell whether you're the actual you looking at the real boxes, or the you in Omega's imagination, looking at simulated boxes?)

Comment author: Johannes_Treutlein 30 January 2017 01:47:01PM 0 points [-]

I suspect this is a confusion about free will. To be concrete, I think that a thermostat has a causal influence on the future, and does not violate determinism. It deterministically observes a sensor, and either turns on a heater or a cooler based on that sensor, in a way that does not flow backwards--turning on the heater manually will not affect the thermostat's attempted actions except indirectly through the eventual effect on the sensor.

Fair point :) What I meant was that for every world history, there is only one causal influence I could possibly have on the future. But CDT reasons through counterfactuals that are physically impossible (e.g. two-boxing in a world where there is money in box A), because it combines world states with actions it wouldn't take in those worlds. EDT just assumes that it's choosing between different histories, which is kind of "magical", but at least all those histories are internally consistent. Interestingly, e.g. Proof-Based DT would probably amount to the same kind of reasoning? Anyway, it's probably a weak point if at all, and I fully agree that the issue is orthogonal to the DT question!

I basically agree with everything else you write, and I don't think it contradicts my main points.

Comment author: cousin_it 24 January 2017 07:23:35PM *  2 points [-]

My thoughts:

1) "Copy-egoistic" and "copy-altruistic" seems misleading, because Omega creates different agents in the heads and tails case. Plain "egoistic" and "altruistic" would work though.

2) Multiple worlds vs single world should be irrelevant to UDT.

3) I think UDT would one-box if it's egoistic, and be indifferent if it's altruistic.

Here's why I think egoistic UDT would one-box. From the problem setup it's provable that one-boxing implies finding money in box A. That's exactly the information that UDT requires for decision making ("logical counterfactual"). It doesn't need to deduce unconditionally that there's money in box A or that it will one-box.

Comment author: Johannes_Treutlein 26 January 2017 05:30:27PM 0 points [-]

I agree with points 1) and 2). Regarding point 3), that's interesting! Do you think one could also prove that if you don't smoke, you can't (or are less likely to) have the gene in the Smoking Lesion? (See also my response to Vladimir Nesov's comment.)

Comment author: Vaniver 26 January 2017 12:51:47AM *  5 points [-]

There seems to be an especially strong intuition of “absence of free will” inherent to the Coin Flip Creation problem. When presented with the problem, many respond that if someone had created their source code, they didn’t have any choice to begin with. But that’s the exact situation in which we all find ourselves at all times!

I think this is missing the point of the objection.

Consider the three different decision theories, CDT, EDT, and LDT; suppose there are three gurus who teach those decision theories to any orphans left in their care. And suppose Omega does the coin flip six times, ends up with three heads children and three tails children, and gives a matched pair to each of the gurus.

When the day comes, the first set of children reason that they can't change the coinflip because the lack of causal dependence, and try to take both boxes. One succeeds, and the other discovers that, mysteriously, they one-boxed instead, and got the million.

The second set of children reason that taking one box is correlated with having the million, and so they try to take just the one box. One succeeds, and the other discovers that, mysteriously, they two-boxed instead, and only got the thousand.

The third set, you know the drill. One one-boxes, the other two-boxes.

The point of decision theories is not that they let you reach from beyond the Matrix and change reality in violation of physics; it's that you predictably act in ways that optimize for various criteria. But this is a decision problem where your action has been divorced from your intended action, and so attributing the victory of heads children to EDT is mistaken, because of the tails child with EDT who wanted to two-box but couldn't.


(Also, Betteridge's Law.)

Comment author: Johannes_Treutlein 26 January 2017 05:07:21PM *  1 point [-]

The point of decision theories is not that they let you reach from beyond the Matrix and change reality in violation of physics; it's that you predictably act in ways that optimize for various criteria.

I agree with this. But I would argue that causal counterfactuals somehow assume that we can "reach from beyond the Matrix and change reality in violation of physics". They work by comparing what would happen if we detached our “action node” from its ancestor nodes and manipulated it in different ways. So causal thinking in some way seems to violate the deterministic way the world works. Needless to say, all decision theories somehow have to reason through counterfactuals, so they all have to form “impossible” hypotheses. My point is that if we assume that we can have a causal influence on the future, then this is already a kind of violation of determinism, and I would reason that assuming that we can also have a retro-causal one on the past doesn’t necessarily make things worse. In some sense, it might even be more in line with how the world works: the future is as fixed as the past, and the EDT approach is to merely “find out” which respective past and future are true.

But this is a decision problem where your action has been divorced from your intended action, and so attributing the victory of heads children to EDT is mistaken, because of the tails child with EDT who wanted to two-box but couldn't.

Hmm, I'm not sure. It seems as thought in your setup, the gurus have to change the children's decision algorithms, in which case of course the correlation would vanish. Or the children use a meta decision theory like "think about the topic and consider what the guru tells you and then try to somehow do whatever winning means". But if Omega created you with the intention of making you one-box or two-box, it could easily just have added some rule or change the meta theory so that you would end up just not being convinced of the "wrong" theory. You would have magically ended up doing (and thinking) the right thing, without "wanting" but not "being able to". I mean, I am trying to convince you of some decision theory right now, and you already have some knowledge and meta decision theory that ultimately will lead you to either adopt or reject it. Maybe the fact that you're not yet convinced shows that you're living in the tails world? ;) Maybe Omega's trick is to make the tails people think about guru cases in order to get them to reject EDT?

One could maybe even object to Newcomb's original problem on similar grounds. Imagine the prediction has already been made 10 years ago. You learned about decision theories and went to one of the gurus in the meantime, and are now confronted with the problem. Are you now free to choose or does the prediction mess with your new, intended action, so that you can't choose the way you want? I don't believe so – you'll feel just as free to choose as if the prediction had happened 10 minutes ago. Only after deciding freely, you find out that you have been determined to decide this way from the beginning, because Omega of course also accounted for the guru.

In general, I tend to think that adding some "outside influence" to a Newcomb's problem either makes it a different decision problem, or it's irrelevant and just confuses things.

View more: Next