Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Comment author: scarcegreengrass 22 June 2017 09:02:30PM 0 points [-]

So, i'm trying to wrap my head around this concept. Let me sketch an example:

Far-future humans have a project where they create millions of organisms that they think could plausibly exist in other universes. They prioritize organisms that might have evolved given whatever astronomical conditions are thought to exist in other universes, and organisms that could plausibly base their behavior on moral philosophy and game theory. They also create intelligent machines, software agents, or anything else that could be common in the multiverse. They make custom habitats for each of these species and instantiate a small population in each one. The humans do this via synthetic biology, actual evolution from scratch (if affordable), or simulation. Each habitat is optimized to be an excellent environment to live in from the perspective of the species or agent inside it. This whole project costs a small fraction of the available resources of the human economy. The game theoretic motive is that, by doing something good for a hypothetical species, there might exist an inaccessible universe in which that species is both living and able to surmise that the humans have done this, and that they will by luck create a small utopia of humans when they do their counterpart project.

Is this an example of the type of cooperation discussed here?

Comment author: Caspar42 23 June 2017 04:42:12PM 0 points [-]

Yep, this is roughly the type of cooperation I have in mind. Some minor points:

Overall, I am not sure whether gains from trade would arise in this specific scenario. Perhaps, it’s not better for the civilizations than if each civilization only builds habitats for itself?

The game theoretic motive is that, by doing something good for a hypothetical species, there might exist an inaccessible universe in which that species is both living and able to surmise that the humans have done this, and that they will by luck create a small utopia of humans when they do their counterpart project.

As I argue in section “No reciprocity needed: whom to treat beneficially”, the benefit doesn’t necessarily come from the species that we benefit. Even if agent X is certain that agent Y cannot benefit X, agent X may still help Y to make it more likely that X receives help from other agents who are in a structurally similar situation w.r.t. Y and think about it in a way similar to X.

Also, the other civilizations don’t need to be able to check whether we helped them, just like in the prisoner’s dilemma against a copy we don’t have to be able to check whether the other copy actually cooperated. It’s enough to know, prior to making one’s own decision, that the copy reasons similarly about these types of problems.

Comment author: scarcegreengrass 22 June 2017 09:02:30PM 0 points [-]

So, i'm trying to wrap my head around this concept. Let me sketch an example:

Far-future humans have a project where they create millions of organisms that they think could plausibly exist in other universes. They prioritize organisms that might have evolved given whatever astronomical conditions are thought to exist in other universes, and organisms that could plausibly base their behavior on moral philosophy and game theory. They also create intelligent machines, software agents, or anything else that could be common in the multiverse. They make custom habitats for each of these species and instantiate a small population in each one. The humans do this via synthetic biology, actual evolution from scratch (if affordable), or simulation. Each habitat is optimized to be an excellent environment to live in from the perspective of the species or agent inside it. This whole project costs a small fraction of the available resources of the human economy. The game theoretic motive is that, by doing something good for a hypothetical species, there might exist an inaccessible universe in which that species is both living and able to surmise that the humans have done this, and that they will by luck create a small utopia of humans when they do their counterpart project.

Is this an example of the type of cooperation discussed here?

Comment author: Caspar42 23 June 2017 04:39:59PM 0 points [-]

Yep, this is roughly the type of cooperation I have in mind. Some minor points:

Overall, I am not sure whether gains from trade would arise in this specific scenario. Perhaps, it’s not better for the civilizations than if each civilization only builds habitats for itself?

The game theoretic motive is that, by doing something good for a hypothetical species, there might exist an inaccessible universe in which that species is both living and able to surmise that the humans have done this, and that they will by luck create a small utopia of humans when they do their counterpart project.

As I argue in section “No reciprocity needed: whom to treat beneficially”, the benefit doesn’t necessarily come from the species that we benefit. Even if agent X is certain that agent Y cannot benefit X, agent X may still help Y to make it more likely that X receives help from other agents who are in a structurally similar situation w.r.t. Y and think about it in a way similar to X.

Also, the other civilizations don’t need to be able to check whether we helped them, just like in the prisoner’s dilemma against a copy we don’t have to be able to check whether the other copy actually cooperated. It’s enough to know, prior to making one’s own decision, that the copy reasons similarly about these types of problems.

Comment author: Caspar42 29 May 2017 01:30:03PM 1 point [-]

Closely related is the law of total expectation: https://en.wikipedia.org/wiki/Law_of_total_expectation

It states that E[E[X|Y]]=E[X].

Comment author: username2 29 May 2017 12:50:09PM 2 points [-]

And if pilot wave theory is correct and these other universes exist only in your head...?

Comment author: Caspar42 29 May 2017 01:14:30PM 2 points [-]

Then it doesn't work unless you believe in some other theory that postulates the existence of a sufficiently large universe or multiverse, Everett is only one option.

Invitation to comment on a draft on multiverse-wide cooperation via alternatives to causal decision theory (FDT/UDT/EDT/...)

5 Caspar42 29 May 2017 08:34AM

I have written a paper about “multiverse-wide cooperation via correlated decision-making” and would like to find a few more people who’d be interested in giving a last round of comments before publication. The basic idea of the paper is described in a talk you can find here. The paper elaborates on many of the ideas and contains a lot of additional material. While the talk assumes a lot of prior knowledge, the paper is meant to be a bit more accessible. So, don’t be disheartened if you find the talk hard to follow — one goal of getting feedback is to find out which parts of the paper could be made more easy to understand.

If you’re interested, please comment or send me a PM. If you do, I will send you a link to a Google Doc with the paper once I'm done with editing, i.e. in about one week. (I’m afraid you’ll need a Google Account to read and comment.) I plan to start typesetting the paper in LaTeX in about a month, so you’ll have three weeks to comment. Since the paper is long, it’s totally fine if you don’t read the whole thing or just browse around a bit.

Comment author: Caspar42 18 May 2017 08:35:53PM 1 point [-]

Another piece of evidence is this minor error in section 9.2 of Peterson's An Introduction to Decision Theory:

According to causal decision theory, the probability that you have the gene given that you read Section 9.2 is equal to the probability that you have the gene given that you stop at Section 9.1. (That is, the probability is independent of your decision to read this section.) Hence, it would be a mistake to think that your chances of leading a normal life would have been any higher had you stopped reading at Section 9.1.

Comment author: entirelyuseless 16 May 2017 02:08:42PM 2 points [-]

I agree that this is part of what confuses the discussion. This is why I have pointed out in previous discussions that in order to be really considering Newcomb / Smoking Lesion, you have to be honestly more convinced the million is in the box after choosing to take one, than you would have been if you had chosen both. Likewise, you have to be honestly more convinced that you have the lesion, after you choose to smoke, than you would have been if you did not. In practice people would tend not to change their minds about that, and therefore they should smoke and take both boxes.

Some relevant discussion here.

Comment author: Caspar42 18 May 2017 01:32:32PM 0 points [-]

Great, thank you!

Comment author: Caspar42 18 May 2017 12:11:57PM *  2 points [-]

In chapter 0.6 of his book Evidence, Decision and Causality, Arif Ahmed also argues that Calvinist predestination is like Newcomb's problem.

Are causal decision theorists trying to outsmart conditional probabilities?

4 Caspar42 16 May 2017 08:01AM

Presumably, this has been discussed somewhere in the past, but I wonder to which extent causal decision theorists (and many other non-evidential decision theorists, too) are trying to make better predictions than (what they think to be) their own conditional probabilities.

 

To state this question more clearly, let’s look at the generic Newcomb-like problem with two actions a1 and a2 (e.g., one-boxing and two-boxing, cooperating or defecting, not smoking or smoking) and two states s1 and s2 (specifying, e.g., whether there is money in both boxes, whether the other agent cooperates, whether one has cancer). The Newcomb-ness is the result of two properties:

  • No matter the state, it is better to take action a2, i.e. u(a2,s1)>u(a1,s1) and u(a2,s2)>u(a1,s2). (There are also problems without dominance where CDT and EDT nonetheless disagree. For simplicity I will assume dominance, here.)

  • The action cannot causally affect the state, but somehow taking a1 gives us evidence that we’re in the preferable state s1. That is, P(s1|a1)>P(s1|a2) and u(a1,s1)>u(a2,s2).

Then, if the latter two differences are large enough, it may be that

E[u|a1] > E[u|a2].

I.e.

P(s1|a1) * u(s1,a1) + P(s2|a1) * u(s2,a1) > P(s1|a2) * u(s1,a2) + P(s2|a2) * u(s2,a2),

despite the dominance.

 

Now, my question is: After having taken one of the two actions, say a1, but before having observed the state, do causal decision theorists really assign the probability P(s1|a1) (specified in the problem description) to being in state s1?

 

I used to think that this was the case. E.g., the way I learned about Newcomb’s problem is that causal decision theorists understand that, once they have said the words “both boxes for me, please”, they assign very low probability to getting the million. So, if there were a period between saying those words and receiving the payoff, they would bet at odds that reveal that they assign a low probability (namely P(s1,a2)) to money being under both boxes.

 

But now I think that some of the disagreement might implicitly be based on a belief that the conditional probabilities stated in the problem description are wrong, i.e. that you shouldn’t bet on them.

 

The first data point was the discussion of CDT in Pearl’s Causality. In sections 1.3.1 and 4.1.1 he emphasizes that he thinks his do-calculus is the correct way of predicting what happens upon taking some actions. (Note that in non-Newcomb-like situations, P(s|do(a)) and P(s|a) yield the same result, see ch. 3.2.2 of Pearl’s Causality.)

 

The second data point is that the smoking intuition in smoking lesion-type problems may often be based on the intuition that the conditional probabilities get it wrong. (This point is also inspired by Pearl’s discussion, but also by the discussion of an FB post by Johannes Treutlein. Also see the paragraph starting with “Then the above formula for deciding whether to pet the cat suggests...” in the computer scientist intro to logical decision theory on Arbital.)

 

Let’s take a specific version of the smoking lesion as an example. Some have argued that an evidential decision theorist shouldn’t go to the doctor because people who go to the doctor are more likely to be sick. If a1 denotes staying at home (or, rather, going anywhere but a doctor) and s1 denotes being healthy, then, so the argument goes, P(s1|a1) > P(s1|a2). I believe that in all practically relevant versions of this problem this difference in probabilities disappears once we take into account all the evidence we already have. This is known as the tickle defense. A version of it that I agree with is given in section 4.3 of Arif Ahmed’s Evidence, Decision and Causality. Anyway, let’s assume that the tickle defense somehow doesn’t apply, such that even if taking into account our entire knowledge base K, P(s1|a1,K) > P(s1|a2,K).

 

I think the reason why many people think one should go to the doctor might be that while asserting P(s1|a1,K) > P(s1|a2,K), they don’t upshift the probability of being sick when they sit in the waiting room. That is, when offered a bet in the waiting room, they wouldn’t accept exactly the betting odds that P(s1|a1,K) and P(s1|a2,K) suggest they should accept.

 

Maybe what is going on here is that people have some intuitive knowledge that they don’t propagate into their stated conditional probability distribution. E.g., their stated probability distribution may represent observed frequencies among people who make their decision without thinking about CDT vs. EDT. However, intuitively they realize that the correlation in the data doesn’t hold up in this naive way.

 

This would also explain why people are more open to EDT’s recommendation in cases where the causal structure is analogous to that in the smoking lesion, but tickle defenses (or, more generally, ways in which a stated probability distribution could differ from the real/intuitive one) don’t apply, e.g. the psychopath button, betting on the past, or the coin flip creation problem.

 

I’d be interested in your opinions. I also wonder whether this has already been discussed elsewhere.

Acknowledgment

Discussions with Johannes Treutlein informed my view on this topic.

Comment author: orthonormal 22 April 2012 04:36:06PM 16 points [-]

If I'd been one of the participants on Hofstadter's original game, I'd have answered him thusly:

"I know where you're going with this experiment— you want all of us to realize that our reasoning is roughly symmetrical, and that it's better if we all cooperate than if we all defect. And if I were playing against a bunch of copies of myself, then I'd cooperate without hesitation.

However, if I were playing against a bunch of traditional game theorists, then the sensible thing is to defect, since I know that they're not going to reason by this line of thought, and so the symmetry is broken. Even if I were playing against a bunch of people who'd cooperate because they think that's more moral, I ought to defect (if I'm acting according to my own self-interest), because they're not thinking in these terms either.

So what I really need to do is to make my best guess about how many of the participants are thinking in this reflexive sort of way, and how many are basing their decisions on completely different lines of thought. And then my choice would in effect be choosing for that block of people and not for the rest, and so I'd need to make my best guess whether it's better for me if I (and the rest of that block) choose to cooperate or if we choose to defect. That depends on how large that block is, how many of the others I expect to cooperate vs. defect, and on the payoff matrix."

At the time he wrote it, the correct choice would have been to defect, because as Hofstadter noted, none of his friends (as brilliant as they were) took anything like that reflexive line of thought. If it were done now, among a group of Less Wrong veterans, I might be convinced to cooperate.

Comment author: Caspar42 01 February 2017 10:27:31AM 0 points [-]

At the time he wrote it, the correct choice would have been to defect, because as Hofstadter noted, none of his friends (as brilliant as they were) took anything like that reflexive line of thought. If it were done now, among a group of Less Wrong veterans, I might be convinced to cooperate.

I would advocate the opposite: Imagine you have never thought about Newcomb-like scenarios before. Therefore, you also don't know how others would act in such problems. Now, you come up with this interesting line of thought about determining the others' choices or correlating with them. Because you are the only data point, your decision should give you a lot of evidence about what others might do, i.e. about whether they will come up with the idea at all and behave in abidance with it.

Now, contrast this with playing the game today. You may have already read studies showing that most philosophers use CDT, that most people one-box in Newcomb's problem, that LWers tend to cooperate. If anything, your decision now gives you less information about what the others will do.

View more: Next