(Part 1 of the Sequence on Applied Causal Inference) 

 

In this sequence, I am going to present a theory on how we can learn about causal effects using observational data.  As an example, we will imagine that you have collected information on a large number of Swedes - let us call them Sven, Olof, Göran,  Gustaf, Annica,  Lill-Babs, Elsa and Astrid. For every Swede, you have recorded data on their gender, whether they smoked or not, and on whether they got cancer during the 10-years of follow-up.   Your goal is to use this dataset to figure out whether smoking causes cancer.   

We are going to use the letter A as a random variable to represent whether they smoked. A can take the value 0 (did not smoke) or 1 (smoked).  When we need to talk about the specific values that A can take, we sometimes use lower case a as a placeholder for 0 or 1.    We use the letter Y as a random variable that represents whether they got cancer, and L to represent their gender. 

The data-generating mechanism and the joint distribution of variables

Imagine you are looking at this data set:

ID	L	A	Y
Name	Sex	Did they smoke?	Did they get cancer?
Sven	Male	Yes	Yes
Olof	Male	No	Yes
Göran	Male	Yes	Yes
Gustaf	Male	No	No
Annica	Female	Yes	Yes
Lill-Babs	Female	Yes	No
Elsa	Female	Yes	No
Astrid	Female	No	No

 

 

This table records information about the joint distribution of the variables L, A and Y.  By looking at it, you can tell that 1/4 of the Swedes were men who smoked and got cancer, 1/8 were men who did not smoke and got cancer, 1/8 were men who did not smoke and did not get cancer etc.  

You can make all sorts of statistics that summarize aspects of the joint distribution.  One such statistic is the correlation between two variables.  If "sex" is correlated with "smoking", it means that if you know somebody's sex, this gives you information that makes it easier to predict whether they smoke.   If knowing about an individual's sex gives no information about whether they smoked, we say that sex and smoking are independent.  We use the symbol ∐ to mean independence. 

When we are interested in causal effects, we are asking what would happen to the joint distribution if we intervened to change the value of a variable.  For example, how many Swedes would get cancer in a hypothetical world where you intervened to make sure they all quit smoking?  

In order to answer this, we have to ask questions about the data generating mechanism. The data generating mechanism is the algorithm that assigns value to the variables, and therefore creates the joint distribution. We will think of the data as being generated by three different algorithms: One for L, one for A and one for Y.    Each of these algorithms takes the previously assigned variables as input, and then outputs a value.    

Questions about the data generating mechanism include “Which variable has its value assigned first?”,  “Which variables from the past (observed or unobserved) are used as inputs” and “If I change whether someone smokes, how will that change propagate to other variables that have their value assigned later".    The last of these questions can be rephrased as "What is the causal effect of smoking”.    

The basic problem of causal inference is that the relationship between the set of possible data generating mechanisms, and the joint distribution of variables, is many-to-one:   For any correlation you observe in the dataset, there are many possible sets of algorithms for L, A and Y that could all account for the observed patterns. For example, if you are looking at a correlation between cancer and smoking, you can tell a story about cancer causing people to take up smoking, or a story about smoking causing people to get cancer, or a story about smoking and cancer sharing a common cause.  

An important thing to note is that even if you have data on absolutely everyone, you still would not be able to distinguish between the possible data generating mechanisms. The problem is not that you have a limited sample. This is therefore not a statistical problem.  What you need to answer the question, is not more people in your study, but a priori causal information.  The purpose of this sequence is to show you how to reason about what prior causal information is necessary, and how to analyze the data if you have measured all the necessary variables. 

Counterfactual Variables and "God's Table":

The first step of causal inference is to translate the English language research question «What is the causal effect of smoking» into a precise, mathematical language.  One possible such language is based on counterfactual variables.  These counterfactual variables allow us to encode the concept of “what would have happened if, possibly contrary to fact, the person smoked”.

We define one counterfactual variable called Y^a=1 which represents the outcome in the person if he smoked, and another counterfactual variable called Y^a=0 which represents the outcome if he did not smoke. Counterfactual variables such as Y^a=0 are mathematical objects that represent part of the data generating mechanism:  The variable tells us what value the mechanism would assign to Y, if we intervened to make sure the person did not smoke. These variables are columns in an imagined dataset that we sometimes call “God’s Table”:

 

ID	A	Y	Ya=1	Ya=0
	Smoking	Cancer	Whether they would have got cancer if they smoked	Whether they would have got cancer if they didn't smoke
Sven	1	1	1	1
Olof	0	1	0	1
Göran	1	1	1	0
Gustaf	0	0	0	0

 

 

 

Let us start by making some points about this dataset.  First, note that the counterfactual variables are variables just like any other column in the spreadsheet.   Therefore, we can use the same type of logic that we use for any other variables.  Second, note that in our framework, counterfactual variables are pre-treatment variables:  They are determined long before treatment is assigned. The effect of treatment is simply to determine whether we see Y^a=0 or Y^a=1 in this individual.

If you had access to God's Table, you would immediately be able to look up the average causal effect, by comparing the column Y^a=1 to the column Y^a=0.  However, the most important point about God’s Table is that we cannot observe Y^a=1 and Y^a=0. We only observe the joint distribution of observed variables, which we can call the “Observed Table”:

 

ID	A	Y
Sven	1	1
Olof	0	1
Göran	1	1
Gustaf	0	0

 

 

The goal of causal inference is to learn about God’s Table using information from the observed table (in combination with a priori causal knowledge).  In particular, we are going to be interested in learning about the distributions of Y^a=1 and Y^a=0, and in how they relate to each other.  

 

Randomized Trials

The “Gold Standard” for estimating the causal effect, is to run a randomized controlled trial where we randomly assign the value of A.   This study design works because you select one random subset of the study population where you observe Y^a=0, and another random subset where you observe Y^a=1.   You therefore have unbiased information about the distribution of both Y^a=0and of Y^a=1. 

An important thing to point out at this stage is that it is not necessary to use an unbiased coin to assign treatment, as long as your use the same coin for everyone.   For instance, the probability of being randomized to A=1 can be 2/3.  You will still see randomly selected subsets of the distribution of both Y^a=0 and Y^a=1, you will just have a larger number of people where you see Y^a=1.     Usually, randomized trials use unbiased coins, but this is simply done because it increases the statistical power. 

Also note that it is possible to run two different randomized controlled trials:  One in men, and another in women.  The first trial will give you an unbiased estimate of the effect in men, and the second trial will give you an unbiased estimate of the effect in women.  If both trials used the same coin, you could think of them as really being one trial. However, if the two trials used different coins, and you pooled them into the same database, your analysis would have to account for the fact that in reality, there were two trials. If you don’t account for this, the results will be biased.  This is called “confounding”. As long as you account for the fact that there really were two trials, you can still recover an estimate of the population average causal effect. This is called “Controlling for Confounding”.

In general, causal inference works by specifying a model that says the data came from a complex trial, ie, one where nature assigned a biased coin depending on the observed past.  For such a trial, there will exist a valid way to recover the overall causal results, but it will require us to think carefully about what the correct analysis is. 

Assumptions of Causal Inference

We will now go through in some more detail about why it is that randomized trials work, ie , the important aspects of this study design that allow us to infer causal relationships, or facts about God’s Table, using information about the joint distribution of observed variables.  

We will start with an “observed table” and build towards “reconstructing” parts of God’s Table.  To do this, we will need three assumptions: These are positivity, consistency and (conditional) exchangeability:

ID	A	Y
Sven	1	1
Olof	0	1
Göran	1	1
Gustaf	0	0

 

 

 

Positivity

Positivity is the assumption that any individual has a positive probability of receiving all values of the treatment variable:   Pr(A=a) > 0 for all values of a.  In other words, you need to have both people who smoke, and people who don't smoke.  If positivity does not hold, you will not have any information about the distribution of Y^a for that value of a, and will therefore not be able to make inferences about it.

We can check whether this assumption holds in the sample, by checking whether there are people who are treated and people who are untreated. If you observe that in any stratum, there are individuals who are treated and individuals who are untreated, you know that positivity holds.  

If we observe a stratum where no individuals are treated (or no individuals are untreated), this can be either for statistical reasons (your randomly did not sample them) or for structural reasons (individuals with these covariates are deterministically never treated).  As we will see later, our models can handle random violations, but not structural violations.

In a randomized controlled trial, positivity holds because you will use a coin that has a positive probability of assigning people to either arm of the trial.

Consistency

The next assumption we are going to make is that if an individual happens to have treatment (A=1), we will observe the counterfactual variable Y^a=1 in this individual. This is the observed table after we make the consistency assumption:

ID	A	Y	Ya=1	Ya=0
Sven	1	1	1	*
Olof	0	1	*	1
Göran	1	1	1	*
Gustaf	0	0	*	0

 

 

 

 

 Making the consistency assumption got us half the way to our goal.  We now have a lot of information about Y^a=1 and Y^a=0. However, half of the data is still missing.

Although consistency seems obvious, it is an assumption, not something that is true by definition.  We can expect the consistency assumption to hold if we have a well-defined intervention (ie, the intervention is a well-defined choice, not an attribute of the individual), and there is no causal interference (one individual’s outcome is not affected by whether another individual was treated).

Consistency may not hold if you have an intervention that is not well-defined:  For example, there may be multiple types of cigarettes. When you measure Y^a=1 in people who smoked, it will actually be a composite of multiple counterfactual variables:  One for people who smoked regular cigarettes (let us call that Y^a=1*) and another for people who smoked e-cigarettes (let us call that Y^a=1#)_.    Since you failed to specify whether you are interested in the effect of regular cigarettes or e-cigarettes, the construct Y^a=1 is a composite without any meaning, and people will be unable to use your results to predict the consequences of their actions.

Exchangeability

To complete the table, we require an additional assumption on the nature of the data. We call this assumption “Exchangeability”.  One possible exchangeability assumption is “Y^a=0 ∐ A and Y^a=1 ∐ A”.   This is the assumption that says “The data came from a randomized controlled trial”. If this assumption is true, you will observe a random subset of the distribution of Y^a=0 in the group where A=0, and a random subset of the distribution of Y^a=1 in the group where A=1.

Exchangeability is a statement about two variables being independent from each other. This means that having information about either one of the variables will not help you predict the value of the other.  Sometimes, variables which are not independent are "conditionally independent".  For example, it is possible that knowing somebody's race helps you predict whether they enjoy eating Hakarl, an Icelandic form of rotting fish.  However, it is also possible that this is just a marker for whether they were born in the ethnically homogenous Iceland. In such a situation, it is possible that once you already know whether somebody is from Iceland, also knowing their race gives you no additional clues as to whether they will enjoy Hakarl.  In this case, the variables "race" and "enjoying hakarl" are conditionally independent, given nationality. 

The reason we care about conditional independence is that sometimes you may be unwilling to assume that marginal exchangeability Y^a=1 ∐ A holds, but you are willing to assume conditional exchangeability Y^a=1 ∐ A  | L.  In this example, let L be sex.  The assumption then says that you can interpret the data as if it came from two different randomized controlled trials: One in men, and one in women. If that is the case, sex is a "confounder". (We will give a definition of confounding in Part 2 of this sequence. )

If the data came from two different randomized controlled trials, one possible approach is to analyze these trials separately. This is called “stratification”.  Stratification gives you effect measures that are conditional on the confounders:  You get one measure of the effect in men, and another in women.  Unfortunately, in more complicated settings, stratification-based methods (including regression) are always biased. In those situations, it is necessary to focus the inference on the marginal distribution of Y^a.

Identification

If marginal exchangeability holds (ie, if the data came from a marginally randomized trial), making inferences about the marginal distribution of Y^a is easy: You can just estimate E[Y^a] as E [Y|A=a].

However, if the data came from a conditionally randomized trial, we will need to think a little bit harder about how to say anything meaningful about E[Y^a]. This process is the central idea of causal inference. We call it “identification”:  The idea is to write an expression for the distribution of a counterfactual variable, purely in terms of observed variables.  If we are able to do this, we have sufficient information to estimate causal effects just by looking at the relevant parts of the joint distribution of observed variables.

The simplest example of identification is standardization.  As an example, we will show a simple proof:

Begin by using the law of total probability to factor out the confounder, in this case L:

·         E(Y^a) = Σ  E(Y^a|L= l) * Pr(L=l)    (The summation sign is over l)

We do this because we know we need to introduce L behind the conditioning sign, in order to be able to use our exchangeability assumption in the next step:   Then,  because Y^a  ∐ A | L,  we are allowed to introduce A=a behind the conditioning sign:

·         E(Y^a) =  Σ  E(Y^a|A=a, L=l) * Pr(L=l)

Finally, use the consistency assumption:   Because we are in the stratum where A=a in all individuals, we can replace Y^a by Y

·         E(Y^a) = Σ E(Y|A=a, L=l) * Pr (L=l)

 

We now have an expression for the counterfactual in terms of quantities that can be observed in the real world, ie, in terms of the joint distribution of A, Y and L. In other words, we have linked the data generating mechanism with the joint distribution – we have “identified”  E(Y^a).  We can therefore estimate E(Y^a)

This identifying expression is valid if and only if L was the only confounder. If we had not observed sufficient variables to obtain conditional exchangeability, it would not be possible to identify the distribution of Y^a : there would be intractable confounding.

Identification is the core concept of causal inference: It is what allows us to link the data generating mechanism to the joint distribution, to something that can be observed in the real world. 

 

The difference between epidemiology and biostatistics

Many people see Epidemiology as «Applied Biostatistics».  This is a misconception. In reality, epidemiology and biostatistics are completely different parts of the problem.  To illustrate what is going on, consider this figure:

 

 

The data generating mechanism first creates a joint distribution of observed variables.  Then, we sample from the joint distribution to obtain data. Biostatistics asks:  If we have a sample, what can we learn about the joint distribution?  Epidemiology asks:  If we have all the information about the joint distribution , what can we learn about the data generating mechanism?   This is a much harder problem, but it can still be analyzed with some rigor.

Epidemiology without Biostatistics is always impossible:  It would not be possible to learn about the data generating mechanism without asking questions about the joint distribution. This usually involves sampling.  Therefore, we will need good statistical estimators of the joint distribution.

Biostatistics without Epidemiology is usually pointless:  The joint distribution of observed variables is simply not interesting in itself. You can make the claim that randomized trials is an example of biostatistics without epidemiology.  However, the epidemiology is still there. It is just not necessary to think about it, because the epidemiologic part of the analysis is trivial

Note that the word “bias” means different things in Epidemiology and Biostatistics.  In Biostatistics, “bias” is a property of a statistical estimator:  We talk about whether ŷ is a biased estimator of E(Y |A).   If an estimator is biased, it means that when you use data from a sample to make inferences about the joint distribution in the population the sample came from, there will be a systematic source of error.

In Epidemiology, “bias” means that you are estimating the wrong thing:  Epidemiological bias is a question about whether E(Y|A) is a valid identification of E(Y^a).   If there is epidemiologic bias, it means that you estimated something in the joint distribution, but that this something does not answer the question you were interested in.    

These are completely different concepts. Both are important and can lead to your estimates being wrong. It is possible for a statistically valid estimator to be biased in the epidemiologic sense, and vice versa.   For your results to be valid, your estimator must be unbiased in both senses.

 

As previously discussed, on June 6th I received a message from jackk, a Trike Admin. He reported that the user Jiro had asked Trike to carry out an investigation to the retributive downvoting that Jiro had been subjected to. The investigation revealed that the user Eugine_Nier had downvoted over half of Jiro's comments, amounting to hundreds of downvotes.

I asked the community's guidance on dealing with the issue, and while the matter was being discussed, I also reviewed previous discussions about mass downvoting and looked for other people who mentioned being the victims of it. I asked Jack to compile reports on several other users who mentioned having been mass-downvoted, and it turned out that Eugine was also overwhelmingly the biggest downvoter of users David_Gerard, daenarys, falenas108, ialdabaoth, shminux, and Tenoke. As this discussion was going on, it turned out that user Ander had also been targeted by Eugine.

I sent two messages to Eugine, requesting an explanation. I received a response today. Eugine admitted his guilt, expressing the opinion that LW's karma system was failing to carry out its purpose of keeping out weak material and that he was engaged in a "weeding" of users who he did not think displayed sufficient rationality.

Needless to say, it is not the place of individual users to unilaterally decide that someone else should be "weeded" out of the community. The Less Wrong content deletion policy contains this clause:

Harrassment of individual users.

If we determine that you're e.g. following a particular user around and leaving insulting comments to them, we reserve the right to delete those comments. (This has happened extremely rarely.)

Although the wording does not explicitly mention downvoting, harassment by downvoting is still harassment. Several users have indicated that they have experienced considerable emotional anguish from the harassment, and have in some cases been discouraged from using Less Wrong at all. This is not a desirable state of affairs, to say the least.

I was originally given my moderator powers on a rather ad-hoc basis, with someone awarding mod privileges to the ten users with the highest karma at the time. The original purpose for that appointment was just to delete spam. Nonetheless, since retributive downvoting has been a clear problem for the community, I asked the community for guidance on dealing with the issue. The rough consensus of the responses seemed to authorize me to deal with the problem as I deemed appropriate.

The fact that Eugine remained quiet about his guilt until directly confronted with the evidence, despite several public discussions of the issue, is indicative of him realizing that he was breaking prevailing social norms. Eugine's actions have worsened the atmosphere of this site, and that atmosphere will remain troubled for as long as he is allowed to remain here.

Therefore, I now announce that Eugine_Nier is permanently banned from posting on LessWrong. This decision is final and will not be changed in response to possible follow-up objections.

Unfortunately, it looks like while a ban prevents posting, it does not actually block a user from casting votes. I have asked jackk to look into the matter and find a way to actually stop the downvoting. Jack indicated earlier on that it would be technically straightforward to apply a negative karma modifier to Eugine's account, and wiping out Eugine's karma balance would prevent him from casting future downvotes. Whatever the easiest solution is, it will be applied as soon as possible.

EDIT 24 July 2014: Banned users are now prohibited from voting.

At the recent Tel Aviv meetup, after a discussion of the open problems in the field of FAI, we reached the conclusion that the problem of logical uncertainty is one of the most major of the problems open today. In this post I will try to give a few insights I had on this problem, which can be thought of as the problem of constructing a (non-degenerate) probability measure over the set of the statements of an arbitrary logical system.

To clarify my goal: I'm trying to make a Solomonoff-like system for assigning probabilities to logical statements. For reasons much like the reasons that Solomonoff Induction is uncomputable, this system will be uncomputable as well. This puts certain limits to it's usefulness, but it's certainly a start. Solomonoff Induction is very useful, if only as a limiting case of computable processes. I believe the system I present here has some of that same value when it comes to the problem of logical uncertainty.

The obvious tack to take is one involving proof-lengths: It is reasonable to say that a sentence that is harder to prove should have a lower measure.

Let's start with a definition:
For each provably true statement φ , let ProofLength(φ) be the length of φ's shortest proof. For all unprovable yet true statements, let ProofLength(φ) be ∞.

Therefore, if we have some probability measure P over true statements in our system, we want P to be monotonic decreasing in regards to proof length. (e.g. P(φ₁)<P(φ₂) ⇔ ProofLength(φ₁)>ProofLength(φ₂))

For obvious reasons, we want to assign probability 1 to our axioms. As an axiom always has a proof of length one (the mere statement of the axiom itself, without any previous statements that it is derived from), we want the property P(φ) = 1 ⇔ ProofLength(φ) = 1.

Lastly, for statements that are unprovable, we have to assign a probability of ½. Why? Let φ be an unprovable statement. Because P is a probability measure, P(φ)+P(~φ) = 1. Exactly one of φ or ~φ is true, but as they are unprovable, we have no way of knowing, even in theory, which is which. Thus, by symmetry, P(φ)=P(~φ)=½.

Given these desiderata, we will see what measure P we can construct that follows them.

For each true statement φ I will define P(φ) to be 2^{-ProofLength(φ)}+2^-1. This seems at first rather arbitrary, but it matches the desiderata in a fairly elegant way. This P is monotonic with regards to the proof length, as we demanded, and it correctly assigns a probability of 1 to our axioms. It also correctly assigns probability ½ to unprovable statements.

For this to be a probability measure over the set of all statements, we still need to define it on the false statements as well. This is trivial, as we can define P(φ)=1-P(~φ) for all false statements φ. This gives us all the properties we might demand of a logical probability measure that is based on proof lengths:

• Statements that can be easily proved true are given more measure than those that are hard (or even impossible) to prove to be true.
• And in reverse for false statements: Statements that can be easily proved to be false are given a lower measure than those that are hard (or even impossible) to prove to be false.
• Specifically, it assigns probability 1 to axioms (and 0 to the negations of axioms.)

I have no idea if this is the best way to construct a logical probability measure, but it seems like a start. This seems like as decent a way as any to assign priors to statements of logical probability.

That handles priors, but it doesn't seem to give an easy way to update on evidence. Obviously, once you prove something is true or false, you want its probability to rise to 1 or drop to 0 accordingly. Also, if you take some steps in the direction of proving something to be true or false, you want its probability to rise or fall accordingly.

To take a logical sentence and just blindly assign it the probability described above, ignoring everything else, is just as bad as taking the Solomonoff prior for the probability of a regular statement (in the standard system of probability) , and refusing to update away from that that. The role of the P described above is very much like that of the role of the Solomonoff prior in normal inductive Bayesian reasoning. This is nice, and perhaps it is a step forwards, but it isn't a system that can be used by itself for making decisions.

Luckily, there is a way to update on information in Solomonoff induction. You merely "slice off" the worlds that are now impossible given the new evidence, and recalculate. (It can be proven that doing so is equivalent to updating using Bayes' Theorem.)

To my delight, I found that something similar is possible with this system too! This is the truly important insight here, as this gives (for the first time, so far as I know) a method for actually updating on logical probabilities, so that as you advance towards a proof, your probability estimate of that sentence being true approaches 1, but only reaches 1 once you have a full proof.

What you do is exactly the same as in Solomonoff induction: Every time you prove something, you update by recalculating the probability of every statement, given that you now now the newly proven thing. Informally, the reasoning is like this: If you proved a statement, that means you know it with probability 1, or in other words, it can be considered as effectively a new axiom. So add it to your axioms, and you will get your updated probability!

In more technical terms, in a given logical system S, P(φ|ψ) will be defined as the P(φ) I described above, just in the logical system S+ψ, rather than is S. This obeys all the properties we want out of an update on evidence: An update increases the probability we assign to a statement that we proved part of, or proved a lemma for, or whatever, and decreases the probability we assign to a statement that we proved part of its negation, or a lemma for the proof of its negation, or the like.

This is not a complete theory of logical uncertainty, but it could be a foundation. It certainly includes some insights I haven't seen before, or at least that I haven't seen explained in these terms. In the upcoming weeks the Tel Aviv meetup group is planning to do a MIRIx workshop on the topic of logical uncertainty, and we hope to make some real strides in it. Perhaps we will expand on this, or perhaps we will come up with some other avenue of attack. If we can give logical uncertainty a formal grounding, that will be a fairly major step. After all, the black box of logical uncertainty sits right at the heart of most attempts to advance AGI, and at the moment it is merely handwaved away in most applications. But eventually it needs an underpinning, and that is what we are aiming at.

Why Talk to Philosophers? Part I. by philosopher of science Wayne Myrvold.

See also Sean Carroll's own blog entry, Physicists Should Stop Saying Silly Things about Philosophy. 

Sean classifies the disparaging comments physicists make about philosophy as follows: "Roughly speaking, physicists tend to have three different kinds of lazy critiques of philosophy: one that is totally dopey, one that is frustratingly annoying, and one that is deeply depressing". Specifically:

• “Philosophy tries to understand the universe by pure thought, without collecting experimental data.”
• “Philosophy is completely useless to the everyday job of a working physicist.”
• “Philosophers care too much about deep-sounding meta-questions, instead of sticking to what can be observed and calculated.”

He counters each argument presented.

---

Personally, I am underwhelmed, since he does not address the point of view that philosophy is great at asking interesting questions but lousy at answering them. Typically, an interesting answer to a philosophical question requires first recasting it in a falsifiable form, so that is becomes a natural science question, be it physics, cognitive sciences, AI research or something else. This is locally known as hacking away at the edges. Philosophical questions don't have philosophical answers.

 

Author's Note: Please let me know in the comments exactly what important background material I have missed, and exactly what I have misunderstood, and please try not to mind that everything here is written in the academic voice.

Abstract: Timeless Decision Theory often seems like the correct way to handle many game-theoretical dilemmas, but has not quite been satisfactorily formalized and still handles certain problems the wrong way.  We present an intuition that helps us extend Causal Decision Theory towards Timeless Decision Theory while adding rigor, and then formalize this intuition.  Along the way, we describe how this intuition can guide both us and programmed agents in various Newcomblike games.

Introduction

One day, a Time Lord called Omega drops out of the sky, walks up to me on the street, and places two boxes in front of me.  One of these is opaque, the other is transparent and contains $1000.  He tells me I can take either the opaque box alone, or both boxes, but that if and only if he predicted using his Time Lord Science I would take just the opaque box, it contains $1,000,000.  He then flies away back to the his home-world of Gallifrey.  I know that whatever prediction he made was/will be correct, because after all he is a Time Lord.

The established, gold-standard algorithm of Causal Decision Theory fails to win the maximum available sum of money on this problem, just as it fails on a symmetrical one-shot Prisoner's Dilemma.  In fact, as human beings, we can say that CDT fails miserably, because while a programmed agent goes "inside the game" and proceeds to earn a good deal less money than it could, we human observers are sitting outside, carefully drawing outcome tables that politely inform us of just how much money our programmed agents are leaving on the table.  While purely philosophical controversies abound in the literature about the original Newcomb's Problem, it is generally obvious from our outcome tables in the Prisoners' Dilemma that "purely rational" CDT agents would very definitely benefit by cooperating, and that actual human beings asked to play the game calculate outcomes as if forming coalitions rather than as if maximizing personal utility -- thus cooperating and winning.  Even in the philosophical debates, it is generally agreed that one-boxers in Newcomb's Problem are, in fact, obtaining more money.

While some have attempted to define rationality as the outputs of specific decision algorithms, we hold with the school of thought that rationality means minimizing regret: a rational agent should select its decision algorithms in order to win as much as it will know it could have won ex-post-facto.  Failing perfection, this optimum should be approximated as closely as possible.

Yudkowsky's Timeless Decision Theory approaches this problem by noting that many so-called decisions are actually outcomes from concurrent or separated instantiations of a single algorithm, that Timeless Decision Theory itself is exactly such an algorithm, and that many decisions (that actually are decisions in the sense that the algorithm deciding them is a utility-maximizing decision-theory) are acausally, timelessly connected.  Agents running TDT will decide not as if they are determining one mere assignment to one mere variable in a causal graph but as if they're determining the output of the computation they implement, and thus of every logical node in the entire graph derived from their computation.  However, it still has some kinks to work out:

Yudkowsky (2010) shows TDT succeeding in the original Newcomb’s problem. Unfortunately, deciding exactly when and where to put the logical nodes, and what conditional probabilities to place on them, is not yet an algorithmic process.

How would TDT look if instantiated in a more mature application? Given a very large and complex network, TDT would modify it in the following way: It would investigate each node, noting the ones that were results of instantiated calculations. Then it would collect these nodes into groups where every node in a group was the result of the same calculation. (Recall that we don’t know what the result is, just that it comes from the same calculation.) For each of these groups, TDT would then add a logical node representing the result of the abstract calculation, and connect it as a parent to each node in the group.  Priors over possible states of the logical nodes would have to come from some other reasoning process, presumably the one that produces causal networks in the first place. Critically, one of these logical nodes would be the result of TDT’s own decision process in this situation. TDT would denote that as the decision node and use the resulting network to calculate the best action by equation 1.1.

The bolding is added by the present authors, as it highlights the issue we intend to address here.  Terms like "timeless" and "acausal" have probably caused more confusion around Timeless Decision Theory than any other aspect of what is actually an understandable and reasonable algorithm.  I will begin by presenting a clearer human-level intuition behind the correct behavior in Newcomb's Problem and the Prisoner's Dilemma, and will then proceed to formalize that intuition in Coq and apply it to sketch a more rigorously algorithmic Timeless Decision Theory.  The formalization of this new intuition avoids problems of infinite self-reference or infinite recursion in reasoning about the algorithms determining decisions of oneself or others.

Timeless decisions are actually entangled with each-other

The kind of apparent retrocausality present in Newcomb's Problem makes no intuitive sense whatsoever.  Not only our intuitions but all our knowledge of science tell us that (absent the dubious phenomenon of closed timelike curves) causal influences always and only flow from the past to the future, never the other way around.  Nonetheless, in the case of Newcomb-like problems, it has been seriously argued that:

the Newcomb problem cannot but be retrocausal, if there is genuine evidential dependence of the predictor’s behaviour on the agent’s choice, from the agent’s point of view.

We do not believe in retrocausality, at least not as an objective feature of the world.  Any subjectively apparent retrocausality, we believe, must be some sort of illusion that reduces to genuine, right-side-up causality.  Timeless or acausal decision-making resolves the apparent retrocausality by noticing that different "agents" in Newcomblike problems are actually reproductions of the same algorithm, and that they can thus be logically correlated without any direct causal link.

We further prime our intuitions about Newcomb-like problems with the observation that CDT-wielding Newcomb players who bind themselves to a precommitment to one-box before Omega predicts their actions will win the $1,000,000:

At t = 0 you can take a pill that turns you into a “one boxer”.  The pill will lead the mad scientist to predict (at t = ½) that you will take one box, and so will cause you to receive £1,000,000 but will also cause you to leave a free £1,000 on the table at t = 1.  CDT tells you to take the pill at t = 0: it is obviously the act, among those available at t = 0, that has the best overall causal consequences.

The "paradox", then, lies in how the CDT agent comes to believe that their choice is completely detached from which box contains how much money, when in fact Omega's prediction of their choice was accurate, and directly caused Omega to place money in boxes accordingly, all of this despite no retrocausality occurring.  Everything makes perfect sense prior to Omega's prediction.

What, then, goes wrong with CDT?  CDT agents will attempt to cheat against Omega: to be predicted as a one-boxer and then actually take both boxes.  If given a way to obtain more money by precommitting to one-boxing, they will do so, but will subsequently feel regret over having followed their precommitment and "irrationally" taken only one box when both contained money.  They may even begin to complain about the presence or absence of free will, as if this could change the game and enable their strategy to actually work.

When we cease such protestations and accept that CDT behaves irrationally, the real question becomes: which outcomes are genuinely possible in Newcomb's Problem, which outcomes are preferable, and why does CDT fail to locate these?

Plainly if we believe that Omega has a negligible or even null error rate, then in fact only two outcomes are possible:

• Our agent is predicted to take both boxes, and does so, receiving only $1000 since Omega has not filled the opaque box.
• Our agent is predicted to take the opaque box, which Omega fills, and the agent does take the opaque box, receiving $1,000,000.

 

Plainly, $1 million is a greater sum than $1000, and the former outcome state is thus preferable to the latter.  We require an algorithm that can search out and select this outcome based on general principles, in any Newcomblike game rather than based on special-case heuristics.

Whence, then, a causal explanation of what to do?  The authors' intuition was sparked by a bit of reading about the famously "spooky" phenomenon of quantum entanglement, also sometimes theorized to involve retrocausality.  Two particles interact and become entangled; from then on, their quantum states will remain correlated until measurement collapses the wave-function of one particle or the other.  Neither party performing a measurement will ever be able to tell which measurement took place first in time, but both measurements will always yield correlated results.  This occurs despite the fact that quantum theory is confirmed to have no hidden variables, and even when general relativity's light-speed limit on the transmission of information prevents the entangled particles from "communicating" any quantum information.  A paradox is apparent and most people find it scientifically unaesthetic.

In reality, there is no paradox at all.  All that has happened is that the pair of particles are in quantum superposition together: their observables are mutually governed by a single joint probability distribution.  The measured observable states do not go from "randomized" to "correlated" as the measurement is made.  The measurement only "samples" a single classical outcome governing both particles from the joint probability distribution that is actually there.  The joint probability distribution was actually caused by the 100% local and slower-than-light interaction that entangled the two particles in the first place.

Likewise for Newcomb's Problem in decision theory.  As the theorists of precommitment had intuited, the outcome is not actually caused when the CDT agent believes itself to be making a decision.  Instead, the outcome was caused when Omega measured the agent and predicted its choice ahead of time: the state of the agent at this time causes both Omega's prediction and the agent's eventual action.

We thus develop an intuition that like a pair of particles, the two correlated decision processes behind Omega's prediction and behind the agent's "real" choice are in some sense entangled: correlated due to a causal interaction in their mutual past.  All we then require to win at Newcomb's Problem is a rigorous conception of such entanglement and a way of handling it algorithmically to make regret-minimizing decisions when entangled.

Formalized decision entanglement

Let us begin by assuming that an agent can be defined as a function from a set of Beliefs and a Decision to an Action.  There will not be very much actual proof-code given here, and what is given was written in the Coq proof assistant.  The proofs, short though they be, were thus mechanically checked before being given here; "do try this at home, kids."

Definition Agent (Beliefs Decision Action: Type) : Type := Beliefs -> Decision -> Action.

We can then broaden and redefine our definition of decision entanglement as saying, essentially, "Two agents are entangled when either one of them would do what the other is doing, were they to trade places and thus beliefs but face equivalent decisions."  More simply, if a certain two agents are entangled over a certain two equivalent decisions, any differences in what decisions they actually make arise from differences in beliefs.

Inductive entangled {Beliefs Decision Action} (a1 a2: Agent Beliefs Decision Action) d1 d2 :=

  | ent : (forall (b: Beliefs), a1 b d1 = a2 b d2) -> d1 = d2 -> entangled a1 a2 d1 d2.

This kind of entanglement can then, quite quickly, be shown to be an equivalence relation, thus partitioning the set of all logical nodes in a causal graph into Yudkowsky's "groups where every node in a group was the result of the same calculation", with these groups being equivalence classes.

Theorem entangled_reflexive {B D A} : forall (a: Agent B D A) d,

  entangled a a d d.

Proof.

  intros.

  constructor.

  intros. reflexivity. reflexivity.

Qed.

Theorem entangled_symmetric {B D A}: forall (a1 a2: Agent B D A) d1 d2,

  entangled a1 a2 d1 d2 ->

  entangled a2 a1 d2 d1.

Proof.

  intros.

  constructor;

  induction H;

    intros; symmetry.

  apply e. apply e0.

Qed.

Theorem entangled_transitive {B D A}: forall (a1 a2 a3: Agent B D A) d1 d2 d3,

  entangled a1 a2 d1 d2 ->

  entangled a2 a3 d2 d3 ->

  entangled a1 a3 d1 d3.

Proof.

  intros a1 a2 a3 d1 d2 d3 H12 H23.

  constructor;

    induction H12; induction H23; subst.

  intros b. rewrite e. rewrite e1.

  reflexivity. reflexivity.

Qed.

Actually proving that this relation holds simply consists of proving that two agents given equivalent decisions will always decide upon the same action (similar to proving program equilibrium) no matter what set of arbitrary beliefs is given them -- hence the usage of a second-order forall.  Proving this does not require actually running the decision function of either agent.  Instead, it requires demonstrating that the abstract-syntax trees of the two decision functions can be made to unify, up to the renaming of universally-quantified variables.  This is what allows us to prove the entanglement relation's symmetry and transitivity: our assumptions give us rewritings known to hold over the universally-quantified agent functions and decisions, thus letting us employ unification as a proof tool without knowing what specific functions we might be handling.

Thanks to employing the unification of syntax trees rather than the actual running of algorithms, we can conservatively extend Causal Decision Theory with logical nodes and entanglement to adequately handle timeless decision-making, without any recourse to retrocausality nor to the potentially-infinitely loops of Sicilian Reasoning.  (Potential applications of timeless decision-making to win at Ro Sham Bo remain an open matter for the imagination.)

Decision-theoretically, since our relation doesn't have to know anything about the given functions other than (forall (b: Beliefs), a1 b d = a2 b d), we can test whether our relationship holds over any two logical/algorithm nodes in an arbitrary causal graph, since all such nodes can be written as functions from their causal inputs to their logical output.  We thus do not need a particular conception of what constitutes an "agent" in order to make decisions rigorously: we only need to know what decision we are making, and where in a given causal graph we are making it.  From there, we can use simple (though inefficient) pairwise testing to find the equivalence class of all logical nodes in the causal graph equivalent to our decision node, and then select a utility-maximizing output for each of those nodes using the logic of ordinary Causal Decision Theory.

The slogan of a Causal Decision Theory with Entanglement (CDT+E) can then be summed up as, "select the decision which maximizes utility for the equivalence class of nodes to which I belong, with all of us acting and exerting our causal effects in concert, across space and time (but subject to our respective belief structures)."

 

The performance of CDT with entanglement on common problems

 

While we have not yet actually programmed a software agent with a CDT+E decision algorithm over Bayesian causal graphs (any readers who can point us to a corpus of preexisting source code for building, testing, and reasoning about decision-theory algorithms will be much appreciated, as we can then replace this wordy section with a formal evaluation), we can provide informal but still somewhat rigorous explanations of what it should do on several popular problems and why.

First, the simplest case: when a CDT+E agent is placed into Newcomb's Problem, provided that the causal graph expresses the "agenty-ness" of whatever code Omega runs to predict our agent's actions, both versions of the agent (the "simulated" and the "real") will look at the causal graph they are given, detect their entanglement with each-other via pairwise checking and proof-searching (which may take large amounts of computational power), and subsequently restrict their decision-making to choose the best outcome over worlds where they both make the same decision.  This will lead the CDT+E agent to take only the opaque box (one-boxing) and win $1,000,000.  This is the same behavior for the same reasons as is obtained with Timeless Decision Theory, but with less human intervention in the reasoning process.

Provided that the CDT+E agent maintains some model of past events in its causal network, the Parfit’s Hitchhiker Problem trivially falls to the same reasoning as found in the original Newcomb’s Problem.

Furthermore, two CDT+E agents placed into the one-shot Prisoners' Dilemma and given knowledge of each-other's algorithms as embodied logical nodes in the two causal graphs will notice that they are entangled, choose the most preferable action over worlds in which both agents choose identically, and thus choose to cooperate.  Should a CDT+E agent playing the one-shot Prisoner's Dilemma against an arbitrary agent with potentially non-identical code fail to prove entanglement with its opponent (fail to prove that its opponent's decisions mirror its own, up to differences in beliefs), it will refuse to trust its opponent and defect.  A more optimal agent for the Prisoners' Dilemma would in fact demand from itself a proof that either it is or is not entangled with its opponent, and would be able to reason specifically about worlds in which the decisions made by two nodes cannot be the same.  Doing so requires the Principle of the Excluded Middle, an axiom not normally used in the constructive logic of automated theorem-proving systems.

Lastly, different versions of CDT+E yield interestingly different results in the Counterfactual Mugging Problem.  Let us assume that the causal graph given to the agent contains three logical nodes: the actual agent making its choice to pay Omega $100, Omega's prediction of what the agent will do in this case, and Omega's imagination of the agent receiving $1,000 had the coin come up the other way.  The version of the entanglement relation here quantifies over decisions themselves at the first-order level, and thus the two versions of the agent who are dealing with the prospect of giving Omega $100 will become entangled.  Despite being entangled, they will see no situation of any benefit to themselves, and will refuse to pay Omega the money.  However, consider the stricter definition of entanglement given below:

Inductive strongly_entangled {Beliefs Decision Action} (a1 a2: Agent Beliefs Decision Action) :=

  | ent : (forall (b: Beliefs) (d: Decision), a1 b d = a2 b d) -> entangled a1 a2.

This definition says that two agents are strongly entangled when they yield the same decisions for every possible pair of beliefs and decision problem that can be given to them.  This continues to match our original intuition regarding decision entanglement: that we are dealing with the same algorithm (agent), with the same values, being instantiated at multiple locations in time and space.  It is somewhat stronger than the reasoning behind Timeless Decision Theory: it can recognize two instantiations of the same agent that face two different decisions, and enable them to reason that they are entangled with each-other.

Under this stronger version of the entanglement relation (whose proofs for being an equivalence relation are somewhat simpler, by the way), a CDT+E agent given the Counterfactual Mugging will recognize itself as entangled not only with the predicted factual version of itself that might give Omega $100, but also with the predicted counterfactual version of itself that receives $1000 on the alternate coin flip.  Each instance of the agent then independently computes the same appropriate tuple of output actions to maximize profit across the entire equivalence class (namely: predicted-factual gives $100, real-factual gives $100, predicted-counterfactual receives $1000).

Switching entirely to the stronger version of entanglement would cause a CDT+E agent to lose certain games requiring cooperation with other agents that are even trivially different (for instance, if one agent likes chocolate and the other hates it, they are not strongly entangled).  These games remain winnable with the weaker, original form of entanglement.

 

Future research

 

Future research could represent the probabilistic possibility of entanglement within a causal graph by writing down multiple parallel logical/algorithm nodes as children of the same parent, each of which exists and acts with a probability conditional on the outcome of the parent node.  A proof engine extended with probabilities over logical sentences (which, to the authors' knowledge, is not yet accomplished for second-order constructive logics of the kind used here) could also begin to assign probabilities to entanglement between logical/algorithm nodes.  These probabilistic beliefs can then integrate into the action-selection algorithm of Causal Decision Theory just like any other probabilistic beliefs; the case of pure logic and pure proof from axioms merely constitutes assigning a degenerate probability of 1.0 to some belief.

Previous researchers have noted that decision-making over probabilistic acausal entanglement with other agents can be used to represent the notion of "universalizability" from Kantian deontological ethics.  We note that entanglements with decision nodes in the past and future of a single given agent actually lead to behavior not unlike a "virtue ethics" (that is, the agent will start trying to enforce desirable properties up and down its own life history).  When we begin to employ probabilities on entanglement, the Kantian and virtue-ethical strategies will become more or less decision-theoretically dominant based on the confidence with which CDT+E agents believe they are entangled with other agents or with their past and future selves.

Acausal trade/cooperation with agents other than the given CDT+E agent itself can also be considered, at least under the weaker definition of entanglement.  In such cases, seemingly undesirable behaviors such as subjection to acausal versions of Pascal's Mugging could appear.  However, entanglements (whether Boolean, constructive, or probabilistically believed-in) occur between logical/decision nodes in the causal graph, which are linked by edges denoting conditional probabilities.  Each CDT+E agent will thus weight the other in accordance with their beliefs about the probability mass of causal link from one to the other, making acausal Muggings have the same impact on decision-making as normal ones.

The discovery that games can have different outcomes under different versions of entanglement leads us to believe that our current concept of entanglement between agents and decisions is incomplete.  We believe it is possible to build a form of entanglement that will pay Omega in the Counterfactual Mugging without trivially losing at the Prisoners’ Dilemma (as strong entanglement can), but our current attempts to do so sacrifice the transitivity of entanglement.  We do not yet know if there are any game-theoretic losses inherent in that sacrifice.  Still, we hope that further development of the entanglement concept can lead to a decision theory that will more fully reflect the "timeless" decision-making intuition of retrospectively detecting rational precommitments and acting according to them in the present.

CDT+E opens up room for a fully formal and algorithmic treatment of the "timeless" decision-making processes proposed by Yudkowsky, including acausal "communication" (regarding symmetry or nonsymmetry) and acausal trade in general.  However, like the original Timeless Decision Theory, it still does not actually have an algorithmic process for placing the logical/decision nodes into the causal graph -- only for dividing the set of all such nodes into equivalence classes based on decision entanglement.  Were such an algorithmic process to be found, it could be used by an agent to locate itself within its model of the world via the stronger definition of entanglement.  This could potentially reduce the problem of naturalizing induction to the subproblems of building a causal model that contains logical or algorithmic nodes, locating the node in the present model whose decisions are strongly entangled with those of the agent, and then proceeding to engage in "virtue ethical" planning for near-future probabilistically strongly-entangled versions of the agent's logical node up to the agent's planning horizon.

 

Acknowledgements

 

The authors would like to thank Joshua and Benjamin Fox for their enlightening lectures on Updateless Decision Theory, and to additionally thank Benjamin Fox in specific for his abundant knowledge, deep intuition and clear guidance regarding acausal decision-making methods that actually win.  Both Benjamin Fox and David Steinberg have our thanks for initial reviewing and help clarifying the text.

 

People who're engaging in learning partly or wholly for the explicit purpose of human capital need to be strategic about their learning choices. Only some subjects develop human capital useful to the person's goals. Within each subject, only some subtopics develop useful human capital. Even within a particular course, the material covered in some weeks could be highly relevant, and the material covered in other weeks need to be relevant. Therefore, learners need to be discerning in figuring out what material to focus their learning effort on and what material to just skim, or even ignore.

Such discernment is most relevant for self-learners who are unconstrained by formal mastery requirements of courses. Self-learners may of course be motivated by many concerns other than human capital acquisition. In particular, they may be learning for pure consumptive reasons, or to signal their smarts to friends. But at any rate, they have more flexibility than people in courses and therefore they can gain more from better discernment.

Those who're taking courses primarily for signaling purposes need to acquire sufficient mastery to attain their desired grade, but even here, they have considerable flexibility:

• People who're already able to get the top grade without stretching themselves too much have flexibility in how to allocate additional time. Should they try to acquire some more mastery of the entire curriculum, or delve deeper into one topic?
• People who're far from getting a top grade may have the same grade-per-unit-effort payoff from delving deep into one subtopic or acquiring a shallow understanding of many topics. Considerations regarding long-term human capital acquisition can then help them decide what path to pursue among paths that confer roughly similar signaling benefits.

What self-learners and people with some flexibility in a formal learning situation need is what I call utilitarian discernment: the ability to figure out what stuff to concentrate on. Ideally, they should be able to figure this out relatively easily:

1. Sequencing within the course: Important topics are often foundational and therefore done early on.
2. Relative time and emphasis placed on topics should give an indicator of their relative importance.
3. Important topics should be explicitly marked as important by course texts, videos, and syllabi.
4. Important topics should receive more emphasis in end-of-course assessments.
5. Important topics should be more frequently listed as prerequisites in follow-on courses covering the sort of material the learner wants to do next.
6. The learner can consult friends and websites: This includes more advanced students and subject matter experts, as well as online sources such as Quora and Less Wrong.

The above work better than nothing, but I think they still leave a lot to be desired. Some obvious pitfalls:

1. Sequencing within the course: While important topics are often done early on because they are foundational, they are sometimes done later because they rely on a synthesis of other knowledge.
2. Relative time and emphasis: Often, courses place more time and emphasis on more difficult topics than more important ones. There's also the element of time and emphasis being placed on topics that subject matter experts find interesting or relevant, rather than topics that are relevant to somebody who does not intend to pursue a lifetime of research in the subject but is learning it to apply it in other subjects. Note also that the signaling story would suggest that more time and emphasis would be given to topics that do a better job at sorting and ranking students' relevant general abilities than to topics that teach relevant knowledge and skills.
3. Important topics marked as important: This is often the case, but it too fails, because what is important to teachers may differ from what is important to students.
4. Emphasis in end-of-course assessments: The relative weight to topics in end-of-course assessments is often in proportion to the time spent on the topics than their relative importance, bringing us back to (1).
5. Important topics should be more frequently listed as prerequisites: This would work well if somebody actually compiled and combined prerequisites for all follow-on courses, but this is a labor-intensive exercise that few people have engaged in.
6. The learner can consult friends and websites: Friends who lack strong subject matter knowledge may simply be guessing or giving too much weight to their personal beliefs and experiences. Many of them may not even remember the material enough to offer an informed judgment. Those who have subject matter knowledge may be too focused on academic relevance within the subject rather than real-world relevance outside it. People may also be biased (in either direction) about how a particular topic taught them general analytical skills because they fail to consider other counterfactual topics that could have achieved a similar effect.

In light of these pitfalls, I'm interested in developing general guidelines for improving one's utilitarian discernment. For this purpose, I list some example head-to-head contest questions. I'd like it if commenters indicated a clear choice of winner for each head-to-head contest (you don't have to indicate a choice of winner for every one, but I would prefer a clear choice rather than lots of branch cases within each contest), then explained their reasoning and how somebody without an inside view or relevant expertise could have come to the same conclusion. For some of the choices I've listed, I think the winner should be clear, whereas for others, the contest is closer. Note that the numbering in this list is independent of the preceding numbering.

1. Middle school and high school mathematics: Manipulating fractions (basic arithmetic operations on fractions) versus solving quadratic equations (you may assume that the treatment of quadratic equations does not require detailed knowledge of fractions)
2. High school physics: Classical mechanics versus geometrical optics
3. Precalculus/functions: Logarithmic and exponential functions versus trigonometric functions
4. Differential calculus: Conceptual definition of derivative as a limit of a difference quotient versus differentiation of trigonometric functions
5. Integral calculus and applications: Integration of rational functions versus solution strategy for separable differential equations
6. Physical chemistry: Stoichiometry versus chemical kinetics
7. Basic biology: Cell biology versus plant taxonomy
8. Micreconomics: Supply and demand curves versus adverse selection

PS: The examples chosen here are all standard topics in the sciences and social sciences ranging from middle school to early college, but my question is more general. I didn't have enough domain knowledge to come up with quick examples of self-learning head-to-head contests for other domains or for learning at other stages of life, but feel free to discuss these in the comments.

I've recently seen a lot of interest in people who are looking to learn programming.  So I put together a quick guide with lots of help from other people: http://everydayutilitarian.com/essays/learn-code

Let me know (via comments here or email - peter@peterhurford.com) if you try this guide, so I can get feedback on how it goes for you.

Also, feel free to also reach out to me with comments on how to improve the guide – I’m still relatively new to programming myself and have not yet implemented all these steps personally.  I'd cross-post it here, but I want to keep the document up-to-date and it would be much easier to do that in just one place.

Cross-posted on By Way of Contradiction

In my morals, at least up until recently, one of the most obvious universal rights was freedom of thought. Agents should be allowed to think whatever they want, and should not be discouraged for doing so. This feels like a terminal value to me, but it is also instrumentally useful. Freedom of thought encourages agents to be rational and search for the truth. If you are punished for believing something true, you might not want to search for truth. This could slow science and hurt everyone. On the other hand, religions often discourage freedom of thought, and this is a major reason for my moral problems with religions. It is not just that religions are wrong, everyone is wrong about lots of stuff. It is that many religious beliefs restrict freedom of thought by punishing doubters with ostracizing or eternal suffering. I recognize that there are some "religions" which do not exhibit this flaw (as much).

Recently, my tune has changed. There are two things which have caused me to question the universality of the virtue of freedom of thought:

1) Some truths can hurt society

Topics like unfriendly artificial intelligence make me question the assumption that I always want intellectual progress in all areas. If we as modern society were to choose any topic which restricting thought about might be very useful, UFAI seems like a good choice. Maybe the freedom of thought in this issue might be a necessary casualty to avoid a much worse conclusion.

2) Simulations

This is the main point I want to talk about. If we get to the point where minds can simulate other minds, then we run into major issues. Should one mind be allowed to simulate another mind and torture it? It seems like the answer should be no, but this rule seems very hard to enforce without sacrificing not only free thought, but what would seem like the most basic right to privacy. Even today, people can have preferences over the thoughts of other people, but our intuition tells us that the one who is doing the thinking should get the final say. If the mind is simulating another mind, shouldn't the simulated mind also have rights? What makes advanced minds simulating torture so much worse than a human today thinking about torture. (Or even worse, thinking about 3^^^^3 people with dust specks in their eyes. (That was a joke, I know we cant actually think about 3^^^^3 people.))

The first thing seems like a possible practical concern, but it does not bother me nearly as much as the second one. The first seems like it is just and example of the basic right of freedom of thought contradicting another basic right of safety. However the second thing confuses me. It makes me wonder whether or not I should treat freedom of thought as a virtue as much as I currently do. I am also genuinely not sure whether or not I believe that advanced minds should not be free to do whatever they want to simulations in their own minds. I think they should not, but I am not sure about this, and I do not know if this restriction should be extended to humans.

What do you think? What is your view on the morality of drawing the line between the rights of a simulator and the rights of a simulatee? Do simulations within human minds have any rights at all? What conditions (if any) would make you think rights should be given to simulations within human minds?

This is a tiny question that I wouldn't be asking if I had paid more attention in economics class. Anyway, a friend of mine was at the mall with me and he needed to go to the mall parking to retrieve his car. However, if he played at the mall casino, the parking fee would be waived. Without much interest, I heard him calculate his options out loud, until he got to this part:

"The parking fee is $4. I might get that amount waived yet lose more than that at the casino. Or I could play at the casino and win, in which case my expected value is whatever I win plus $4..."

At that moment I felt I had to intervene:

"You don't get to add the parking fee to your expected value if you win at the casino; you merely don't substract it."

But he kept insisting that he could add it. We didn't meet later to check his numbers, but I was left with this question.

Was my objection accurate?