[AI disclosure: I used Opus 5.5 for light edits after my first draft]
Functional decision theory (Yudkowsky and Soares 2017) proposes choosing actions as follows (equation 4 in the paper):
Here is the agent's probability distribution, is a graph saying how changing a logical variable affects other variables, is the observation history, and is utility. The underlines denote dequoting: is a variable in the agent's model representing the output of the FDT algorithm on these inputs.
In this post, by FDT I mean this specific proposal, not TDT, UDT, or logical EDT. FDT, admittedly, relies on a operator applied to a logical statement, which is on shaky philosophical foundations:
Equation (4) is sufficient for present purposes, though it rests on shakier philosophical foundations than (3). Pearl (2000) has given a compelling philosophical account of how to deduce the structure of causation from observation and experiment, but no such formal treatment has yet been given to the problem of deducing the structure of other kinds of subjunctive dependence. Equation (4) works given a graph that accurately describes how changing the value of a logical variable affects other variables, but it is not yet clear how to construct such a thing—nor even whether it can be done in a satisfactory manner within Pearl’s framework. Figuring out how to deduce the structure of subjunctive dependencies from observation and experiment is perhaps the largest open problem in the study of functional decision theory.
The paper itself recognizes the problem of deducing the structure of logical subjunctive dependence (aka logical counterfactuals) as an open and difficult problem, although elsewhere (Section 3) it describes the difficulty as "technical rather than philosophical". This post argues that the problem is both difficult and philosophical. My own opinion goes further: this is not a well-posed problem, "computational logical counterfactuals" (which I will explain below) do not exist, and FDT needs computational logical counterfactuals, so FDT is ill-defined and cannot be adequately formalized. However, before defending my stronger opinion, it helps to argue disjunctively that the problem is difficult, impossible, or ill-posed. Although the FDT paper admits to this disjunction, it would help to make the point sharper.
First, what is a computational logical counterfactual? I assume we are working in a system that can reason about Turing machines, such as PA or ZFC. Call our basic mathematical theory . For now, I assume is first-order. In the language of , we can encode statements about the outputs of Turing machines. I will use the following notation:
states that machine halts on input , where is a natural number or finite string.
states that machine , on input , halts and returns , where is a natural number or finite string.
Statements like these are common in Gödelian analysis of PA-like first-order theories. Natural numbers can encode finite strings and vice versa; I omit the details. I assume is powerful enough to "prove statements about Turing machines" in the following sense:
If is true, then . (Here, and are constant natural numbers.)
If is true, then . (Here, , , and are constant natural numbers.)
is consistent.
.
.
PA satisfies these conditions (at least according to ZFC, which proves PA consistent; conditions 1 and 2 follow from PA's -completeness). I say a computational logical counterfactual is one of the following:
, in some case where in fact .
, in some case where in fact .
, in some case where in fact and .
, where implies one of the above, both provably in and "obviously".
(Grammatically, a "logical counterfactual" here is a probability evaluation if it were the case that some (actually true) mathematical statement were false. "Logical counterfactual" could more generally in philosophy be taken to mean statements like "if PA were inconsistent, then PA could prove its inconsistency", but FDT uses a probabilistic version, so this is the notion of logical counterfactual I will analyze.)
Of course, I haven't formalized the notion of "obviously". I could say that a counter- logical counterfactual is one of the form where . Then computational logical counterfactuals are a subset of counter- logical counterfactuals: those which are on statements that pretty clearly imply (falsely, though not necessarily obviously falsely) that some machine does not halt, or does not return some particular result. (By conditions 1, 2, and 4, each of the first three cases is counter-, and so is anything that -provably implies one of them.) Roughly, an implication is obvious when a human mathematician using GPT-6-Astra-Pro could, without much difficulty, show that .
FDT needs computational logical counterfactuals
First, I will argue that FDT needs computational logical counterfactuals. Recall that FDT takes an expectation under . The intention is that FDT is an algorithm:
Furthermore, FDT agents do not need to evaluate counterpossibilities in full generality; they only need to reason about questions like “How would this predictor’s prediction of my action change if the FDT algorithm had a different output?”
The paper also implies that such algorithms are computable, rather than e.g. using halting oracles:
When a prisoner following FDT imagines the world in which she defects, she imagines a world in which the function she is computing outputs defect; and since her twin is computing the same decision function, she assumes the twin will also defect in that world.
Therefore, if there is a working FDT algorithm at all, it is reasonable to say that can be written as , where is a Turing machine and is a finite string encoding the inputs .[1] I claim this "can be written as" step is obvious as long as the FDT algorithm is specified.
Let's evaluate whether is a computational logical counterfactual. First, suppose . Our rules give no clear way to classify this as a computational logical counterfactual, so say it's not. Next, suppose . Then this isn't even a counterfactual, so it's not a computational logical counterfactual. Finally, suppose for some . Then, by our definition, this is a computational logical counterfactual.
Recall that is a Turing machine for the FDT algorithm. If does not halt on , we have a problem: the "algorithm" outputs no action in this situation. This would be a serious flaw in the FDT algorithm. Otherwise, ; in particular, for some . The FDT algorithm evaluates -interventions on each of its possible actions, iterating over them. In the non-trivial case, that includes both and actions other than . Evaluating involves no counterfactual at all. But evaluating for involves a computational logical counterfactual.
I hope I have shown that FDT really needs computational logical counterfactuals. The alternatives seem to be that (a) the FDT algorithm fails to halt in some situations, or (b) only one action is available, making the problem trivial. To emphasize: any specification of FDT must specify computational logical counterfactuals. If computational logical counterfactuals are unspecifiable, then so is FDT.
A cat-belling problem
I propose that specifying computational logical counterfactuals is a cat-belling problem for specifying FDT: a step we have some reason to believe is both (a) required for specifying FDT and (b) difficult, if not impossible.
An outside-view argument for difficulty: the FDT paper dates to 2017, and the Timeless Decision Theory paper (Yudkowsky 2010) discusses "counterfactual worlds where a deterministic computation returns an output it does not actually return" (a computational logical counterfactual in my sense) as an important input to decision theory, claiming that "Any agent’s ability to make a decision, and the specific decision made, is determined by the agent’s ability to visualize logically impossible counterfactuals". So it's been at least 16 years, and no one has written down a definition of computational logical counterfactuals, despite the Yudkowskian decision theory tradition claiming that they are a critical ingredient for decision theory. On the outside view, that sure indicates difficulty.
If, as I believe, computational logical counterfactuals are ill-posed, that would explain why efforts to specify them have failed. But I don't want to be overconfident; perhaps it is possible to specify them, and it has just been hard so far. I'll now give inside-view arguments for difficulty.
Inside view: which axiom breaks?
First, note that a computational logical counterfactual might, in principle, be probabilistic; the FDT algorithm takes expectations under -interventions. This comes up in Pearlian Bayes nets, where an intervention on a node need not lead to a single joint assignment of values to variables, but may instead produce a distribution. Still, Pearlian Bayes nets allow "sampling" a joint assignment from that distribution, using some external source of randomness. I assume computational logical counterfactuals are similar: it's possible to sample a joint assignment of truth values to logical sentences in a counterfactual.
So let's say we sample one or more "logical counterfactual worlds". This sampling need not happen all at once; it could involve lazy evaluation. For example, if we ask whether the trillionth digit of is 5 in this world, we may need to wait a long time (and keep sampling random bits) for the algorithm to tell us. What we ask of this sample is the following: are all of 's axioms true in the sample?
Of course, it could be that 's axioms all hold with some non-negligible probability and fail with some non-negligible probability. The problem with all of 's axioms being true is the following. If in fact , then (condition 2). By the soundness theorem, holds in every model of . (This goes for first-order models of , but more importantly for now, it also goes for complete consistent extensions of , which contain every theorem of .) So if the sample satisfies all of 's axioms and also respects the rules of classical first-order logic, it satisfies , and therefore (by condition 4) does not satisfy for . But satisfying was the point of the computational logical counterfactual.
So, for computational logical counterfactuals to be sensible, the samples each need to violate some axiom(s) of . Which axiom is violated can differ between samples, but each sample needs to violate an axiom of .
Say is PA. Now, which axioms of PA would you be happy to see violated in logical counterfactuals? They all look pretty common-sensical. The induction schema is the most suspicious one. Robinson arithmetic (denoted ) is essentially PA without induction.[2] Is this promising for computational logical counterfactuals?
The problem is that when , we have . The basic reason is that showing a Turing machine returns a specific value, given that it does, only requires checking a finite computation trace, which comes down to a finite sequence of additions and multiplications. There is no need to invoke induction. (Technically: is -complete; see Smith, An Introduction to Gödel's Theorems, "Q is -complete".) Relatedly, is strong enough for Gödel's first incompleteness theorem (Raatikainen, SEP §1.2.1; Tarski, Mostowski, and Robinson 1953). So even is too strong for computational logical counterfactuals.[3]
But none of 's axioms look like good candidates to mess with; they do things like define addition and multiplication. Neither do the axioms of first-order classical logic, which specify the meanings of logical connectives and quantifiers. So it looks like samples of computational logical counterfactuals need to almost always violate some axiom of (or of first-order classical logic), but there is no good place to posit the violation.
This is the basic inside-view argument for difficulty. If computational logical counterfactuals can be sampled from, their samples must (almost always) violate or first-order classical logic. There is no obvious way either to reject the possibility of sampling, or to decide which axioms of (or of first-order classical logic) are worth violating. Any violation could lead to all sorts of crazy consequences, perhaps with corresponding Troll Bridge problems; say, the troll blows you up if any axiom of or of first-order logic is violated.
Proposals that don't bell the cat
Now I will address various ways people might claim that logical counterfactuals exist and are feasible to specify.
Logical conditionals. People may mentally substitute "logical conditional" (i.e. subjective probability conditional on some logical statement, under some mechanism for logical uncertainty) for "logical counterfactual", and then point to something like Logical Induction (Garrabrant et al. 2016) for logical conditionals. Logical Induction does show it is possible to reason theoretically about what computationally bounded agents might believe about logical statements. Yet these are conditionals, not counterfactuals. They may work for specifying the evidential variant of FDT sketched in footnote 9 of the FDT paper, but not FDT itself.
Model-theoretic counterfactuals. People may argue that (non-computational) counterfactuals on mathematical facts are well-defined. This might be true: PA is very likely consistent, and yet (if so) PA + is also consistent, by Gödel's second incompleteness theorem, making it possible to reason from a false mathematical premise. Still, since , this is not a counter-PA counterfactual, let alone a computational logical counterfactual. (Note that is equivalent to , for a machine that searches for a PA-proof of . It makes a non-halting machine halt, which my definition does not count.) A secondary, but probably less important, point is that this might give conditionals rather than counterfactuals: perhaps we can model the beliefs of an agent who is uncertain about and who reasons conditional on .
Physical counterfactuals. People may imagine a physical computer running. Partway through, a cosmic ray hits the computer and its state changes, so a different computer state results. It is as if the same original program returned a different value. But this is not a computational logical counterfactual. It is a physical counterfactual on a physical state change to the computer, and the change means the physical computer no longer properly implements the original program. If used for decision theory, this might yield something CDT-like.
I hope it is clear from the above why I think these proposals fail to bell the cat of computational logical counterfactuals. If in fact , these proposals say nothing useful about for .
Conclusion
For FDT to be well-specified, computational logical counterfactuals need to be well-specified, at least for the case of counterfactuals on the FDT algorithm returning different values. Furthermore, specification is essential to FDT, since FDT needs to counterfact on some algorithm returning a different value, and every particular algorithm is specific. Computational logical counterfactuals are difficult-to-impossible to specify, based both on outside-view considerations, and inside-view considerations. I propose that computational logical counterfactuals are a cat-belling problem for FDT, since FDT needs them, while they are difficult; and it's common for attempts at defining "logical counterfactuals" or FDT to glance off this critical problem.
References
Garrabrant, S., Benson-Tilsen, T., Critch, A., Soares, N., and Taylor, J. (2016). Logical Induction. arXiv:1609.03543.
This assumes and have finite representations, as they must if they are inputs to an algorithm. ↩︎
More precisely, drops the induction schema and adds the single axiom that every nonzero number is a successor, which PA proves by induction. ↩︎
Whether also proves uniqueness of outputs (condition 4) for specific and depends on details of the arithmetization. But this doesn't help: any sample satisfying still satisfies , along with every other true halting fact, such as the output of a predictor's simulation of . Such a sample is not a world in which returns instead of, and nothing downstream of 's output changes. ↩︎
[AI disclosure: I used Opus 5.5 for light edits after my first draft]
Functional decision theory (Yudkowsky and Soares 2017) proposes choosing actions as follows (equation 4 in the paper):
Here
In this post, by FDT I mean this specific proposal, not TDT, UDT, or logical EDT. FDT, admittedly, relies on a
The paper itself recognizes the problem of deducing the structure of logical subjunctive dependence (aka logical counterfactuals) as an open and difficult problem, although elsewhere (Section 3) it describes the difficulty as "technical rather than philosophical". This post argues that the problem is both difficult and philosophical. My own opinion goes further: this is not a well-posed problem, "computational logical counterfactuals" (which I will explain below) do not exist, and FDT needs computational logical counterfactuals, so FDT is ill-defined and cannot be adequately formalized. However, before defending my stronger opinion, it helps to argue disjunctively that the problem is difficult, impossible, or ill-posed. Although the FDT paper admits to this disjunction, it would help to make the point sharper.
(I previously critiqued logical counterfactuals in 2020; Wei Dai briefly criticized logical counterfactuals in 2019).
Computational logical counterfactuals
First, what is a computational logical counterfactual? I assume we are working in a system that can reason about Turing machines, such as PA or ZFC. Call our basic mathematical theory
Statements like these are common in Gödelian analysis of PA-like first-order theories. Natural numbers can encode finite strings and vice versa; I omit the details. I assume
PA satisfies these conditions (at least according to ZFC, which proves PA consistent; conditions 1 and 2 follow from PA's
(Grammatically, a "logical counterfactual" here is a probability evaluation if it were the case that some (actually true) mathematical statement were false. "Logical counterfactual" could more generally in philosophy be taken to mean statements like "if PA were inconsistent, then PA could prove its inconsistency", but FDT uses a probabilistic version, so this is the notion of logical counterfactual I will analyze.)
Of course, I haven't formalized the notion of "obviously". I could say that a counter-
FDT needs computational logical counterfactuals
First, I will argue that FDT needs computational logical counterfactuals. Recall that FDT takes an expectation under
The paper also implies that such algorithms are computable, rather than e.g. using halting oracles:
Therefore, if there is a working FDT algorithm at all, it is reasonable to say that
Let's evaluate whether
Recall that
I hope I have shown that FDT really needs computational logical counterfactuals. The alternatives seem to be that (a) the FDT algorithm fails to halt in some situations, or (b) only one action is available, making the problem trivial. To emphasize: any specification of FDT must specify computational logical counterfactuals. If computational logical counterfactuals are unspecifiable, then so is FDT.
A cat-belling problem
I propose that specifying computational logical counterfactuals is a cat-belling problem for specifying FDT: a step we have some reason to believe is both (a) required for specifying FDT and (b) difficult, if not impossible.
An outside-view argument for difficulty: the FDT paper dates to 2017, and the Timeless Decision Theory paper (Yudkowsky 2010) discusses "counterfactual worlds where a deterministic computation returns an output it does not actually return" (a computational logical counterfactual in my sense) as an important input to decision theory, claiming that "Any agent’s ability to make a decision, and the specific decision made, is determined by the agent’s ability to visualize logically impossible counterfactuals". So it's been at least 16 years, and no one has written down a definition of computational logical counterfactuals, despite the Yudkowskian decision theory tradition claiming that they are a critical ingredient for decision theory. On the outside view, that sure indicates difficulty.
If, as I believe, computational logical counterfactuals are ill-posed, that would explain why efforts to specify them have failed. But I don't want to be overconfident; perhaps it is possible to specify them, and it has just been hard so far. I'll now give inside-view arguments for difficulty.
Inside view: which axiom breaks?
First, note that a computational logical counterfactual might, in principle, be probabilistic; the FDT algorithm takes expectations under
So let's say we sample one or more "logical counterfactual worlds". This sampling need not happen all at once; it could involve lazy evaluation. For example, if we ask whether the trillionth digit of
Of course, it could be that
So, for computational logical counterfactuals to be sensible, the samples each need to violate some axiom(s) of
Say
The problem is that when
But none of
This is the basic inside-view argument for difficulty. If computational logical counterfactuals can be sampled from, their samples must (almost always) violate
Proposals that don't bell the cat
Now I will address various ways people might claim that logical counterfactuals exist and are feasible to specify.
Logical conditionals. People may mentally substitute "logical conditional" (i.e. subjective probability conditional on some logical statement, under some mechanism for logical uncertainty) for "logical counterfactual", and then point to something like Logical Induction (Garrabrant et al. 2016) for logical conditionals. Logical Induction does show it is possible to reason theoretically about what computationally bounded agents might believe about logical statements. Yet these are conditionals, not counterfactuals. They may work for specifying the evidential variant of FDT sketched in footnote 9 of the FDT paper, but not FDT itself.
Model-theoretic counterfactuals. People may argue that (non-computational) counterfactuals on mathematical facts are well-defined. This might be true: PA is very likely consistent, and yet (if so) PA +
Physical counterfactuals. People may imagine a physical computer running. Partway through, a cosmic ray hits the computer and its state changes, so a different computer state results. It is as if the same original program returned a different value. But this is not a computational logical counterfactual. It is a physical counterfactual on a physical state change to the computer, and the change means the physical computer no longer properly implements the original program. If used for decision theory, this might yield something CDT-like.
I hope it is clear from the above why I think these proposals fail to bell the cat of computational logical counterfactuals. If in fact
Conclusion
For FDT to be well-specified, computational logical counterfactuals need to be well-specified, at least for the case of counterfactuals on the FDT algorithm returning different values. Furthermore, specification is essential to FDT, since FDT needs to counterfact on some algorithm returning a different value, and every particular algorithm is specific. Computational logical counterfactuals are difficult-to-impossible to specify, based both on outside-view considerations, and inside-view considerations. I propose that computational logical counterfactuals are a cat-belling problem for FDT, since FDT needs them, while they are difficult; and it's common for attempts at defining "logical counterfactuals" or FDT to glance off this critical problem.
References
This assumes
More precisely,
Whether