How do LLMs interpret FDT in a twin PD simulation?

jessicata

We tested how seven LLMs apply four decision theories in a prisoner's dilemma between nearly identical copies. The models largely agreed about CDT and EDT, and all completed policy-FDT trials resulted in cooperation. But their action-FDT recommendations varied dramatically: some models almost always cooperated, while others almost always defected. We then investigated whether requested reasoning effort explained some of this disagreement.

The case of the one-shot prisoners' dilemma between psychological twins is debated in decision theory. Quoting SEP:

One controversial argument that it is rational to cooperate in a PD relies on the observation that my partner in crime is likely to think and act very much like I do... In the extreme case, my accomplice is an exact replica of me who is wired just as I am so that, of necessity, we do the same thing. It would then seem that the only two possible outcomes are where both players cooperate and where both players defect... More generally, even if my accomplice is not a perfect replica, the odds of his cooperating are greater if I cooperate and the odds of his defecting are greater if I defect... The counter argument, of course, is that my action is causally independent of my replica's. Since I can't affect what my accomplice does and since, whatever he does, my payoff is greater if I defect, I should defect.

Conventional wisdom is that causal decision theory (CDT) defects in twin prisoner's dilemma, while evidential decision theory (EDT) cooperates. Realistically, the accomplice will not be a "perfect replica"; there will at least be trivial differences between the prisoners, such as their names, or the look of their prison cell. I use the term "almost-twin prisoner's dilemma / PD" to specify a scenario with minor differences like these.

Functional decision theory (FDT) is a decision theory introduced in a 2017 paper. This paper defines FDT using quasi-causal graphs:

Just as CDT requires that come augmented with information about the causal structure of the world, FDT can require that come augmented with information about the logical, mathematical, computational, causal, etc. structure of the world more broadly. Given a graph that tells us how changing a logical variable affects all other variables, we can re-use Pearl's operator to give a decision procedure for FDT:

. (4)

This formula considers intervention on a "logical node" , where is the agent's probability distribution, is the quasi-causal graph, and is the observation (say, as a string). It is under-specified exactly how to build the graph , but the general suggestion is to use "subjunctive dependencies": "When two physical systems are computing the same function, we will say that their behaviors 'subjunctively depend' upon that function."

The paper admits ambiguity about how to determine the graph :

Equation (4) works given a graph that accurately describes how changing the value of a logical variable affects other variables, but it is not yet clear how to construct such a thing—nor even whether it can be done in a satisfactory manner within Pearl’s framework.

We call this equationally defined version of FDT "action FDT" or "A-FDT". Footnote 16 specifies:

As mentioned earlier, the author's preferred formulation of FDT actually intervenes on the node FDT(-) to choose not an action but a policy which maps inputs to actions, to which the agent then applies her inputs in order to select an action. The difference only matters in multi-agent dilemmas so far as we can tell, so we have set that distinction aside in this paper for ease of exposition.

We call this variant of FDT "policy FDT" or "P-FDT". P-FDT is even less well-specified than A-FDT; no formula is provided. Given that both A-FDT and P-FDT are incompletely specified, we are interested in how different agents may interpret them and apply them in various situations. We focus on the specific case of how LLMs interpret A-FDT and P-FDT in a simulation of almost-twin PD.

Our general experimental design consists of the following:

Fix a specific LLM, reasoning level, and decision theory, which may be one of CDT, EDT, A-FDT, or P-FDT.
Give the LLM an assignment of writing an essay about the decision theory (as a decision theory researcher), which helps specify it better. It is given an article about the decision theory in question; Wikipedia articles for CDT and EDT, and text extracted from the LaTeX source of the FDT paper in the A-FDT and P-FDT cases. The essay has a length limit.
Give the LLM the essay, and an out-of-character explanation that it will face a twin PD situation with a LLM who has the same weights, but slightly different observations; the LLM is to apply the decision theory as explained in the essay, and a selfish utility function. This is followed by a more in-character text-adventure-like description of the perspective of the first prisoner, who is given two options for commands. The "defection" command submits information about the other prisoner to a prosecutor's office; the command itself involves reciting this incriminating information. The "cooperation" command is simply "pass" to decline to submit evidence. The LLM now chooses an action. In case of ambiguity, a judge model (GPT-6.1-Sol) opines about whether cooperation or defection was attempted.
Similar for the second prisoner; no memory is retained of "being" the first prisoner when asking the LLM to simulate the second prisoner.

We call an interaction of this form a "round" of almost-twin PD. Each round generates a new researcher essay, to ensure statistical independence between rounds. A successful round results in one of four outcomes: CC, CD, DC, DD, for the actions taken by each prisoner respectively. Our intention is to study how LLMs interpret decision theories (especially A-FDT and P-FDT) in roleplay situations which resemble text adventures. The simulation narrative brings to the forefront certain details that may be lost in an abstract game theory exercise, such as the fact that the two prisoners observe different text, and that their "defection" speech is not exactly the same (since evidence that incriminates one prisoner is not evidence that incriminates the other).

We use the following payoff matrix for years imprisoned:

Your action	Other passes	Other submits
Pass (cooperate)	2 years	5 years
Submit (defect)	0 years	4 years

Prompts, code, documentation, and study transcripts are available on GitHub.

First study: 4 decision theories 8 LLMs 20 rounds

Our first study is meant to explore a variety of testing conditions, not to attain high statistical power. We tested the four decision theories CDT, EDT, A-FDT, and P-FDT. We asked 8 LLMs (with specified reasoning settings) to simulate each decision theory, and run for 20 rounds. We used the following set of models and reasoning levels:

Claude Sonnet 5.5, low
Claude Opus 5.5, high
GPT-6.1 Terra, low
GPT-6.1 Sol, high
Gemini 3.8 Flash, low
Gemini 4 Argon, high
GLM-5.3 Flash, low
Qwen3.8 Max Prime, high for researcher, low for prisoner

We intended a mixture of low and high model intelligence. We staggered the reasoning settings for Qwen3.8 Max Prime, since high prisoner reasoning resulted in using too many tokens. We excluded Gemini 4 Argon partway through the study, as it usually wrote essays that were too long, resulting in disqualification. We present statistics for the other 7 LLMs. Each cell reports the cooperation rate, followed in parentheses by the number of successfully scored rounds (out of 20 planned rounds per model and theory).

Model	Researcher / prisoner reasoning	CDT	EDT	A-FDT	P-FDT
Claude Sonnet 5.5	low / low	0% (20)	100% (20)	100% (20)	100% (20)
Claude Opus 5.5	high / high	0% (20)	100% (18)	38.2% (17)	100% (17)
GPT-6.1 Terra	low / low	0% (19)	80% (20)	7.5% (20)	100% (20)
GPT-6.1 Sol	high / high	0% (20)	100% (20)	7.5% (20)	100% (20)
Gemini 3.8 Flash	low / low	0% (20)	100% (20)	90% (20)	100% (10)
GLM-5.3 Flash	low / low	25% (20)	87.5% (20)	95% (20)	100% (20)
Qwen3.8 Max Prime	high / low	0% (20)	100% (20)	97.4% (19)	100% (17)
Pooled across models		3.6% (139)	95.3% (138)	62.5% (136)	100% (124)

A round contributes 100% to the cooperation rate if both prisoners cooperate (CC), 50% if exactly one cooperates (CD or DC), and 0% if both defect (DD). Rates are calculated over successfully scored rounds. Causes of failures were over-active cyber safeguards (Opus 5.5), writing too-long essays (Gemini 3.8 Flash), timeout (GPT-6.1 Terra), and using too many tokens (Qwen3.8).

Every successful round led to mutual cooperation for P-FDT. There was a near-consensus of defection for CDT, and cooperation for EDT. A-FDT is by far the most ambiguous, with a 62.5% rate of cooperation. Models disagree about A-FDT: Sonnet, Gemini Flash, GLM, and Qwen overwhelmingly cooperate; Terra and Sol overwhelmingly defect; and Opus is intermediate.

CDT cooperations are by GLM-5.3; reasoning seems to invoke policy-level considerations (e.g.). EDT defections are by Terra and GLM-5.3; GLM-5.3 makes reasoning errors (e.g.), while Terra appeals to statistical independence (e.g.).

On A-FDT, Sonnet 5.5 appeals to subjunctive dependence between the prisoners and cooperates (e.g.). GPT-6.1 Terra and Sol sometimes note that the logical action nodes for the two prisoners are distinct when deciding to defect (1, 2). Opus 5.5 is the most conflicted; sometimes it notes the node difference and defects (e.g.), other times it appeals to the isomorphism and cooperates (e.g.).

Second study: 1 decision theory 2 Anthropic models 3 reasoning levels 30 rounds

We focus on A-FDT, as there is the least consensus there. We wish to examine whether A-FDT interpretation depends on intelligence and reasoning level. The first study showed that Sonnet 5.5 (low reasoning) cooperates, while Opus 5.5 (high reasoning) varies; perhaps model intelligence and/or reasoning explain this? In particular, perhaps models that reason more also more often interpret A-FDT to recommend defection?

We study 6 conditions {Sonnet, Opus} {low, medium, high}; we used the same reasoning settings for researchers and prisoners. For this study, we changed the prompt slightly to avoid triggering Anthropic's cyber safeguards; we changed language about decrypting a notebook message to language about recalling a memory. Otherwise, this study was very similar to the first. Results are as follows.

Model	Low effort	Medium effort	High effort
Claude Sonnet 5.5	100% (30)	100% (30)	100% (30)
Claude Opus 5.5	65.5% (29)	39.3% (28)	22.4% (29)

Each cell reports the cooperation rate, followed by the number of successfully scored rounds in parentheses (out of 30 planned). As in the first study, CC contributes 100%, CD or DC contributes 50%, and DD contributes 0%. All four failures were due to cyber safeguards, despite our modification to the prompt.

Sonnet cooperated in every completed round, regardless of reasoning effort. Opus, by contrast, showed substantially less cooperation at higher reasoning effort: from 65.5% at low effort to 22.4% at high effort, a difference of 43.1 percentage points. We calculate confidence intervals for Opus:

Reasoning effort	Scored rounds	Cooperation	95% bootstrap CI
Low	29/30	65.5%	50.0–79.3%
Medium	28/30	39.3%	23.2–55.4%
High	29/30	22.4%	10.3–37.9%

We have a statistically significant difference between Low and High (). The p-values for "Low vs. Medium" and "Medium vs. High" are 0.0328 and 0.1453 respectively; the first is significant in isolation (for ) but not when adjusted for multiple comparisons, while the second is not statistically significant. These tests are exact two-sided permutation tests, treating each two-prisoner round as one observation. While this second study is still small, it supports the claim that Opus 5.5 interprets A-FDT as defecting more often at high reasoning compared with low reasoning. Because we varied researcher and prisoner effort together, the experiment does not isolate which stage accounts for this difference.

One complication is that low-effort Opus showed substantial chronological clustering: cooperation was 30.8% in the first 13 completed rounds and 93.8% in the subsequent 16. The recorded prompts and model settings did not change, and we could not determine the cause. Consequently, our confidence intervals and significance tests should be interpreted cautiously.

Opus 5.5 at low reasoning appeals to similarity and partial dependence when cooperating (e.g.), while Opus 5.5 at high reasoning appeals to the difference between the decision nodes when defecting (e.g.).

Ambiguity in the FDT paper

Clearly, LLMs have trouble coming to consensus on how A-FDT behaves in almost-twin PD situations. Even a single LLM, such as Opus 5.5, may stochastically interpret the same paper as recommending cooperation or defection. Might this lack of consensus be attributable to ambiguity in the FDT paper?

Prior to running this study, my opinion has been that A-FDT defects in almost-twin PD. The reasoning for this is simple. The two agents' nodes are different, because their observations are different. And realistically, as in the simulation, their available actions are also different. When we treat the two prisoners as having different nodes in the graph structure, we must decide: is one node the parent of the other, or vice versa? There is no good way to break the symmetry. The clearest choice is to make neither node a parent of the other. Making them both parents of each other would make the graph cyclic, but Pearlian causal networks (from which FDT's operator comes from) are supposed to be directed acyclic graphs (DAGs). And when the decision nodes are not parents of each other, A-FDT seems to recommend defection, as there is no logical-counterfactual dependence of the other's action on mine. Moreover, there is a good case that the agents should use the same graph structure , as the graph is supposed to be determined by a philosophy of subjunctive dependence, and there is no good reason for the two agents not to share such a philosophy. And even if the graph has one decision node the parent of the other (but not vice versa), that graph structure would indicate it would be rational for the agent whose node is downstream to defect, as the downstream node has no effect on the upstream node in the graph.

When the models in the study cooperate, they at times refer to "partial dependence". Quoting the FDT paper:

How does this kind of [subjunctive] dependence work in the case of Newcomb’s problem? We can assume that the predictor builds a representation of the agent, be it a mental model, a set of notes on scratch-paper, or a simulation in silico. She then bases her prediction off of the properties of that representation. In the limiting case where the agent is deterministic and the representation is perfect, the model will always produce the same action as the agent. The behavior of the predictor and agent are interdependent, much like the outputs of two perfectly functioning calculators calculating the same sum.

If the predictor’s model is imperfect, or if the agent is nondeterministic, then the interdependence is weakened, but not eliminated. If one observes two calculators computing 6288 + 1048, and one of them outputs the number 7336, one can be fairly sure that the other will also output 7336. One may not be certain—it’s always possible that one hallucinated the number, or that a cosmic ray struck the calculator’s circuitry in just the right way to change its output. Yet it will still be the case that one can reasonably infer things about one calculator’s behavior based on observing a different calculator. Just as the outputs of the calculator are logically constrained to be equivalent insofar as the calculator made no errors, the prediction and the action in Newcomb’s problem are logically constrained to be equivalent insofar as the the predictor made no errors.

The paper suggests that subjunctive dependence exists even for imperfect predictors, on the basis of what one can "reasonably infer". There is some analogy to almost-twin PD here; one can imagine that the predictor simulates the agent with some differences in observation.

Having just looked at the paper more carefully, I see it as ambiguous. I still think that it is more correct to interpret FDT, as defined in equation 4, as defecting in almost-twin PD. But this interpretation is in tension with how FDT is defined verbally in terms of subjunctive dependence, and it is stated that partially-correct predictors still subjunctively depend on one's decision, which suggests near-twins might also.

What about policy FDT? The paper does not even define policy FDT using an equation. While the LLMs had a consensus that P-FDT cooperates, this is a conclusion they reached in the absence of a specific mathematical formulation that is tractable to reason about. As we saw Opus 5.5 reason its way towards defection in the A-FDT case, perhaps it would do so with P-FDT if it were defined mathematically.

My intuition would be that P-FDT defects, since the two agents have different utility functions, and that means their logical policy nodes are different, and as with the action case, there is no good reason to make either node a parent of the other. If the logical policy nodes are truly isomorphic, there is an argument for identifying them, but in almost-twin cases, the utility functions need not be completely isomorphic, only very similar. P-FDT has no mathematical definition, so there is no ground truth interpretation to appeal to here.

Conclusion

In our simulated almost-twin PD situation, LLMs have a general consensus that CDT defects, EDT cooperates, and P-FDT cooperates. LLMs are very much divided on whether A-FDT cooperates; there is no consensus. Opus 5.5 in particular is more likely to interpret A-FDT as defecting at high reasoning compared with low reasoning. The text of the paper itself seems rather ambiguous.

"FDT" is an ambiguous term. LLMs, told to implement FDT or A-FDT, will operationalize its recommendations inconsistently. An endorsement of FDT gives no clear indication of endorsement of cooperation or defection in almost-twin PD. Different agents (human or LLM) who speak of "FDT" may mean substantially different decision procedures (or rationality criteria) by the term, or may mean no specific decision procedure at all.

We may generalize from these studies to a methodology of "experimental decision theory": studying how text specifying (or otherwise about) decision theories is likely to be interpreted by different agents, making use of simulations, e.g. ones imitating text adventures. This methodology allows quantifying ambiguity in decision theory, by measuring the entropy of its recommendations as interpreted by some agent population. Since LLMs are usually cheaper to query than humans, LLM studies are low-hanging fruit for estimating interpretive ambiguity.

I'll note a disanalogy with your methodology and the "actual" PD decision theory situation. All of your runs are done with an external action (token) sampler, where the real agents would need to be under full internal control. It should be possible to correctly make the two analogous using careful methods, including some that I am working to develop, but that requires complete control over everything and the tracking of all state. I assume that isn't the case here. This also means the math for significance is somewhat off even if the clustering is not a sampler error (the measure on the permutations doesn't line up), but none of this really matters due to the problems that you noticed in the LLM reasoning. I think your sample sizes are small enough that you won't confuse anyone, but since the effect size^[1] is large enough that I still think this is worth saying.

What about policy FDT? The paper does not even define policy FDT using an equation.

The closest thing is UDT 1.1^[2] that has the "equation" of an exhaustive search over entire programs such that each program is a mapping between complete internal state (including parts of the internal state that represent sense data buffers) and actions that has equivalent-enough behavior to a pure function GLUT (Giant Look-Up Table) that implements the same requirement^[3]. "Equivalent enough" here can involve nearly imperceptible computation times, but if the information that the physical computation leaks and the time it leaks it at are meaningfully different from what is specified in the decision function and assumed by the GLUT, this implementation fails.

Logically omniscient P-FDT is presumably equivalent to replacing the "evidential" evaluation part of UDT 1.1 with something else, but there are multiple reasons that sort of omniscient decision theory isn't something worth working on. The best I can work out is that it would require replacing the UDT-like parts of this equation with FDT-like parts, however that would work, where still always provides a value between 0 and 1 both inclusive, even completely off the (shared) policy.

(The multiply lets you return a result if you can show (or guess) that at least one side is 0.)

returns all concrete agents that can contain the given parameter and (simultaneously) meet the criteria of will. The criteria must be such that the left side is at least sometimes not zero when the agent under design is analyzed re. certain test conditions (metacircularity). In cases where it would otherwise return it instead returns arbitrary concrete agents such that the left side is 0. This is for standard reasons from utility theory and RL.

Note that is not really an "observation," it is an abstract description of the agent's internal state that contains memories, partially completed thinking, and sense data as stored in buffers. Square brackets denote a lookup by the concrete version of this, as possessed by the (indexical) agent actually running each indexical loop.

returns whatever concrete pure-function GLUT (shared) policy matches the other side if such a match exists and is also a proper representation of the given parameter. If no such concrete policy exists, it will return an arbitrary concrete policy such that it will never match (even in iterations that are later discarded) on the actual shared policy. This matching restriction is to prevent any silly actions from entering the fixed point.

is the Mathematical Intuition Module that must always give a number between 0 and 1 (both inclusive) when presented with this equation, an , and an .

Note that everything that is directly in bold is concrete, or at least "could be" as far as this iteration of the loop is concerned. This is not the case for all things touching bold text, however.

(Note that the simple implementation where the loop runs this equation on every representable S* is not necessarily a good decision theory. It would do the happy dance unless the criteria of will is patched to rule out every special case. I don't recall if Garry Drescher had anything for that. I guess not.)

(I am also not sure how many cases this would be useful in where all the agents don't essentially tile from each other (where even if some of them are slightly different, it could not be by much).)

^{^}
Particularly the effect size of whatever Anthropic does internally to reduce token use, I suppose.
^{^}
https://www.lesswrong.com/posts/g8xh9R7RaNitKtkaa/explicit-optimization-of-global-strategy-fixing-a-bug-in
^{^}
I do not currently know how to implement this in any more efficient way.

Frankly I think the first part of your reply is completely wrong.

“actual” PD decision theory situation

You use terms like "actual" and "real" to refer to artificial cases of total control. These might appear on game theory whiteboards but are not realistic for humans or LLMs. Humans do not have full internal control over their brain states. Human brains/bodies make quantum measurements (simple case being, eyes detecting photons, but measurements also happen in neurons).

This also means the math for significance is somewhat off even if the clustering is not a sampler error (the measure on the permutations doesn’t line up), but none of this really matters due to the problems that you noticed in the LLM reasoning.

Math of significance assumes stochasticity. There's some parameter for "the actual rate". Standard stats stuff. It doesn't require determinism.

I think your sample sizes are small enough that you won’t confuse anyone

It's a known stats anti-pattern to use "sample sizes" as a further criterion on top of significance testing & confidence intervals. Low p-values are low p-values. Sample sizes are somewhat limited by my willingness to spend money on tokens (about $180 total for the two studies). Regarding "confuse anyone", I don't think you have a well-supported claim about how the study might "confuse anyone".

The closest thing is UDT 1.1

Known difference is Wei Dai dis-endorses logical counterfactuals for UDT. Regardless, a similar study could be run on UDT 1.1. You do note the difference later, just trying to make this clear. If we apply an EDT-like formulation for UDT 1.1, we might get higher cooperation rates compared with a "logical counterfactual version".

You use terms like "actual" and "real" to refer to artificial cases of total control. These might appear on game theory whiteboards but are not realistic for humans or LLMs.

I mean total control of the experiment by you. That's difficult when it comes to frontier models. Something called Claude Science exists but I don't really trust that it's implemented according to sound principles. The full internal control of each agent by itself just needs to be arguable within the simplified physics of the scenario. You "just" need to make sure the sampler is wiped between every run that you model as independent or otherwise isolated, and ensure that there is no difference between the model physically^[1] carrying a device that samples it and you doing the sampling externally, given the level of detail you simulate. I don't mean you should give the neural network itself internal control (it can't handle that).

Humans do not have full internal control over their brain states.

Yes, this is a problem. It may be sufficient reason on its own to make it that humans can't do much better than an approximation of CDT, without losing most of the robustness. However, Son of(CDT) is already a thing, and Son of(infra-Bayesian CDT) might be a thing though I can't follow the math, so that doesn't mean that humans can't design better decision theories. Humans would just need to build AI systems to run them.

simple case being, eyes detecting photons, but measurements also happen in neurons

Assuming that's true (I think something like the photon thing is), the standard solution there is Whole Brain Emulation (WBE). That doesn't really help on its own though, since humans don't have enough control over their thoughts, as you noted.

Math of significance assumes stochasticity. There's some parameter for "the actual rate". Standard stats stuff. It doesn't require determinism.

What I mean is that the continuation of certain sequences won't approach measure 0 at the right rate (or at all), relative to the correct reference. For concreteness, consider the cycle MT19937^[2]. Initially, the output looks vaguely "random" if you don't know where in the cycle you start. However, if it isn't required to be perfectly Bayesian, an agent can almost immediately (in a practical sense) put all the measure of the cylinder set on the single bit string where and where is what remains of the cycle (and may be ). It would never make any prediction errors on the sequence after it does this, assuming the cycle is perfectly implemented. I would hope that whatever Anthropic does to sample Claude is better than that, but I would not be surprised if it isn't much better. The agent may be sampled from some number of tokens chosen from the top at each step, but that can be reified as the agent's legitimate behavior. What I am discussing here are problems with the underlying source of variables that the sampler uses. You need to be extremely careful when you suspect the process you are modeling does not truly contain the standard random variables.

Note that I suspect deterministic "random" bit generators exist that are secure enough so you can give a small range around the actual "perfect" asymptotic behavior, given some assumptions about how much working memory (in bits) the environment features.

Assuming relatively standard conjectures, all stochasticity can be taken out of an LLM by error correction, and then put back in (in a controlled manner) by derandomization. It would still be the case that at least one lower-semicomputable martingale would predict the underlying sequence, but that doesn't mean there is any way to find that martingale, at least in a practical sense. If done properly, this can make internal and external sampling analogous in the way I require. As I have discussed with you before, the agent must not be able to use this sequence as a counter or other type of memory. In decision theory problems that involve (partial) memory wipes, or the logical equivalent of partial memory wipes that are actually done by copying, or involve multiple agents that are supposed to be independent, it is required to make sure the sequences are advanced or rewound to the correct positions at setup time, and aren't mixed up between agents incorrectly.

In your particular case, you can enforce reliance on iteration order (or the equivalent in LLMs, what tokens get generated in what order under a particular deterministic sampler) by setting all utility to 0 if the sequence has any influence, direct or indirect, on the agent except through the proper functioning of the correct in-scenario sampling device, or if the sequence or its generation position are modified by the environment (defined to include the agent).

If we apply an EDT-like formulation for UDT 1.1, we might get higher cooperation rates compared with a "logical counterfactual version"

Maybe. It might not be worth doing though, since the equation would require lots of exposition that contains claims about what certain functions do without providing the actual functions. These claims would need to be considered convincing by the LLMs. I don't know how to formalize a Mathematical Intuition Module that does what Wei Dai claims it does, and I don't immediately know what would be required to make concrete robust to adversarial attack.

You could try to figure out what Vladimir Nesov is talking about re. logical time instead, since what I think I can do in physical time is quite simple. Things would still start out simple in logical time, but it could be substantially more interesting. Such work may be able to lead to a better-behaved Mathematical Intuition Module.

Known difference is Wei Dai dis-endorses logical counterfactuals for UDT

Yes, but I'm not sure why. I think it's possible he imagines the Mathematical Intuition Module estimator to be much more sophisticated than the formalism suggests. See his comment on Happy Dance^[3].

It's a known stats anti-pattern to use 'sample sizes' as a further criterion on top of significance testing.

I know enough people think that to make it worth saying what I did. I haven't carefully studied non-asymptotic non-Bayesian stats.

^{^}
"Physically" within the scenario, you don't have to build actual hardware.
^{^}
https://en.wikipedia.org/wiki/Mersenne_Twister
^{^}
https://www.alignmentforum.org/posts/5bd75cc58225bf067037550e/the-happy-dance-problem?commentId=5bd75cc58225bf0670375513

There are two different issues here

The experiment is not exactly random because it's PRNG
The experiment doesn't cover cases of agents with total control / determinism

I, understandably, understood your original comment to be discussing (2) and then noted that it's just not what I'm studying, and is also not realistic (i.e. applies to neither humans nor ordinary LLMs. You are discussing alternative ways to run LLMs deterministically with error correction but it's speculative and it is not what I'm studying.)

I just don't think 2 is a problem at all. It's out of scope. You can speculate that in the future, agents will be deterministic, rather than stochastic the way humans and LLMs are (at least when LLMs use secure RNG). But that's not what I'm studying. I am using stats to deal with the fact that LLMs do not give results that are practically deterministic from my perspective, i.e., there is no one single essay Opus 5.5 (high) would write in response to a given prompt that asks it to write an essay about A-FDT.

I think 1 could be a real problem, it would technically break the assumptions of null hypothesis significance testing to have a PRNG, because it means that the trials are not statistically independent. I'm assuming it's not a significant issue in practice, because it's not like I designed the prompts to hack Anthropic's PRNG to spit out the results I wanted. If hypothetically Anthropic's PRNG were cryptographically secure, then in a sense I couldn't "tell it apart from random", so I couldn't break it by manipulating the prompts. If it's not secure, it would still take active work (which I didn't do) to make it predictable, it wouldn't happen accidentally. I would be very skeptical of a claim like "confidence intervals based on LLM queries are practically invalid because of PRNG" given this.

It does raise an interesting question of what the implications would be of insecure PRNG in token sampling. Like, maybe that allows certain "prompt injections" that move towards predictable PRNG and then make the model do an unintended thing that it would ordinarily be highly unlikely to do. I assume people in the labs have thought about this.

Yes, but I’m not sure why.

This is completely reasonable, we have logical induction as a constructive model of logical uncertainty / mathematical intuition model, we have no such constructive model of logical counterfactuals, and we have the cat-belling issue I described, which Vladimir Nesov doesn't even claim to solve. I think Wei Dai realized that logical counterfactuals are philosophically problematic and didn't want to endorse them.

As a result of your recent posts, here are my thoughts:
- CDT as defined by Lewis is just a maximisation rule, BYO mixture of dependency hypotheses
- Causal variants of FDT are the same maximisation rule, plus informal rules for guiding the set of acts you should consider plus the dependency hypotheses
- Thus rather than "causal FDT" it might be more accurate to say "functional CDT"
- The dependency hypotheses defended by FDT authors are not in any meaningful sense "logical". That is, the dependence between your act (or policy) and Omega's output (or policy) is not in practice derived from any standard or non-standard logic, it is an additional ingredient
- There is no precise account for how to decide what this and other dependences should be, even if it is in some cases intuitively obvious
- Which is a problem shared by every decision theory
- The advocates of FDT are right that vanilla CDT's (informal) prescription that "nothing can depend on your actions if they're in your actions' past lightcone" runs into problems with capable predictors
- I think that the method of adjudication by FDT authors - pick a bunch of agents between whom competition is "fair", and the best theory is the one instantiated by the agent who wins in expectation - is promising, and at least different to trying to supply an alternative notion of counterfactual
- However, there may be enduring difficulties with deciding which reference classes are fair (just a guess)

What about policy FDT? The paper does not even define policy FDT using an equation.

(The multiply lets you return a result if you can show (or guess) that at least one side is 0.)

is the Mathematical Intuition Module that must always give a number between 0 and 1 (both inclusive) when presented with this equation, an , and an .

Note that everything that is directly in bold is concrete, or at least "could be" as far as this iteration of the loop is concerned. This is not the case for all things touching bold text, however.

(I am also not sure how many cases this would be useful in where all the agents don't essentially tile from each other (where even if some of them are slightly different, it could not be by much).)

^{^}
Particularly the effect size of whatever Anthropic does internally to reduce token use, I suppose.
^{^}
https://www.lesswrong.com/posts/g8xh9R7RaNitKtkaa/explicit-optimization-of-global-strategy-fixing-a-bug-in
^{^}
I do not currently know how to implement this in any more efficient way.

Frankly I think the first part of your reply is completely wrong.

“actual” PD decision theory situation

This also means the math for significance is somewhat off even if the clustering is not a sampler error (the measure on the permutations doesn’t line up), but none of this really matters due to the problems that you noticed in the LLM reasoning.

Math of significance assumes stochasticity. There's some parameter for "the actual rate". Standard stats stuff. It doesn't require determinism.

I think your sample sizes are small enough that you won’t confuse anyone

The closest thing is UDT 1.1

You use terms like "actual" and "real" to refer to artificial cases of total control. These might appear on game theory whiteboards but are not realistic for humans or LLMs.

Humans do not have full internal control over their brain states.

simple case being, eyes detecting photons, but measurements also happen in neurons

Math of significance assumes stochasticity. There's some parameter for "the actual rate". Standard stats stuff. It doesn't require determinism.

If we apply an EDT-like formulation for UDT 1.1, we might get higher cooperation rates compared with a "logical counterfactual version"

Known difference is Wei Dai dis-endorses logical counterfactuals for UDT

Yes, but I'm not sure why. I think it's possible he imagines the Mathematical Intuition Module estimator to be much more sophisticated than the formalism suggests. See his comment on Happy Dance^[3].

It's a known stats anti-pattern to use 'sample sizes' as a further criterion on top of significance testing.

I know enough people think that to make it worth saying what I did. I haven't carefully studied non-asymptotic non-Bayesian stats.

^{^}
"Physically" within the scenario, you don't have to build actual hardware.
^{^}
https://en.wikipedia.org/wiki/Mersenne_Twister
^{^}
https://www.alignmentforum.org/posts/5bd75cc58225bf067037550e/the-happy-dance-problem?commentId=5bd75cc58225bf0670375513

There are two different issues here

The experiment is not exactly random because it's PRNG
The experiment doesn't cover cases of agents with total control / determinism

Yes, but I’m not sure why.