All of SMK's Comments + Replies

I think it would be good if you made clear in the abstract what your contributions to the literature are, and how your results relate to those of e.g. Kierland and Monton (2005).

Yes, of course we can still quibble with the assumptions (like the OP does in some cases), which is why I say "moderate evidence" rather than "completely watertight proof", but given how natural the assumptions are, the evidence is good.

Completeness is arguably not natural (see e.g. Aumann, 1962; Bradley, 2017, §11.5). In particular, I think it is clearly not a requirement of rationality.

Edit: Actually, I think my deeper objection is that most of the critiques here (made by Sammy) are just wrong. For example, of course Dutch books/money pumps frequently get invoked to justify VNM axioms. See for example this.

Sami never mentioned money pumps. And "the Dutch books arguments" are arguments for probabilism and other credal norms^[1], not the vNM axioms.

1. ^{^}

   Again, see Pettigrew (2020) (here is a PDF from Richard's webpage).

You are conflating the Dutch book arguments for probabilism (Pettigrew, 2020) with the money-pump arguments for the vNM axioms (Gustafsson, 2022). 

The normal VNM approach is to start with an agent whose behavior satisfies some common sense conditions: can't be money pumped and so on.

Nitpicks: (1) the vNM theorem is arguably about preference, not choice and behavior; and (2) "can't be money pumped" is not one of the conditions in the theorem.

and Silvia's work

Typo: it's Sylvia.

I wrote "I'm really not sure at this point whether UDT is even on the right track" in UDT shows that decision theory is more puzzling than ever which I think you've read? Did you perhaps miss that part?

Yes, missed or forgot about that sentence, sorry.

(BTW this issue/doubt about whether UDT / paying CM is normative for humans is item 1 in the above linked post. Thought I'd point that out since it may not be obvious at first glance.)

Thanks.

Do you have more examples where making such distinctions would be helpful?

I was mostly thinking about discuss... (read more)

Here's a related idea that is maybe clearer: Suppose an agent has the ability to self-modify to use any decision theory, would they decide to stick with their current decision theory? (I'm actually not sure what term has been explicitly defined to mean this, so I'll just call it "self-endorsement" for now.)

This sounds similar to what's called "self-recommendation"—see e.g. Skyrms (1982, pp. 707-709), Meacham (2010, §3.3) and Pettigrew (2023). In the abstract Pettigrew writes: "A decision theory is self-recommending if, when you ask it which decision theory... (read more)

Thanks for the clarification!

I do understand from the SEP, like Wei, that sophisticated means "backwards planning", and resolute means "being able to commit to a policy" (correct me if I'm wrong).

That seems roughly correct, but note that there are different interpretations of resolute choice floating around^[1], and I think McClennen's (1990) presentation is somewhat unclear at times. Sometimes resoluteness seems to be about the ability to make internal commitments, and other times it seems to be about being sensitive to the dynamic context in a particular ... (read more)

I think Sami's comment is entirely fair given the language and framing of the original post. It is of course fine to forget about references, but e.g. "I find it curious that none of my ideas have a following in academia or have been reinvented/rediscovered by academia" and "Clearly academia has some blind spots, but how big?" reads like you don't consider it a possilbity that you might have re-invented something yourself, and that academics are at fault for not taking up your ideas.

I don't think cohesive decision theory is being discussed much, but I'm not sure. Perhaps because the theory is mainly used to argue against the claim that "every decision rule will lead agents who can’t bind themselves to disaster" (p. 20, footnote 34) in the paper, and discussion of its independent interest is relegated to a footnote (footnote 34).

It would be interesting to get an overview of what these are. Or if that's too hard to write down, and there are no ready references, what are your own interests in decision theory?

Yeah, that would be too hard. You might want to look at these SEP entries: Decision Theory, Normative Theories of Rational Choice: Expected Utility, Normative Theories of Rational Choice: Rivals to Expected Utility and Causal Decision Theory. To give an example of what I'm interested in, I think it is really important to take into account unawareness and awareness growth (see §5... (read more)

There are many many interesting questions in decision theory, and "dimensions" along which decision theories can vary, not just the three usually discussed on LessWrong. It's not clear to me why (i) philosophers should focus on the dimensions you primarily seem to be interested in, and (ii) what is so special about the particular combination you mention (is there some interesting interaction I don't know about maybe?). Furthermore, note that most philosophers probably do not share your intuitions: I'm pretty sure most of them would e.g. pay in counterfactu... (read more)

This is indeed what happens to the best-known decision theories (CDT and EDT): they want to commit to paying, but if they don’t, by the time they get to the Heads world they don’t pay. We call this dynamic instability, because different (temporal) versions of the agent seem to be working against each other.

Unless you are using "dynamic stability" to mean something other than "dynamic consistency", I don't think this is quite right. The standard philosophical theory of dynamic choice, sophisticated choice (see e.g. the SEP entry on decision theory), would not pay but is still dynamically consistent. 

The reason for the former is that I (and others) have been unable to find a rigorous formulation of it that doesn't have serious open problems. (I and I guess other decision theory researchers in this community currently think that UDT is more of a relatively promising direction to explore, rather than a good decision theory per se.)

That's fair. But what is it then that you expect academics to engage with? How would you describe this research direction, and why do you think it's interesting and/or important?

Could you perhaps say something about what a Kripkean semantics would look like for your logic?

On your first point: as Sami writes, resolute choice is mentioned in the introductory SEP article on dynamic choice (it even has its own section!), as well as in the SEP article on decision theory. And SEP is the first place you go when you want to learn about philosophical topics and find references.

On your second point: as I wrote in my comment above, (i) academics have produced seemingly similar ideas to e.g. updatelessness (well before they were written up on LW) so it is unclear why academics should engage with less rigorous, unpublished proposals tha... (read more)

I think the main reason why UDT is not discussed in academia is that it is not a sufficiently rigorous proposal, as well as there not being a published paper on it. Hilary Greaves says the following in this 80k episode:

Then as many of your listeners will know, in the space of AI research, people have been throwing around terms like ‘functional decision theory’ and ‘timeless decision theory’ and ‘updateless decision theory’. I think it’s a lot less clear exactly what these putative alternatives are supposed to be. The literature on those kinds of decision t

Thanks.

I am pretty sure they're interchangeable however.

Do you have a reference for this? Or perhaps there is a quick proof that could convince me?

We assume two rules of inference:

Necessitation: $⊢x⟹⊢\square x$ 
Distributivity: $⊢\square (x\to y)⟹⊢\square x\to \square y$

Is there a reason why this differs from the standard presentation of K? Normally you would say that K is generated by the following (coupled with substitution):

Axioms:
- All tautologies of propositional logic.
- Distribution: $\square (x\to y)\to (\square x\to \square y)$.

Rules of inference:
- Necessitation: $⟨x,\square x⟩$.
- Modus ponens: $⟨x\to y,x,y⟩$.

Some people know that they do not have a solution. Andy Egan, in "Some Counterexamples to Causal Decision Theory" (1999, Philosophical Review)

This should say 2007.

These people all defect in PD and two-box in Newcomb. 

Spohn argues for one-boxing in Reversing 30 years of discussion: why causal decision theorists should one-box.

Thanks.

Roughly, you don't actually get to commit your future-self to things. Instead, you just do what you (in expectation) would have committed yourself to given some reconstructed prior.

Agreed.

Just as a literature pointer: If I recall correctly, Chris Meacham's approach in "Binding and Its Consequences" is ultimately to estimate your initial credence function and perform the action from the plan with the highest EU according to that function.

Yes, that's a great paper! (I think we might have had a footnote on cohesive decision theory in a draft of... (read more)

What do you mean by "the Bayesian Conditionalization thing" in this context? (Just epistemically speaking, standard Conditionalization is inadequate for dealing with cases of awareness growth. Suppose, for example, that one was aware of propositions {X, Y}, and that this set later expands to {X, Y, Z}. Before this expansion, one had a credence P(X ∨ Y) = 1, meaning Conditionalization recommends remaining certain in X ∨ Y; i.e., one is only permitted to place a credence P(Z) = 0. Maybe you are referring to something like Reverse Bayesianism?)

It's fairly clear to me that the authors do not have any specific and precise method in mind, Bjerring or no Bjerring.

Of course they don't have a specific proposal in the paper. I'm just saying that it seems like they would want to be more precise, or that a full specification requires more work on counterpossibles (which you seem to be arguing against). From the abstract:

While not necessary for considering classic decision theory problems, we note that a full specification of FDT will require a non-trivial theory of logical counterfactuals and algorithmic

Right, but it's fairly clear to me that this is not what the authors have in mind. For example, they cite Bjerring (2014), who proposes very specific and precise extensions of the Lewis-Stalnaker semantics.

(See my response to gjm's comment.)

To be sure, switching to Bet 1 is great evidence that $\neg P$ is true (that's the whole point), but that's not the sort of reasoning FDT recommends. Rather, the question is if we take the Peano axioms to be downstream of the output of the algorithm in the relevant sense. 

As the authors make clear, FDT is supposed to be "structurally similar" to CDT ^[1], and in the same way CDT regards the history and the laws to be out of their control in Ahmed's problems, FDT should arguably regard the Peano axioms to be out of their control (i.e., "upstrea... (read more)

Regarding Sleeping Counterfact: there seems to be two updates you could make, and thus there should be conceptual space for two interesting ways of being updatelessness in this problem; you could be 'anthropically updateless', i.e., not update on your existence in the standard Thirder way, and you could also be updateless with respect to the researchers asking for money (just as in counterfactual mugging). And it seems like these two variants will make different recommendations.

Suppose you make the first update, but not the second. Then the evidentialist v... (read more)

Interesting! Did thinking about those variants make you update your credences in SIA/SSA (or else)?

No, not really! This was mostly just for fun.

My follow-up question for almost all of them though, is based on use of the word "should" in the question. Since it presumably is not any moral version of "should", it's presumably a meaning in the direction of "best achieves a desired outcome".

The 'should' only designates what you think epistemic rationality requires of you in the situation. That might be something consequentialist (which is what I think you mean by "best achieves a desired outcome"), like maximizing accuracy^[1], but it need not be; you could think there are other norms^[2]. 

To see wh... (read more)

Ah, okay, got it. Sorry about the confusion. That description seems right to me, fwiw.

Thanks for clarifying. I still don't think this is exactly what people usually mean by ECL, but perhaps it's not super important what words we use. (I think the issue is that your model of the acausal interaction—i.e. a PD with survival on the line—is different to the toy model of ECL I have in my head where cooperation consists in benefitting the values of the other player [without regard for their life per se]. As I understand it, this is essentially the principal model used in the original ECL paper as well.)

The effective correlation is likely to be (much) larger for someone using UDT.

Could you say more about why you think this? (Or, have you written about this somewhere else?) I think I agree if by "UDT" you mean something like "EDT + updatelessness"^[1]; but if you are essentially equating UDT with FDT, I would expect the "correlation"/"logi-causal effect" to be pretty minor in practice due to the apparent brittleness of "logical causation".

Correlation and kindness also have an important nonlinear interaction, which is often discussed under the heading of “ev

Related: A bargaining-theoretic approach to moral uncertainty by Greaves and Cotton-Barratt. Section 6 is especially interesting where they highlight a problem with the Nash approach; namely that the NBS is variant to whether (sub-)agents are bargaining over all decision problems (which they are currently facing and think they will face with nonzero probability) simultaneously, or whether all bargaining problems are treated separately and you find the solution for each individual problem—one at a time.

In the 'grand-world' model, (sub-)agents can bargain ac... (read more)

Now, let's pretend you are an egalitarian. You still want to satisfy everyone's goals, and so you go behind the veil of ignorance, and forget who you are. The difference is that now you are not trying to maximize expected expected utility, and instead are trying to maximize worst-case expected utility.

Nitpick: I think this is a somewhat controversial and nonstandard definition of egalitarianism. Rather, this is the decision theory underlying Rawls' 'justice as fairness'; and, yes, Rawls claimed that his theory was egalitarian (if I remember correctly), but... (read more)

I agree that the latter two examples have Moorean vibes, but I don't think they strictly speaking can be classified as such (especially the last one). (Perhaps you are not saying this?) They could just be understood as instances of modus tollens, where the irrationality is not that they recognize that their belief has a non-epistemic generator, but rather that they have an absurdly high credence in $\neg Q$, i.e. "my parents wouldn't be wrong" and "philosophers could/should not be out of jobs".

The same holds if Alice is confident in Bob's relevant conditional behavior for some other reason, but can't literally view Bob's source code. Alice evaluates counterfactuals based on "how would Bob behave if I do X? what about if I do Y?", since those are the differences that can affect utility; knowing the details of Bob's algorithm doesn't matter if those details are screened off by Bob's functional behavior.

Hm. What kind of dependence is involved here? Doesn't seem like a case of subjunctive dependence as defined in the FDT papers; the two algorithms a... (read more)

I'm not claiming this (again, it's about relative not absolute likelihood).

I'm confused. I was comparing the likelihood of (3) to the likelihood of (1) and (2); i.e. saying something about relative likelihood, no?

I'm not saying this is likely, just that this is the most plausible path I see by which UDT leads to nice things for us.

I meant for my main argument to be directed at the claim of relative likelihood; sorry if that was not clear. So I guess my question is: do you think the updatelessness-based trade you described is the most plausible type of acau... (read more)

I had something like the following in mind: you are playing the PD against someone implementing "AlienDT" which you know nothing about except that (i) it's a completely different algorithm to the one you are implementing, and (ii) that it nonetheless outputs the same action/policy as the algorithm you are implementing with some high probability (say 0.9), in a given decision problem.

It seems to me that you should definitely cooperate in this case, but I have no idea how logi-causalist decision theories are supposed to arrive at that conclusion (if at all).

What's your take on playing a PD against someone who is implementing a different decision algorithm to the one you are implementing, albeit strongly (logically) correlated in terms of outputs?

Insofar as I have hope in decision theory leading us to have nice things, it mostly comes via the possibility that a fully-fleshed-out version of UDT would recommend updating "all the way back" to a point where there's uncertainty about which agent you are. (I haven't thought about this much and this could be crazy.)

This was surprising to me. For one, that seems like way too much updatelessness. Do you have in mind an agent self-modifying into something like that? If so, when and why? Plausibly this would be after the point of the agent knowing whether it ... (read more)

From Arif Ahmed's Evidence, Decision and Causality (ch. 5.4, p. 142-143; links mine):

Deliberating agents should take their choice to be between worlds that differ over the past as well as over the future. In particular, they differ over the effects of the present choice but also over its unknown causes. Typically these past differences will be microphysical differences that don’t matter to anyone. But in Betting on the Past they matter to Alice.

. . .

On this new picture, which arises naturally from [evidential decision theory]. . ., it is misleading t

The difficulty is in how to weight the frequency/importance of the situations they face. 

I agree with this. On the one hand, you could just have a bunch of Procreation problems which would lead to the FDTer ending up with a smaller pot of money; or you could of course have a lot of Counterfactual Muggings in which case the FDTer would come out on top—at least in the limit.

Ah, nice. I was just about to recommend sections 2.6.2 and 3 of Multiverse-wide Cooperation via Correlated Decision Making by Caspar.

From this, they trivially conclude that EDT will have higher stakes than CDT: if there are more Good Twins (Evil Twins), EDT will recommend one-boxing (two-boxing) very strongly, since this will provide evidence to you about many agents doing the same. But I'm not satisfied with this answer, because if you don't know whether more Good Twins or Evil Twins exist, you won't be obtaining that evidence (upon taking the decision)!

I don't think this is a situation of evidential symmetry which would warrant a uniform distribution (i.e. you can't just say that "you... (read more)

Procreation* gives both FDT and CDT agents (and indeed, all agents) the same dilemma. FDT agents procreate and live miserably; CDT agents don't procreate and almost certainly don't exist. FDT beats CDT in this dilemma.

This doesn't seem right: you already exist! In order to say that "FDT beats CDT" I think you have to argue that one should care about the number of branches you exist in—which is what you plausibly have uncertainty about, not about whether this very instance of you exists. (And this is arguably just about preferences, as Christiano writes abo... (read more)