I argue that the correct notion of logical counterfactual is the one which an agent would self-modify to have if the agent lacked it, and that this significantly constrains the possibilities. The argument is rather informal, and I'm not really sure if many of the points are new, so I'm posting it in Discussion.

Where the idea is coming from

(This section is an extended motivation, and may be skipped.)

At the most recent MIRIxLA, Alex Mennen led a discussion attempting to apply conservation of expected evidence to the Demski prior. The basic idea was that approximations of the Demski prior discussed so far can do quite unreasonable things before converging to good probabilities, even those that are uniformly coherent. Can we describe a property for approximations such that the probabilities given at finite times are in some sense reasonable? Based on the idea of conservation of expected evidence, is it possible for an approximation to give finite probabilities such that (in some sense) the probability that the estimate will drift up later is balanced by the probability it will drift down?

We didn't come up with any formulations of this idea which seemed both possible and nontrivial, and we came away with the feeling that something like what we were after would not be possible in the end. The version which we spent the longest time discussing was this one:

(1) $\forall t, a \leq b : E_{ϕ} (P (ϕ) | P_{t} (ϕ) \in [a, b]) \in [a, b]$

$P_{t} ()$ represents the probability estimate at time $t$ , while $P ()$ is the probability reached in the limit of the approximation process. The expectation is being taken over $ϕ$ ; we suppose a probability distribution $μ (ϕ)$ which we're sampling sentences from. At any time $t$ , sampling only $ϕ$ whose probability estimate lands in a range $[a, b]$ , we want the expected value of the eventual probabilities to be in that same range. (Although inspired by conservation of expected evidence, this might also be seen as a notion of calibration.)

Although we convinced ourselves that this was very likely impossible, we didn't come up with a proof. None of our variations to the definition seemed very promising either.

Somewhat facetiously, I offered the following resolution to our difficulties. One thing which seems artificial about (1) is the expectation over $ϕ$ based on $μ$ . What this accomplishes is to inject a kind of made-up uncertainty into the logically certain world; if we turned a blind eye to the

...