It seems to me you are neglecting the proposition "A->B"

Do you know what truth tables are? The statement "A->B" can be represented on a truth table. A and B can be possible. not-A and B can be possible. Not-A and not-B can be possible. But A and not-B is impossible.

A->B and the four statements about the truth table are interchangeable. Even though when I talk about the truth table, I never need to use the "->" symbol. They contain the same content because A->B says that A and not-B is impossible, and saying that A and not-B is impossible says that A->B. For example, "it raining but not being wet outside is impossible."

In the language of probability, saying that P(B|A)=1 means that A and not-B is impossible, while leaving the other possibilities able to vary freely. The product rule says P(A and not-B) = P(A) * P(not-B | A). What's P(not-B | A) if P(B | A)=1? It's zero, because it's the negation of our assumption.

Writing out things in classical logic doesn't just mean putting P() around the same symbols. It means making things behave the same way.

If this somehow violates Savage or Cox's theorems I'd like to know why

Well, Cox's theorem has as a requirement that when your axioms are completely certain, you assign probability 1 to all classical consequences of those axioms. Assigning probability 0.5 to any of those consequences thus violates Cox's theorem. But this is kind of unsatisfying, so: where do we violate the product rule?

Suppose our robot knows that P(wet outside | raining) = 1. And it observes that it's raining, so P(rain)=1. But it's having trouble figuring out whether it's wet outside within its time limit, so it just gives up and says P(wet outside)=0.5. Has it violated the product rule? Yes. P(wet outside) >= P(wet outside and raining) = P(wet outside | rain) * P(rain) = 1.

If we accept that the axioms have probability 1, we can deduce the consequences with certainty using the product rule. If at any point we stop deducing the consequences with certainty, this means we have stopped using the product rule.

But it turns out that there is one true probability distribution over mathematical statements, given the axioms. The right distribution is obtained by straightforward application of the product rule - never mind that it takes 4^^^3 steps - and if you deviate from the right distribution that means you violate the product rule at some point.

This does not seem right to me. I feel like you are sneakily trying to condition all of the robots probabilities on mathematical proofs that it does not have a-priori. E.g. consider A, A->B, therefore B. To learn that P(A->B)=1, the robot has to do a big calculation to obtain the proof. After this, it can conclude that P(B|A,A->B)=1. But before it has the proof, it should still have some P(B|A)!=1.

Sure, it seems tempting to call the probabilities you would have after obtaining all the proofs of everything the "true" probabilties, but to me it doesn't actually seem different to the claim that "after I roll my dice an infinity of times, I will know the 'true' probability of rolling a 1". I should still have some beliefs about a one being rolled before I have observed vast numbers of rolls.

In other words I suggest that proof of mathematical relationships should be treated exactly the same as any other data/evidence.

edit: in fact surely one has to consider this so that the robot can incorporate the cost of computing the proof into its loss function, in order to decide if it should bother doing it or not. Knowing the answer for certain may still not be worth the time it takes (not to mention that even after computing the proof the robot may still not have total confidence in it; if it is a really long proof, the probability that cosmic rays have caused lots of bit-flips to mess up the logic may become significant. If the robot knows it cannot ever get the answer with sufficient confidence within the given time constraints, it must choose an action which accounts for this. And the logic it uses should be just the same as how it knows when to stop rolling die).

edit2: I realised I was a little sloppy above; let me make it clearer here:

The robot knows P(B|A,A->B)=1 apriori. But it does not know "A->B" is true apriori. It therefore calculates

P(B|A) = P(B|A,A->B) P(A->B|A) + P(B|A,not A->B) P(not A->B|A) = P(A->B|A)

After it obtains proof that "A->B", call this p, we have P(A->B|A,p) = 1, so

P(B|A,p) = P(B|A,A->B,p) P(A->B|A,p) + P(B|A,not A->B,p) P(not A->B|A,p)

collapses to

P(B|A,p) = P(B|A,A->B,p) = P(B|A,A->B) = 1

But I don't think it is reasonable to skip straight to this final statement, unless the cost of obtaining p is negligible.

edit3: If this somehow violates Savage or Cox's theorems I'd like to know why :).

Yeah, that sounds right. You could say that a "true" number is a model parameter that fits the observed data well.

I can pass along a recommendation I have received: Operational Subjective Statistical Methods by Frank Lad. I haven't read the book myself, so I can't actually vouch for it, but it was described to me as "excellent". I don't know if it is actively prediction-centered, but it should at least be compatible with that philosophy.

An example of a "true number" is mass. We can measure the mass of a person or a car, and we use these values in engineering all the time. An example of a "fake number" is utility. I've never seen a concrete utility value used anywhere, though I always hear about nice mathematical laws that it must obey.

It is interesting that you choose mass as your prototypical "true" number. You say we can "measure" the mass of a person or car. This is true in the sense that we have a complex physical model of reality, and in one of the most superficial levels of this model (Newtonian mechanics) there exist some abstract numbers which characterise the motions of "objects" in response to "forces". So "measuring" mass seems to only mean that we collect some data, fit this Newtonian model to that data, and extract relatively precise values for this parameter we call "mass".

Most of your examples of "fake" numbers seem to be to be definable in exactly analogous terms. Your main gripe seems to be that different people try to use the same word to describe parameters in different models, or perhaps that there do not even exist mathematical models for some of them; do you agree? To use a fun phrase I saw recently, the problem is that we are wasting time with "linguistic warfare" when we should be busy building better models?

I understand this, though I hadn't thought of it with such clear terminology. I think the point Jonah was making was that in many cases, people are talking about propensities/frequencies when they refer to probabilities. So it's not so much that Jonah or I are confusing epistemic probabilities with propensities/frequencies, it's that many people use the term "probability" to refer to the latter. With language used this way, the probability distribution for this model parameter can be called the "probability distribution of the probability estimate." If you reserve the term probability exclusive to epistemic probability (degree of belief) then this would constitute an abuse of language.

I'd guess that in Geisser-style predictive inference, the meaning or reality or what-have-you of G is to be found in the way it encodes the dependence (or maybe, compresses the description) of the joint multivariate predictive distribution. But like I say, that's not my school of thought -- I'm happy to admit the possibility of physical model parameters -- so I really am just guessing.

Yup, I'm referring to de Finetti's theorem. Thing is, de Finetti himself would have denied that there is such a thing as a parameter -- he was all about only assigning probabilities to observable, bet-on-able things. That's why he developed his representation theorem. From his perspective, p arises as a distinct mathematical entity merely as a result of the representation provided by exchangeability. The meaning of p is to be found in the predictive distribution; to describe p as a bias parameter is to reify a concept which has no place in de Finetti's Bayesian approach.

Now, I'm not a de-Finetti-style subjective Bayesian. For me, it's enough to note that the math is the same whether one conceives of p as stochastic model parameter or as the degree of plausibility of any single outcome. That's why I say it's not either/or.

This is just a parameter of a stochastic model, not a degree of belief.

This is not exactly correct. It's true that in general there's a sharp distinction to be made between model parameters (which govern/summarize/encode properties of the entire stochastic process) and degrees of belief for various outcomes, but that distinction becomes very blurry in the current context.

What's going on here is that the probability distribution for the observable outcomes is infinitely exchangeable. Infinite exchangeability gives rise to a certain representation for the predictive distribution under which the prior expected limiting frequency is mathematically equal to the marginal prior probability for any single outcome. So under exchangeability, it's not an either/or -- it's a both/and.