Cox's theorem literally has as a desideratum that the results should be identical to classical logic when you're completely certain about the axioms. This is what's violated.
I'll try illustrating this with an example.
Suppose we have a calculator that wants to add 298+587. If it can only take small steps, it just has to start with 298, and then do 298+1=299 (keeping track that this is step 1), 299+1=300 (step 2), 300+1=301 (that's 3), etc, until it reaches 884+1=885 (step 587), at which point the calculator put's "885" on the screen as its last step. And so our calculator, when you input 298+587, outputs 885.
In order to add two big numbers, our calculator only ever had to add 1 to things - the stored number and the step count - which is pretty cool.
The output follow inexorably follows from the rules of this calculator. When the first input was 298 and the step count is 587, the stored number is 885, and when the step count is equal to the second input, the calculator puts the stored number on the screen. We cannot get a different answer without using a calculator that follows different rules.
Now suppose we build a calculator that does the same thing but with different labels. We ask it for P(298+587=885 | standard axioms). So it start with something it knows - P(298=298)=1. Then it moves to P(298+1=299)=1. Then P(298+2=300)=1. Eventually it reaches P(298+587=885)=1, so it outputs "1." This is just a different label on our normal calculator. Every step it increments two numbers, one on the left and one on the fight, and then it stops when the incremented number on the left reaches 587. It's the same process.
This is the translation that obey's Cox's theorem. It's the same process as classical logic, because when your'e certain about axioms probabilistic logic and classical logic are equivalent.
Logical uncertainty is like saying that P(298+587=885 | standard axioms) = 0.5. This violates Cox's theorem because it doesn't agree with our "re-labeled calculator." But why is that bad / weird?
Well, suppose that even when we're logically uncertain, we can agree that P(298=298)=1, and that P(298+1=299)=1. So why can't we just do the calculator process, and keep adding 1 to both sides until we reach the desired answer? There must be some step where the calculator process breaks down, where our logically uncertain agent accepts that , e.g., P(298+100=398)=1, but thinks P(298+101=399)=0.5
In classical probability, this is not allowed. If you think that P(298+100=398)=1, and that adding 1 to both sides preserves probability (one of the axioms that defines addition), then it follows that P(298+101=391)=1. Anything else is... illogical.
Thanks for taking the time to elaborate.
I don't recall that desideratum in Jaynes' derivations. I think it is not needed. Why should it be needed? Certainty about axioms is a million miles from certainty about all their consequences, as seems to be the exact point of your series.
Help me out, what am I not understanding?
Followup To: Logic as Probability
If we design a robot that acts as if it's uncertain about mathematical statements, that violates some desiderata for probability. But realistic robots cannot prove all theorems; they have to be uncertain about hard math problems.
In the name of practicality, we want a foundation for decision-making that captures what it means to make a good decision, even with limited resources. "Good" means that even though our real-world robot can't make decisions well enough to satisfy Savage's theorem, we want to approximate that ideal, not throw it out. Although I don't have the one best answer to give you, in this post we'll take some steps forward.
Part of the sequence Logical Uncertainty
Previous Post: Logic as Probability
Next post: Solutions and Open Problems