In particular, I intuitively believe that "my beliefs about the integers are consistent, because the integers exist". That's an uncomfortable situation to be in, because we know that a consistent theory can't assert its own consistency.
That is true, however you don't appear to be asserting the consistency of your beliefs, you are asserting the consistency of a particular subset of your beliefs which does not contain the assertion of its consistency. This is not in conflict with Gödel's incompleteness theorem which implies that no theory may consistently assert its own consistency. Gödel's incompleteness theorem does not forbid proofs of consistency by more powerful theories: for example there are proofs of the consistency of Peano arithmetic
Yeah, that's a fair point. If I believed the sentence "my beliefs about the integers are consistent", it would be a pretty complicated sentence about the integers, containing an encoding of itself by the diagonal lemma. Maybe you're right that I don't actually believe that, not even intuitively. I just believe a bunch of other sentences, and believe that they are consistent. That would agree with the conclusion of the post, that my beliefs about the integers (both actual and extrapolated) can be covered by some specific formal theory.
My beliefs about the integers are a little fuzzy. I believe the things that ZFC can prove about the integers, but there seems to be more than that. In particular, I intuitively believe that "my beliefs about the integers are consistent, because the integers exist". That's an uncomfortable situation to be in, because we know that a consistent theory can't assert its own consistency.
Should I conclude that my beliefs about the integers can't be covered by any single formal theory? That's a tempting line of thought, but it reminds me of all these people claiming that the human mind is uncomputable, or that humans will always be smarter than machines. It feels like being on the wrong side of history.
It's also dangerous to believe that "the integers exist" due to my having clear intuitions about them, because humans sometimes make mistakes. Before Russell's paradox, someone could be forgiven for thinking that the objects of naive set theory "exist" because they have clear intuitions about sets, but they would be wrong nonetheless.
Let's explore the other direction instead. What if there was some way to extrapolate my fuzzy beliefs about the integers? In full generality, the outcome of such a process should be a Turing machine that prints sentences about integers which I believe in. Such a machine would encode some effectively generated theory about the integers, which we know cannot assert its own consistency and be consistent at the same time.
So it seems that in the process of extracting my "consistent extrapolated beliefs", something has to give. At some point, my belief in my own consistency has to go, if I want the final result to be consistent.
But if I already know that much about the outcome, it might make sense for me to change my beliefs now, and end up with something like this: "All my beliefs about the integers follow from some specific formal theory that I don't know yet. In particular, I don't believe that my beliefs about the integers are consistent."
I'm not sure if there are gaps in the above reasoning, and I don't know if using probabilistic reflection changes the conclusions any. What do you think?