For Godel-Bayes issues, you can start with the responses to my post on the subject. (I've since learned and remembered more about Godel.)

We should have the ability to talk about subjective uncertainty in, at the very least, particular proofs and probabilities. I don't know that we can. But I like the following argument, which I recall seeing here somewhere:

If there exists a perfect probability calculation based on a set of background information, it must take this uncertainty into account. Therefore, applying this uncertainty again to the answer would mean double-counting the evidence, which is strictly verboten. We therefore cannot use this line of reasoning to produce a contradiction. Barring other arguments, we can assume the uncertainty equals a really small fraction.