If you buy the Bayesian argument (e.g. in Jaynes) that there is a single correct Pr(A|I) where I is your state of information and A is any proposition in question, then p, an estimate of Pr(A|I), is wrong if and only if p != Pr(A|I). In practice, we virtually never know Pr(A|I), so we can't make this check. But as far as a conceptual understanding goes, that's it - if, as I said, you buy the argument.

In practice, we check the things gwern mentioned.