Goldbach's conjecture is ...
Fixed. Thanks.
P(X) = (P(will be proven(X)) + P(is true but unprovable(X))) / (P(will be proven(X)) + P(will be disproven(X)) + P(is true but unprovable(X)))
It still only works on assumptions that might generally make good approximations, but aren't necessarily true.
More to the point, Phil was suggesting that something is false on the basis that it would be difficult to prove if true, but easy to prove if false. In other words, he was using it specifically because that example was one where the implicit assumption in his approximation, that the relative values of P(will be proven(X)) to P(will be disproven(X)) are about the same as P(is true but will not be proven(X)) to P(is false but will not be disproven(X)), was particularly far from the truth.
Is there a reason to suspect that a counterexample wouldn't be a very large number that hasn't been considered yet?
Suppose they'll check every number up to n. Suppose also that we're using a logarithmic prior as to where the first counterexample is. Suppose also we'll eventually find it. It has the same probability of being from one to sqrt(n) as from sqrt(n) to n. This means that if m, the number of numbers we've already checked, is more than sqrt(n), we've probably already found it. Since m is pretty big, we're probably not going to manage to check as many numbers as we've already checked m times over.
Looking into it more, they've been using more clever ways to prove that it's true up to certain numbers, and we know it's true for up to about 4*10^18. It would be impossible to even check a number that size without some kind of clever method, let alone check enough to find the counter-example.
Consider some disputed proposition X. Suppose there appeared to be a limited number of ways of proving and of disproving X. No one has yet constructed a proof or disproof, but you have a feeling for how likely it is that someone will.
For instance, take Fermat's Last Theorem or the 4-color problem. For each of them, at one point in time, there was no proof, but people had some sense of the prior probability of observing the lack of a counterexample given the space searched so far. They could use that to assign a probability of there being a counterexample (and hence a disproof) [1]. Later, there was an alleged proof, and people could estimate the probability that the proof was correct based on the reputation of the prover and the approach used. At that point, people could assign values to both P(will_be_proven(X)) and P(will_be_disproven(X)).
Is it reasonable to assign P(X) = P(will_be_proven(X)) / (P(will_be_proven(X)) + P(will_be_disproven(X))) ?
If so, consider X = "free will exists". One could argue that the term "free will" is defined such that it is impossible to detect it, or to prove that it exists. But if one could prove that the many worlds interpretation of quantum mechanics is correct, that would constitute a disproof of X. Then P(will_be_proven(X)) / (P(will_be_proven(X)) + P(will_be_disproven(X))) = 0.
Is it possible for this to happen when you know that X is not undecidable? If so, what do you do then?
1. The computation is not as simple as it might appear, because you need to adjust for the selection effect of mathematicians being interested only in conjectures with no counterexamples.