This post describes how one can use many independent arguments to justifiably develop very high confidence in the truth of a statement. It ends with a case study: Euler’s use of several independent lines of evidence to develop confidence in the validity of his unrigorous solution to the Basel Problem.

Justifiably high confidence

In Einstein’s Arrogance, Eliezer described how Einstein could have been justifiably confident in the correctness of the theory of general relativity, even if it were to fail an empirical test:

In 1919, Sir Arthur Eddington led expeditions to Brazil and to the island of Principe, aiming to observe solar eclipses and thereby test an experimental prediction of Einstein's novel theory of General Relativity. A journalist asked Einstein what he would do if Eddington's observations failed to match his theory. Einstein famously replied: "Then I would feel sorry for the good Lord. The theory is correct."

[…]

If Einstein had enough observational evidence to single out the correct equations of General Relativity in the first place, then he probably had enough evidence to be damn sure that General Relativity was true.

[…]

"Then I would feel sorry for the good Lord; the theory is correct," doesn't sound nearly as appalling when you look at it from that perspective. And remember that General Relativity was correct, from all the vast space of possibilities.

Limits on the confidence conferred by a single argument

In Confidence levels inside and outside an argument Yvain described how one can't develop very high confidence in a statement based on a single good argument that makes the prediction with very high confidence, because there’s a sizable (even if small) chance that the apparently good argument is actually wrong.

How can this be reconciled with Einstein's justifiably high confidence in the truth of general relativity?

The use of many independent lines of evidence to develop high confidence

Something that I’ve always admired about Carl Shulman is that he often presents many different arguments in favor of a position, rather than a single argument. In Maximizing Cost-effectiveness via Critical Inquiry Holden Karnofsky wrote about how one can improve one’s confidence in a true statement by examining the statement from many different angles. Even though one can’t gain very high confidence in a statement via a single argument, one can gain very high confidence in the truth of a statement if many independent lines of evidence support it.

Epistemology in the human world is murky:

• Most statements are inherently ill-defined
• There's a fair amount of ambiguity as to which statements can rationally be assigned truth value with very high confidence.
• There's ambiguity as to what constitutes good evidence. 

To illustrate the principle mentioned above, it's instructive to consider an example in the domain of mathematics, which is a much simpler domain.

Euler’s solution of the Basel Problem 

In 1735, Euler solved the famous problem of finding the value of the “Basel Sum,” which is defined to be the sum of reciprocals of squares of positive integers. He found that the value is equal to ‘pi squared divided by 6.’

Euler's initial method of solution is striking in that it relies on an assumption which was unproven at the time, and which is not true in full generality. Euler assumed the product formula for the sine function, which expresses the sine function as an infinite product of linear polynomials, each corresponding to a real root of the sine function. In hindsight, there are a number of reasons why Euler should have been concerned that the product formula sine might not be valid:

• Its truth hinges on the sine function being differentiable as a function of a complex variable, and on the sine function having no non-real complex roots. A reading of Chapter 5 of Euler: The Master of Us All suggests that Euler was unfamiliar with these facts in 1735 (though he seems to have discovered them later).
  
  
• Not every function of a complex variable can be written as an infinite product in this way: the fact that the sine function in particular can be depends on the fact that the sine function is an odd function, as well as the fact that the sine function has no worse than exponential growth rate in magnitude. It seems unlikely that Euler was aware that these conditions were necessary. 

The general theorem of which the product formula for sine is a special case is the Weierstrass factorization theorem, which wasn’t proved until the mid/late 1800’s.

And indeed, Euler was initially concerned that the product formula for sine might not be valid. So in view of the (at the time) dubious nature of the product formula for sine, how could Euler have known that he had correctly determined the value of the Basel sum?

Euler: The Master of Us All describes some of Euler's reasons:

The analogy with polynomial functions

A polynomial function with all roots real can be written as products of linear polynomials corresponding to the roots of the function. Isaac Newton had shown that the sine function can be written as an infinite polynomial. So one might guess that the sine function can be written as a product of linear polynomials, in analogy with polynomial functions.

This evidence is weak, because not all polynomials have all roots real, and because statements that are true of finite objects often break down when one passes to infinite objects.

Numerical evidence from the Basel sum and its analogs

If a statement implies a nontrivial true statement, that’s evidence that the original statement is true. (This is just Bayes’ Theorem, and the converse of the fact that Absence of Evidence Is Evidence of Absence.)

In 1731, Euler found that the Basel Sum is equal to 1.644934, up to six places past the decimal point. This decimal approximation agrees with that of ‘pi squared over 6.’ So the assumption that the product formula for sine is true implies something that’s both true and nontrivial.

The above numerical confirmation leaves open the possibility that the Basel Sum and ‘pi squared over 6’ differ by a very small amount. It can happen that two apparently unrelated and simple mathematical quantities differ by less than a trillionth, so this is a legitimate concern.  

To assuage this concern, one can consider the fact that Euler’s method yields the values of the sum of reciprocals of k^th powers of positive integers for every even integer k, and check these formulas for numerical accuracy, finding that they hold with high precision. It’s less likely that all of them are just barely wrong than it is that a single one is just barely wrong. So this a nontrivial amount of further evidence that the product formula for sine is true.

An independent derivation of the Wallis product formula and of the Leibniz series

As stated above, if a statement implies a nontrivial true statement, this is evidence that the original statement is true. Euler used the product formula for sine to deduce the Wallis product formula for pi, which had been known since 1655. By assuming the existence of a formula analogous to the product formula for sine, this time for '1 minus sine,' Euler deduced the Leibniz formula for pi. 

Upon doing so, Euler wrote "For our method, which may appear to some as not reliable enough, a great confirmation comes here to light. Therefore, we should not doubt at all of the other things which are derived by the same method."

Remark: After writing this post, I learned that George Polya gave an overlapping discussion of Euler's work on the Basel Problem on pages 17-21 of Mathematics and Plausible Reasoning, Volume 1: Induction and Analogy in Mathematics. I found Euler's deduction of the Leibniz formula for pi in Polya's book. Polya's book contains more case studies of the same type.

Conclusion

Euler had good reason to believe that his derivation of the value of the Basel sum was valid, even though a rigorous proof that his derivation was valid was years or decades away. 

How much confidence is rational based on the evidence available at the time depends on the degree to which the different lines of evidence are independent. The lines of evidence appear to be independent, but could have subtle interdependencies.

Nevertheless, I've never heard of an example of mathematical statement so robustly supported that turned out to be false. In the context of the fact that there's been a huge amount of mathematical research since then, it may be reasonable to conclude that the appropriate confidence level would have been 99.9999+%. [Edit: I lightly edited this paragraph — see this comment thread.]

It may be appropriate to hedge, with Confidence levels inside and outside an argument in mind, because the arguments that I make in this post may themselves be wrong :-).

The example of Euler's work on the Basel Problem highlights the use of many independent lines of evidence to develop very high confidence in a statement: something which occurs and which can occur in many domains.

Note: I formerly worked as a research analyst at GiveWell. All views here are my own.