I feel this reply I made captures the link between proof, evidence, and elegance, in both scientific and mathematical fields.

That is to say, where proof is equivalent for two mutually exclusive theories (because sometimes things are proven logically outside mathematics, and not everything in mathematics are proven), evidence is used as a tiebreaker.

And where evidence is equivalent for two mutually exclusive theories (requiring of course that proof also be equivalent), elegance is used as a tiebreaker.