I think when trying to study information and its relation to mathematical truth, we must start off practical and should be talking about provability in formal systems of logic. I don't actually know of any rigorous connections between the two notions, but I can think of an argument that "mathematical truths contain zero information" might be false based on indirect connections between existing work on proof theory and information theory. But I don't want to give that interpretation yet because I would like to first ask Skatche if he wanted to elaborate on his statement a little better or point to some references for us to read.

The philosophical question is interesting too, and I would agree that a set theory without the axiom of infinity seems pretty adequate for describing our experiences of reality. I'm not sure if its the most harmonious, however...