"But my dear sir, if the fact of 2 + 3 = 5 exists somewhere outside your brain... then where is it?"

A mathematical truth can be formalized as output of a proof checking algorithm, and output of an algorithm can be verified to an arbitrary level of certainty (by running it again and again, on redundant substrate). When you say that something is mathematically true, it can be considered an estimation of counterfactual that includes building of such a machine.