My guess, completely unsubstantiated by any knowledge of neuroscience, is that when mathematicians do math, they are creating and manipulating physical models in their brains.

I strongly agree! Assuming physics, "belief in math" is equivalent to the belief that these models behave very consistently in my and others' brains, and reflect other regions of physical reality effectively. But even without that, whatever this floaty thing in my mind I call math is, my "belief in math" is one that I hold as convictingly and implicitly as anything else I'm aware of.