I'm a little confused as to which of two positions this is advocating:

1. Numbers are real, serious things, but the way that we pick them out is by having a categorical set of axioms. They're interesting to talk about because lots of things in the world behave like them (to some degree).

2. Mathematical talk is actually talk about what follows from certain axioms. This is interesting to talk about because lots of things obey the axioms and so exhibit the theorems (to some degree).

I read it as (1), with a side order of (2). Mathematical talk is also about what follows from certain axioms. The axioms nail it down so that mathematicians can be sure what other mathematicians are talking about.

Both of these have some problems. The first one requires you to have weird, non-physical numbery-things.

Not weird, non-physical numbery-things, just non-physical numbery-things. If they seem weird, maybe it's because we only noticed them a few thousand years ago.

Not only this, but they're a special exception to the theory of reference that's been developed so far, in that you can refer to them without having a causal connection.

No more than a magnetic field is a special exception to the theory of elasticity. It's just a phenomenon that is not described by that theory.