They say that the axioms pin down the numbers, regardless of how physical objects behave or start behaving.

Not sure who "they" are. May I suggest reading up on model theory? Maybe Enderton's "mathematical logic" (?).

It's important to think separately about the "model" (the thing we are studying), the "language" (where we make statements about the model), and the "theory" (a set of statements in a language). The same theory in a particular language may and in fact generally does apply to multiple distinct models. The same model (object) may be described by theories in different languages of different strengths.

So in one sense Peano axioms "pin down" the natural numbers, but in another sense they don't because we can invent crazy objects that contain a lot more than just the natural numbers to which Peano axioms also apply (this is the content of the Lowenheim-Skolem theorem).

http://en.wikipedia.org/wiki/L%C3%B6wenheim%E2%80%93Skolem_theorem

We can use a more powerful language than first order logic to describe the natural numbers, and that would rule out some of the crazy models. That will capture more properties of the natural number line, but not everything.

Physicists play a similar game to logicians, except perhaps a bit less formally. But their models (in the sense of 'object of study') "bite back."

It's confusing that to a model theorist "the model" refers to the territory, while to a statistician "the model" refers to the map.