You're right. PA is still consistent (i.e. has a model) even if

 N = the set of strings of the form S*0
0 = the string "0"
S = the function that prepends "S" to its argument

fails to be one because of the way string concatenation works. There's nothing mathematically special about theories that can use physical objects as a model.

(Minor quibble: the definition of addition isn't an axiom. It's just a relation definable in the first-order theory of arithmetic.)