If, whenever we took 2 bananas and stuck them together with 2 more bananas, we ended up with 3 bananas, 2+2=4 would still be 'true' in the abstract sense that it proceeds naturally from the axioms[.]

I'm not so sure of that. If putting 2 S's next to 2 S's got us 3 S's, we could prove 2+2=3 in PA with the usual definition of addition:

(dfn) \a. 0 + a = a
(dfn) \ab. Sb + a = b + Sa
\a. SS0 + a = S0 + Sa = 0 + SSa = SSa
SS0 + SS0 = SSS0

Depending on the universe's other rules for putting n things next to m things, we might also be able to derive "2+2=4". In this case, we would decide that PA is inconsistent! Whatever the other rules are, this already shows that the "abstract" conclusions we can draw from a set of axioms depend on the way symbol manipulation works in our world.

I don't think this is really a problem for your argument, but it's an interesting complication. Many (most?) physical facts seem to have no influence on the symbolic manipulations we can use to derive them. For instance, symbolically computing a series for pi doesn't seem to involve any actual circles the way shuffling symbols to add 2 and 2 in PA involves putting SS next to SS.