There are some points of view that sometimes do require mathematical statements to be dependent on reality (i.e. constructivism, actual versus potential infinity debate, etc).

Sometimes it is intuitive to require mathematics to behave this way, i.e. 'natural' numbers are called that for a reason, and they better behave like the apples or I'm postulating a change in nomenclature.

P.S. Ii seems to me the OP's wording wasn't precise enough. I can very well imagine a situation in which some basic addition would yield non obvious results (like addition inside modulo N number space).