Hmm... I don't have a good answer. My intuition is that integers are "simple enough" and, in particular, sufficiently unambiguous, to be part of the territory, but complex numbers are not. However even a tiny bit of reflection shows that my idea of "simple enough" is arbitrary.

I guess we've fallen into the "is mathematics real?" tar pit. Probably shouldn't thrash around too much :-)