Even if arithmetic fails to define a framework where they're separate, that would simply mean that we need to redefine stuff, figure out a better way to describe our intuitive way of understanding how they're different, and thus changing the framework towards that intuitive understanding.

Man that would be hard, though!

I imagine that if this happened, math would become, at least for a while, an empirical science. People would study the derivations that led to contradictions. Proscriptions of such derivations would be taken as provisional, "this-is-how-our-universe-seems-to-work" axioms.

(Of course I don't actually expect arithmetic to ever be shown to be inconsistent.)