I can do better. I can give you a complete, decidable, axiomatized system that does that: first order real arithmetic. However, in this system you can't talk about integers in any useful way.

We can do better than that: first order real arithmetic + PA + a set of axioms embedding the PA integers into R in the obvious way. This is a second order system where I can't talk about uncountable ordinals. However, this system doesn't let us talk about sets.

Note that in both these cases we've done this by minimizing how much we can talk about sets. Is there some easy way to do this where we can talk about set a reasonable amount?

I'm not sure. Answering that may be difficult (I don't think the question is necessarily well-defined.) However, I suspect that the following meets one's intuition as an affirmative answer: Take ZFC without regularity, replacement or infinity, choice, power set or foundation. Then add as an axiom that there exists a set R that has the structure of a totally ordered field with the least-upper bound property.

This structure allows me to talk about most things I want to do with the reals while probably not being able to prove nice claims about Hartogs numbers which should make proving the existence of uncountable ordinals tough. It would not surprise me too much if one could get away with this system with the axiom of the power set thrown also. But it also wouldn't surprise me either if one can find sneaky ways to get info about ordinals.

Note that none of these systems are at all natural in any intuitive sense. With the exception of first-order reals they are clear attempts to deliberately lobotomize systems. (ETA: Even first order reals is a system which we care about more for logic and model theoretic considerations than any concrete natural appreciation of the system.) Without having your goal in advance or some similar goal I don't think anyone would ever think about these systems unless they were a near immortal who was passing the time by examining lots of different axiomatic systems.

While thinking about this I realized that I don't know an even more basic question: Can one deal with what Eliezer wants by taking out the axiom schema of replacement, choice, and foundation? The answer to this is not obvious to me, and in some sense this is a more natural system. If this is the case then one would have a robust system in which most of modern mathematics could be done but you wouldn't have your solution. However, I suspect that this system is enough to prove the existence of the least uncountable ordinal.