you can't talk about the integers or the reals in first-order logic.

It's more accurate to say that you can't talk about arbitrary subsets of the integers or the reals in first-order logic.

Accepting that the integers have a supremum is a whole different kettle of fish from accepting that the collection of finite integers seems to go on without bound.

I agree. This is the difference between completed and potential infinity. Nelson.

Second, taking a supremum once, using a clearly defined computable notation and a halting machine that can compare any two representations, is a whole different kettle of fish than talking about the supremum of all possible ways to define countable well-orderings to and beyond computable recursion.

I'm not so sure. Everything you've ever talked about, uncountable ordinals and all, you've talked about using computable notation. Computable, period is a whole different kettle of fish.