Seems to be an established conversation around this point, see: https://en.wikipedia.org/wiki/Ordinal_utility https://en.wikipedia.org/wiki/Cardinal_utility

"The idea of cardinal utility is considered outdated except for specific contexts such as decision making under risk, utilitarian welfare evaluations, and discounted utilities for intertemporal evaluations where it is still applied. Elsewhere, such as in general consumer theory, ordinal utility with its weaker assumptions Is preferred because results that are just as strong can be derived."

Or you could go back to the original Theory of Games proof, which I believe was ordinal- it's going to depend on your axioms. In that document, Von Neumann definitely didn't go so far as to treat utility as simply an integer.