There are mathematical structures that allow for comparison but not arithmetic. For instance, a total order on a set is a relation < such that (1) for any x,y we have exactly one of x<y, x=y, y<x and (2) for any x,y,z if x<y<z then x<z. (We say it's trichotomous and transitive.)
The usual ordering on (say) the real numbers is a total order (and it's "compatible with arithmetic" in a useful sense), but there are totally ordered sets that don't look much like any system of numbers.
There are weaker notions of ordering (e.g., a partial order is the same except that condition 1 says "at most one" instead of "exactly one", and allows for some things to be incomparable with others; a preorder allows things to compare equal without being the same object) and stronger ones (e.g., a well-order is a total order with the sometimes-useful property that there's no infinite "descending chain" a1 > a2 > a3 > ... -- this is the property you need for mathematical induction to work).
The ordinals, which some people have mentioned, are a generalization of the non-negative integers, and comparison of ordinals is a well-order, but arithmetic (albeit slightly strange arithmetic) is possible on the ordinals and if you're thinking about preferences then the ordinals aren't likely to be the sort of structure you want.
I've talked earlier about integral and differential ethics, in the context of population ethics. The idea is that the argument for the repugnant conclusion (and its associate, the very repugnant conclusion) is dependent on a series of trillions of steps, each of which are intuitively acceptable (adding happy people, making happiness more equal), but reaching a conclusion that is intuitively bad - namely, that we can improve the world by creating trillions of people in torturous and unremitting agony, as long as balance it out by creating enough happy people as well.
Differential reasoning accepts each step, and concludes that the repugnant conclusions are actually acceptable, because each step is sound. Integral reasoning accepts that the repugnant conclusion is repugnant, and concludes that some step along the way must therefore be rejected.
Notice that key word, "therefore". Some intermediate step is rejected, but not for intrinsic reasons, but purely because of the consequence. There is nothing special about the step that is rejected, it's just a relatively arbitrary barrier to stop the process (compare with the paradox of the heap).
Indeed, things can go awry when people attempt to fix the repugnant conclusion (a conclusion they rejected through integral reasoning) using differential methods. Things like the "person-affecting view" have their own ridiculousness and paradoxes (it's ok to bring a baby into the world if it will have a miserable life; we don't need to care about future generations if we randomise conceptions, etc...) and I would posit that it's because they are trying to fix global/integral issues using local/differential tools.
The relevance of this? It seems that integral tools might be better suited to deal with the bad convergence of AI problem. We could set up plausibly intuitive differential criteria (such as self-consistency), but institute overriding integral criteria that can override these if they go too far. I think there may be some interesting ideas in that area, potentially. The cost is that integral ideas are generally seen as less elegant, or harder to justify.