Some general remarks on incomparability: Human preferences may be best modelled not by a total ordering but by a preorder, which means that when you compare A and B there are four different possible outcomes:
and these last two are not at all the same thing. In particular, if A and B are incomparable it doesn't follow that anything better than A is also better than B.
(I don't think this is saying anything you haven't already figured out, but you may be glad to know that such ideas are already out there.)
John Conway's beautiful theory of combinatorial games puts a preorder relation on games, which I've often thought might be a useful model for human preferences. (It's related to his theory of "surreal numbers", which might also be useful; e.g., its infinities and infinitesimals provide a way, should we want one, of representing situations where one thing is literally infinitely more important than another, even though the other still matters a nonzero amount.) Here, it turns out that the right notion of equality is "if you combine game A with a role-reversed version of game B in a certain way, the resulting game is always won by whoever moves second" whereas incomparability is "... the resulting game is always won by whoever moves first"; if two games are equal then giving either player in the combined game a tiny advantage makes them win, but if two games are incomparable then there may be a substantial range of handicaps you can give either player while still leaving the resulting game a first-player win and hence leaving the two games incomparable.
On the specific question of multiple utility functions: It's by no means the only way to get this sort of incomparability, but I agree that something of the kind is probably going on in human brains: two subsystems reporting preferences that pull in different directions, and no stable and well defined way to adjudicate between them.
[EDITED to add:] But I agree with ChaosMote that actually "utility functions" may not be the best name for these things, not only for ChaosMote's reason that our behaviour may not be well modelled as maximizing anything but also because maybe it's best to reserve the term "utility function" for something that attempts to describe overall preferences.
I think a useful meaning of "incomparable" is "you should think a very long time before deciding between these". In situations like these, the right decision is not to immediately decide between them, but to think a lot about the decision and related issues. Sure, if someone has to make a split-second decision, they will probably choose whichever sounds better to them. But if given a long time, they might think about it a lot and still not be sure which is better.
This seems a bit similar to multiple utility functions in that if you have multiple utility functions then you might have to think a lot and resolve lots of deep philosophical issues to really determine how you should weight these functions. But even people who are only using one utility function can have lots of uncertainty about which outcome is better, and this uncertainty might slowly reduce (or be found to be intractable) if they think about the issue more. I think the outcomes would seem similarly incomparable to this person.
I think a useful meaning of "incomparable" is "you should think a very long time before deciding between these"
Indeed, sometimes whether or not two options are incomparable depends on how much computational power your brain is ready to spend calculating and comparing the differences. Things that are incomparable might become comparable if you think about them more. However, when one is faced with the need to decide between the two options, one has to use heuristics. For example, in his book "Predictably irrational" Dan Ariely writes:
But there's one aspect of relativity that consistently trips us up. It's this: we not only tend to compare things with one another but also tend to focus on comparing things that are easily comparable—and avoid comparing things that cannot be compared easily. That may be a confusing thought, so let me give you an example. Suppose you're shopping for a house in a new town. Your real estate agent guides you to three houses, all of which interest you. One of them is a contemporary, and two are colonials. All three cost about the same; they are all equally desirable; and the only difference is that one of the colonials (the "decoy") needs a new roof and the owner has knocked a few thousand dollars off the price to cover the additional expense.
So which one will you choose?
The chances are good that you will not choose the contemporary and you will not choose the colonial that needs the new roof, but you will choose the other colonial. Why? Here's the rationale (which is actually quite irrational). We like to make decisions based on comparisons. In the case of the three houses, we don't know much about the contemporary (we don't have another house to compare it with), so that house goes on the sidelines. But we do know that one of the colonials is better than the other one. That is, the colonial with the good roof is better than the one with the bad roof. Therefore, we will reason that it is better overall and go for the colonial with the good roof, spurning the contemporary and the colonial that needs a new roof.
[...]
Here's another example of the decoy effect. Suppose you are planning a honeymoon in Europe. You've already decided to go to one of the major romantic cities and have narrowed your choices to Rome and Paris, your two favorites. The travel agent presents you with the vacation packages for each city, which includes airfare, hotel accommodations, sightseeing tours, and a free breakfast every morning. Which would you select?
For most people, the decision between a week in Rome and a week in Paris is not effortless. Rome has the Coliseum; Paris, the Louvre. Both have a romantic ambience, fabulous food, and fashionable shopping. It's not an easy call. But suppose you were offered a third option: Rome without the free breakfast, called -Rome or the decoy.
If you were to consider these three options (Paris, Rome, -Rome), you would immediately recognize that whereas Rome with the free breakfast is about as appealing as Paris with the free breakfast, the inferior option, which is Rome without the free breakfast, is a step down. The comparison between the clearly inferior option (-Rome) makes Rome with the free breakfast seem even better. In fact, -Rome makes Rome with the free breakfast look so good that you judge it to be even better than the diffkult-to-compare option, Paris with the free breakfast.
So, it seems that one possible heuristic is to try to match your options against yet more alternatives and the option that wins more (and loses less) matches is "declared a winner". As you can see, the result that is obtained using this particular heuristic depends on what kind of alternatives the initial options are compared against. Therefore this heuristic is probably not good enough to reveal which option is "truly better" unless, perhaps, the choice of alternatives is somehow "balanced" (in some sense, I am not sure how to define it exactly).
It seems to me, that in many case if one employs more and more (and better) heuristics one can (maybe after quite a lot of time spent deliberating the choice) approach finding out which option is "truly better". However, the edge case is also interesting. As you can see, the decision is not made instantly, it might take a lot of time. What if your preferences are less stable in a given period of time than your computational power allows you to calculate during that period of time? Can two options be said to be equal if your own brain does not have enough computational power to consistently distinguish between them seemingly even in principle, even if more powerful brain could make such decision (given the same level of instability of preferences)? What about creatures that have very little computational power? Furthermore, aren't preferences themselves usually defined in terms of decision making? At the moment I am a bit confused about this.
I thought of the idea that maybe the human decision maker has multiple utility functions that when you try to combine them into one function some parts of the original functions don't necessarily translate well... sounds like the "shards of desire" are actually a bunch of different utility functions.
This is an interesting perspective, and I would agree that we humans typically have multiple decision criteria which often can't be combined well. However, I don't think it is quite right to call then utility functions. Humans are adaptation-executers, not fitness-maximizers - so it is more like we each have a bunch of behavioral patterns that we apply when appropriate, and potential conflicts arise when multiple patterns apply.
It might useful to look at Pareto dominance and related ideas, and the way they are used to define concrete algorithms for multi-objective optimisation, eg NSGA2 which is probably the most used.
I thought, if outcome A is not more preferred than outcome B, and outcome B is not more preferred than outcome A, then of course the decision maker is indifferent between both outcomes, right? But if that's the case, the decision maker should be able to just flip a coin to decide. Not only that, but adding even a tiny amount of extra value to one of the outcomes should always make that outcome be preferred. So why can't a human decision maker just make up their mind about their preferences between "incomparable" outcomes until they're forced to choose between them? Also, if a human decision maker is really indifferent between both outcomes, then they should be able to know that ahead of time and have a plan for deciding, such as flipping a coin. And, if they're really indifferent between both outcomes, then they should not be regretting and/or doubting their decision before an outcome even occurs regardless of which act they choose. Right?
There are some very interesting ideas in the rest of this topic; I'm going to come at it from a bit of a different angle.
I think a lot of the indecision/doubt/regret comes from uncertainty. Decision problems usually have the options and payoffs mapped explicitly, but for real decisions we often have to estimate, without any rigorous way to do so. And our estimates are not just inaccurate, but also imprecise: it can be really hard to tell the difference between perfect indifference, slightly favoring A, and slightly favoring B.
On this view, the "waffling between incomparable outcomes" can be interpreted as "attempting to resolve severe uncertainty" - trying to get the EV calculation nailed down, before you can shut up and multiply.
Suggested reading: Holdouts Against Universal Comparability and Lattice Theories of Probability in an appendix to Probability Theory: The Logic Of Science
Some people (including me) have made comments along these lines before. There's nothing theoretically wrong with the view that evolutionary history may have created multiple less-than-coordnated utility functions that happen to share one brain.
The consequences have some serious implications, though. If a single human has multiple utility functions, it is highly unlikely (for reasons similar to Arrow's Paradox) that these work out compatibly enough that you can have an organism-wide utility expressed as a real number (as opposed to a hypercomplex number or matrix). And if you have to map utility to a hypercomplex number or matrix, you can't "shut up and multiply", because while 7*3^^^3 is always a really big number, matrix math is a lot more complicated. Utilitarianism becomes mathematically intractable as a result.
I thought of the idea that maybe the human decision maker has multiple utility functions that when you try to combine them into one function some parts of the original functions don't necessarily translate well... sounds like the "shards of desire" are actually a bunch of different utility functions.
I hereby request a research-filled thread of what to do when you feel like you're in this situation, which I believe has been called "welfare economics" in the literature.
I know that this idea might sound a little weird at first, so just hear me out please?
A couple weeks ago I was pondering decision problems where a human decision maker has to choose between two acts that lead to two "incomparable" outcomes. I thought, if outcome A is not more preferred than outcome B, and outcome B is not more preferred than outcome A, then of course the decision maker is indifferent between both outcomes, right? But if that's the case, the decision maker should be able to just flip a coin to decide. Not only that, but adding even a tiny amount of extra value to one of the outcomes should always make that outcome be preferred. So why can't a human decision maker just make up their mind about their preferences between "incomparable" outcomes until they're forced to choose between them? Also, if a human decision maker is really indifferent between both outcomes, then they should be able to know that ahead of time and have a plan for deciding, such as flipping a coin. And, if they're really indifferent between both outcomes, then they should not be regretting and/or doubting their decision before an outcome even occurs regardless of which act they choose. Right?
I thought of the idea that maybe the human decision maker has multiple utility functions that when you try to combine them into one function some parts of the original functions don't necessarily translate well. Like some sort of discontinuity that corresponds to "incomparable" outcomes, or something. Granted, it's been a while since I've taken Calculus, so I'm not really sure how that would look on a graph.
I had read Yudkowsky's "Thou Art Godshatter" a couple months ago, and there was a point where it said "one pure utility function splintered into a thousand shards of desire". That sounds like the "shards of desire" are actually a bunch of different utility functions.
I'd like to know what others think of this idea. Strengths? Weaknesses? Implications?