Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Post Your Utility Function

28 Post author: taw 04 June 2009 05:05AM

A lot of rationalist thinking about ethics and economy assumes we have very well defined utility functions - knowing exactly our preferences between states and events, not only being able to compare them (I prefer X to Y), but assigning precise numbers to every combinations of them (p% chance of X equals q% chance of Y). Because everyone wants more money, you should theoretically even be able to assign exact numerical values to positive outcomes in your life.

I did a small experiment of making a list of things I wanted, and giving them point value. I must say this experiment ended up in a failure - thinking "If I had X, would I take Y instead", and "If I had Y, would I take X instead" very often resulted in a pair of "No"s. Even thinking about multiple Xs/Ys for one Y/X usually led me to deciding they're really incomparable. Outcomes related to similar subject were relatively comparable, those in different areas in life were usually not.

I finally decided on some vague numbers and evaluated the results two months later. My success on some fields was really big, on other fields not at all, and the only thing that was clear was that numbers I assigned were completely wrong.

This leads me to two possible conclusions:

  • I don't know how to draw utility functions, but they are a good model of my preferences, and I could learn how to do it.
  • Utility functions are really bad match for human preferences, and one of the major premises we accept is wrong.

Anybody else tried assigning numeric values to different outcomes outside very narrow subject matter? Have you succeeded and want to share some pointers? Or failed and want to share some thought on that?

I understand that details of many utility functions will be highly personal, but if you can share your successful ones, that would be great.

Comments (273)

Comment author: Eliezer_Yudkowsky 04 June 2009 05:39:51PM 11 points [-]

Utility functions are really bad match for human preferences, and one of the major premises we accept is wrong.

They may be a bad descriptive match. But in prescriptive terms, how do you "help" someone without a utility function?

Comment author: Wei_Dai 05 June 2009 11:16:21AM 7 points [-]

To help someone, you don't need him to have an utility function, just preferences. Those preferences do have to have some internal consistency. But the consistency criteria you need to in order to help someone seem strictly weaker than the ones needed to establish an utility function. Among the von Neumann-Morgenstern axioms, maybe only completeness and transitivity are needed.

For example, suppose I know someone who currently faces choices A and B, and I know that if I also offer him choice C, his preferences will remain complete and transitive. Then I'd be helping him, or at least not hurting him, if I offered him choice C, without knowing anything else about his beliefs or values.

Or did you have some other notion of "help" in mind?

Comment author: MichaelBishop 04 June 2009 06:27:57PM *  2 points [-]

Furthermore, utility functions actually aren't too bad as a descriptive match when you are primarily concerned about aggregate outcomes. They may be almost useless when you try to write one that describes your own choices and preferences perfectly, but they are a good enough approximation that they are useful for understanding how the choices of individuals aggregate: see the discipline of economics. This is a good place for the George Box quote: "All models are wrong, but some are useful."

Comment author: brian_jaress 04 June 2009 06:22:06PM 0 points [-]

They may be a bad descriptive match. But in prescriptive terms, how do you "help" someone without a utility function?

Isn't "helping" a situation where the prescription is derived from the description? Are you suggesting we lie about others' desires so we can more easily claim to help satisfy them?

Helping others can be very tricky. I like to wait until someone has picked a specific, short term goal. Then I decide whether to help them with that goal, and how much.

Comment author: conchis 04 June 2009 07:07:56PM 0 points [-]

Isn't "helping" a situation where the prescription is derived from the description?

Not necessarily. There are lots of plausible moral theories under which individuals' desires don't determine their well-being.

Comment author: MichaelBishop 04 June 2009 06:37:39PM 0 points [-]

I think Eliezer is simply saying: "I can't do everything, therefore I must decide where I think the marginal benefits are greatest. This is equivalent to attempting to maximize some utility function."

Comment author: Vladimir_Nesov 04 June 2009 06:23:56PM *  0 points [-]

Derivation of prescription from description isn't trivial.

That's the difference between finding the best plan, and conceding for a suboptimal plan because you ran out of thought.

Comment author: brian_jaress 04 June 2009 06:50:51PM 0 points [-]

I agree with both those statements, but I'm not completely sure how you're relating them to what I wrote.

Do you mean that the difficulty of going from a full description to a prescription justifies using this particular simpler description instead?

It might. I doubt it because utility functions seem so different in spirit from the reality, but it might. Just remember it's not the only choice.

Comment author: Vladimir_Nesov 04 June 2009 07:00:31PM *  0 points [-]

A simple utility function can be descriptive in simple economic models, but taken as descriptive, such function doesn't form a valid foundation for the (accurate) prescriptive model.

On the other hand, when you start from an accurate description of human behavior, it's not easy to extract from it a prescriptive model that could be used as a criterion for improvement, but utility function (plus prior) seems to be a reasonable format for such a prescriptive model if you manage to construct it somehow.

Comment author: brian_jaress 04 June 2009 07:25:50PM 0 points [-]

In that case, we disagree about whether the format seems reasonable (for this purpose).

Comment deleted 04 June 2009 11:25:03AM *  [-]
Comment author: Cyan 05 June 2009 01:43:15PM 1 point [-]

Where X is some set. What should that set be? Certainly it shouldn't be the set of states of the universe, because then you can't say that you enjoy certain processes (such as bringing up a child, as opposed to the child just appearing). Perhaps the set of possible histories of the universe is a better candidate.

Part of the problem is the X is necessarily based in the map, not the territory. There will always be the chance that one will learn something that radically changes the map, so it seems like an explicit statement of f will have to involve all possible maps that one might have.

Comment author: conchis 05 June 2009 02:57:28PM 0 points [-]

Part of the problem is the X is necessarily based in the map, not the territory.

Necessarily? If I place value on e.g. "my friends actually liking and respecting me", rather than just "the subjective sense that my friends like and respect me" then my utility function seems to be responding directly to the territory rather than the map. (It also means that I won't ever really know my utility, but that's true of lots of things.) Some people argue that things can't affect one's well-being unless one somehow experiences them, but that's a contentious position.

Am I missing your intended meaning?

Comment author: pjeby 05 June 2009 06:24:10PM -2 points [-]

If I place value on e.g. "my friends actually liking and respecting me", rather than just "the subjective sense that my friends like and respect me" then my utility function seems to be responding directly to the territory rather than the map.

In practice, your definition of what "liking and respecting me" means -- i.e., what evidence you expect to see in the world of that -- is part of the map, not the territory.

Suppose, for example, that your friends really and truly like and respect you... but they have to beat you up and call you names, for some other reason. Does that match what you actually value? It's out there in the territory, after all.

That is, is merely knowing that they "like and respect you" enough? Or is that phrase really just a shorthand in your map for a set of behaviors and non-behaviors that you actually value?

Note that if you argue that, "if they really liked and respected me, they wouldn't do that", then you are now back to talking about your map of what that phrase means, as opposed to what someone else's map is.

System 2 thinking is very tricky this way -- it's prone to manipulating symbols as if they were the things they're merely pointing at, as though the map were the territory... when the only things that exist in its perceptual sphere are the labels on the map.

Most of the time, when we think we're talking about the territory, we're talking about the shapes on the map, but words aren't even the shapes on the map!

Comment author: orthonormal 09 June 2009 05:54:52AM *  7 points [-]

We have only access to our current map to tell us about the territory, yes. But we have strong intuitions about how we would act if we could explicitly choose that our future map permanently diverge from our current map (which we currently see as the territory). If we (again by our current map) believe that this divergence would conform less to the territory (as opposed to a new map created by learning information), many of us would oppose that change even against pretty high stakes.

I mean, if Omega told me that I had to choose between

  • (A) my sister on Mars being well but cut off from all contact with me, or
  • (B) my sister being killed but a nonsentient chatbot impersonating her to me in happy weekly chats,

and that in either case my memory of this choice would be wiped when I made it, I would choose (A) without hesitation.

I understand that calling our current map "the territory" looks like a categorical error, but rejecting conchis' point entirely is the wrong response. There's a very real and valid sense in which our minds oppose what they calculate (by the current map) to be divergences between the future map and the territory.

Comment author: pjeby 10 June 2009 03:14:40AM *  1 point [-]

I would choose (A) without hesitation.

Of course, because the immediate pain of the thought of choosing B would outweigh the longer-term lesser pain of the thought of losing contact with your sister.

This has nothing to do with whether the events actually occur, and everything to do with your mapping of the experience of the conditions, as you imagine them for purposes of making a decision.

That is, the model you make of the future may refer to a hypothetical reality, but the thing you actually evaluate is not that reality, but your own reaction to that reality -- your present-tense experience in response to a constructed fiction made of previous experiences

It so happens that there is some correspondence between this (real) process and the way we would prefer to think we establish and evaluate our preferences. Specifically, both models will generate similar results, most of the time. It's just that the reasons we end up with for the responses are quite different.

There's a very real and valid sense in which our minds oppose what they calculate (by the current map) to be divergences between the future map and the territory.

But calling that latter concept "territory" is still a category error, because what you are using to evaluate it is still your perception of how you would experience the change.

We do not have preferences that are not about experience or our emotional labeling thereof; to the extent that we have "rational" preferences it is because they will ultimately lead to some desired emotion or sensation.

However, our brains are constructed in such a way so as to allow us to plausibly overlook and deny this fact, so that we can be honestly "sincere" in our altruism... specifically by claiming that our responses are "really" about things outside ourselves.

For example, your choice of "A" allows you to self-signal altruism, even if your sister would actually prefer death to being imprisoned on Mars for the rest of her life! Your choice isn't about making her life better, it's about you feeling better for the brief moment that you're aware you did something.

(That is, if you cared about something closer to the reality of what happens to your sister, rather than your experience of it, you'd have hesitated in that choice long enough to ask Omega whether she would prefer death to being imprisoned on Mars.)

Comment author: Cyan 10 June 2009 03:49:29AM *  2 points [-]

That is, the model you make of the future may refer to a hypothetical reality, but the thing you actually evaluate is not that reality, but your own reaction to that reality -- your present-tense experience in response to a constructed fiction made of previous experiences.

I affirm this, but it does not follow that:

This has nothing to do with whether the events actually occur...

Just because the events that occur are not the proximate cause of an experience or preference does not mean that these things have nothing to do with external reality. This whole line of argument ignores the fact that our experience of life is entangled with the territory, albeit as mediated by our maps.

Comment author: pjeby 10 June 2009 05:25:17PM 2 points [-]

Just because the events that occur are not the proximate cause of an experience or preference does not mean that these things have nothing to do with external reality. This whole line of argument ignores the fact that our experience of life is entangled with the territory, albeit as mediated by our maps.

And a thermostat's map is also "entangled" with the territory, but as loqi pointed out, what it really prefers is only that its input sensor match its temperature setting!

I am not saying there are no isomorphisms between the shape of our preferences and the shape of reality, I am saying that assuming this isomorphism means the preferences are therefore "about" the territory is mind projection.

If you look at a thermostat, you can project that it was made by an optimizing process that "wanted" it to do certain things by responding to the territory, and that thus, the thermostat's map is "about" the territory. And in the same way, you can look at a human and project that it was made by an optimizing process (evolution) that "wanted" it to do certain thing by responding to the territory.

However, the "aboutness" of the thermostat does not reside in the thermostat; it resides in the maker of the thermostat, if it can be said to exist at all! (In fact, this "aboutness" cannot exist, because it is not a material entity; it's a mental entity - the idea of aboutness.)

So despite the existence of inputs and outputs, both the human and the thermostat do their "preference" calculations inside the closed box of their respective models of the world.

It just so happens that humans' model of the world also includes a Mind Projection device, that causes humans to see intention and purpose everywhere they look. And when they look through this lens at themselves, they imagine that their preferences are about the territory... which then keeps them from noticing various kinds of erroneous reasoning and subgoal stomps.

For that matter, it keeps them from noticing things like the idea that if you practice being a pessimist, nothing good can last for you, because you've trained yourself to find bad things about anything. (And vice versa for optimists.)

Ostensibly, optimism and pessimism are "about" the outside world, but in fact, they're simply mechanical, homeostatic processes very much like a thermostat.

I am not a solipsist nor do I believe people "create your own reality", with respect to the actual territory. What I'm saying is that people are deluded about the degree of isomorphism between their preferences and reality, because they confuse the map with the territory. And even with maximal isomorphism between preference and reality, they are still living in the closed box of their model.

It is reasonable to assume that existence actually exists, but all we can actually reason about is our experience of it, "inside the box".

Comment author: saturn 10 June 2009 04:21:49AM 1 point [-]

That is, if you cared about something closer to the reality of what happens to your sister, rather than your experience of it, you'd have hesitated in that choice long enough to ask Omega whether she would prefer death to being imprisoned on Mars.

And what if he did ask?

Comment author: pjeby 10 June 2009 05:02:08PM 0 points [-]

And what if he did ask?

Then, as I said, he cares about something closer to the reality.

The major point I've been trying to make in this thread is that because human preferences are not just in the map but of the map, is that it allows people to persist in delusions about their motivations. And not asking the question is a perfect example of the sort of decision error this can produce!

However, asking the question doesn't magically make the preference about the territory either; in order to prefer the future include his sister's best interests, he must first have an experience of the sister and a reason to wish well of her. But it's still better than not asking, which is basically wireheading.

The irony I find in this discussion is that people seem to think I'm in favor of wireheading because I point out that we're all doing it, all the time. When in fact, the usefulness of being aware that it's all wireheading, is that it makes you better at noticing when you're doing it less-usefully.

The fact that he hadn't asked his sister, or about his sister's actual well-being instantly jumped off the screen at me, because it was (to me) obvious wireheading.

So, you could say that I'm biased by my belief to notice wireheading more, but that's an advantage for a rationalist, not a disadvantage.

Comment author: Vladimir_Nesov 10 June 2009 05:07:32PM *  0 points [-]

The major point I've been trying to make in this thread is that because human preferences are not just in the map but of the map, is that it allows people to persist in delusions about their motivations.

Is human knowledge also not just in the map, but exclusively of the map? If not, what's the difference?

Comment author: pjeby 10 June 2009 06:00:00PM 1 point [-]

Is human knowledge also not just in the map, but exclusively of the map? If not, what's the difference?

Any knowledge about the actual territory can in principle be reduced to mechanical form without the presence of a human being in the system.

To put it another way, a preference is not a procedure, process, or product. The very use of the word "preference" is a mind projection - mechanical systems do not have "preferences" - they just have behavior.

The only reason we even think we have preferences in the first place (let alone that they're about the territory!) is because we have inbuilt mind projection. The very idea of having preferences is hardwired into the model we use for thinking about other animals and people.

Comment author: orthonormal 10 June 2009 03:53:31PM 0 points [-]

(That is, if you cared about something closer to the reality of what happens to your sister, rather than your experience of it, you'd have hesitated in that choice long enough to ask Omega whether she would prefer death to being imprisoned on Mars.)

Be charitable in your interpretation, and remember the Least Convenient Possible World principle. I was presuming that the setup was such that being alive on Mars wouldn't be a 'fate worse than death' for her; if it were, I'd choose differently. If you prefer, take the same hypothetical but with me on Mars, choosing whether she stayed alive on Earth; or let choice B include subjecting her to an awful fate rather than death.

That is, the model you make of the future may refer to a hypothetical reality, but the thing you actually evaluate is not that reality, but your own reaction to that reality -- your present-tense experience in response to a constructed fiction made of previous experiences.

I would say rather that my reaction is my evaluation of an imagined future world. The essence of many decision algorithms is to model possible futures and compare them to some criteria. In this case, I have complicated unconscious affective criteria for imagined futures (which dovetail well with my affective criteria for states of affairs I directly experience), and my affective reaction generally determines my actions.

We do not have preferences that are not about experience or our emotional labeling thereof; to the extent that we have "rational" preferences it is because they will ultimately lead to some desired emotion or sensation.

To the extent this is true (as in the sense of my previous sentence), it is a tautology. I understand what you're arguing against: the notion that what we actually execute matches a rational consequentialist calculus of our conscious ideals. I am not asserting this; I believe that our affective algorithms do often operate under more selfish and basic criteria, and that they fixate on the most salient possibilities instead of weighing probabilities properly, among other things.

However, these affective algorithms do appear to respond more strongly to certain facets of "how I expect the world to be" than to facets of "how I expect to think the world is" when the two conflict (with an added penalty for the expectation of being deceived), and I don't find that problematic on any level.

Comment author: pjeby 10 June 2009 05:31:20PM *  -1 points [-]

If you prefer, take the same hypothetical but with me on Mars, choosing whether she stayed alive on Earth; or let choice B include subjecting her to an awful fate rather than death.

As I said, it's still going to be about your experience during the moments until your memory is erased.

I understand what you're arguing against: the notion that what we actually execute matches a rational consequentialist calculus of our conscious ideals.

I took that as a given, actually. ;-) What I'm really arguing against is the naive self-applied mind projection fallacy that causes people to see themselves as decision-making agents -- i.e., beings with "souls", if you will. Asserting that your preferences are "about" the territory is the same sort of error as saying that the thermostat "wants" it to be a certain temperature. The "wanting" is not in the thermostat, it's in the thermostat's maker.

Of course, it makes for convenient language to say it wants, but we should not confuse this with thinking the thermostat can really "want" anything but for its input and setting to match. And the same goes for humans.

(This is not a mere fine point of tautological philosophy; human preferences in general suffer from high degrees of subgoal stomp, chaotic loops, and other undesirable consequences arising as a direct result of this erroneous projection. Understanding the actual nature of preferences makes it easier to dissolve these confusions.)

Comment author: Alicorn 09 June 2009 06:56:41AM 0 points [-]

I wish I could upvote this two or three times. Thank you.

Comment author: Vladimir_Nesov 09 June 2009 12:19:16PM 0 points [-]

What features of that comment made it communicate something new to you? What was it that got communicated?

The comment restated a claim that a certain relationship is desirable as a claim that given that it's desirable, there is a process that establishes it to be true. It's interesting how this restatement could pierce inferential distance: is preference less trustworthy than a fact, and so demonstrating the conversion of preference into a fact strengthens the case?

Comment author: Alicorn 09 June 2009 04:22:12PM *  0 points [-]

I'd been following this topic and getting frustrated with my inability to put my opinion on the whole preferences-about-the-territory thing into words, and I thought that orthonomal's comment accomplished it very nicely. I don't think I understand your other question.

Comment author: orthonormal 09 June 2009 11:50:00PM 0 points [-]

Given the length of the thread I branched from, it looks like you and P.J. Eby ended up talking past each other to some extent, and I think that you both failed to distinguish explicitly between the current map (which is what you calculate the territory to be) and a hypothetical future map.

P.J. Eby was (correctly) insisting that your utility function is only in contact with your current map, not the territory directly. You were (correctly) insisting that your utility function cares about (what it calculates to be) the future territory, and not just the future map.

Is that a fair statement of the key points?

Comment author: Vladimir_Nesov 10 June 2009 12:36:56AM 0 points [-]

Utility function is no more "in contact" with your current map than the actual truth of 2+2=4 is in contact with display of a calculator that displays the statement. Utility function may care about past territory (and even counterfactual territory) as well as future territory, with map being its part. Keeping a map in good health is instrumentally a very strong move: just by injecting an agent with your preferences somewhere in the territory you improve it immensely.

Comment author: orthonormal 10 June 2009 03:38:12PM 0 points [-]

While there might exist some abstracted idealized dynamic that is a mathematical object independent of your map, any feasible heuristic for calculating your utility function (including, of course, any calculation you actually do) will depend on your map.

If Omega came through tomorrow and made all pigs conscious with human-like thoughts and emotions, my moral views on pig farming wouldn't be instantly changed; only when information about this development gets to me and my map gets altered will I start assigning a much higher disutility to factory farming of pigs.

Or, to put it another way, a decision algorithm refers directly to the possible worlds in the territory (and their probabilities, etc), but it evaluates these referents by looking at the corresponding objects in its current map. I think that, since we're talking about practical purposes, this is a relevant point.

Keeping a map in good health is instrumentally a very strong move: just by injecting an agent with your preferences somewhere in the territory you improve it immensely.

Agree completely. Of the worlds where my future map looks to diverge from the territory, though, I'm generally more repulsed by the ones in which my map says it's fine where it's not than by the opposite.

Comment author: saturn 06 June 2009 07:48:47AM *  5 points [-]

Utility is about the territory in the same sense that the map is about the territory; the map tells us the way the territory is, utility tells us the way we want the territory to be. Us non-wireheaders want an accurate map because it's the territory we care about.

Supposing utility is not about the territory but about the map, we get people who want nothing more than to sabotage their own mapmaking capabilities. If the territory is not what we care about, maintaining the correspondence of map to territory would be a pointless waste of effort. Wireheading would look like an unambiguously good idea, not just to some people but to everyone.

Conchis' example of wanting his friends to really like and respect him is correct. He may have failed to explicitly point out that he also has an unrelated preference for not being beaten up. He's also in the unfortunate position of valuing something he can't know about without using long chains of messy inductive inferences. But his values are still about the territory, and he wants his map to accurately reflect the territory because it's the territory he cares about.

Comment author: pjeby 06 June 2009 04:06:41PM -1 points [-]

Utility is about the territory in the same sense that the map is about the territory; the map tells us the way the territory is, utility tells us the way we want the territory to be. Us non-wireheaders want an accurate map because it's the territory we care about.

I am only saying that the entire stack of concepts you have just mentioned exists only in your map.

Supposing utility is not about the territory but about the map, we get people who want nothing more than to sabotage their own mapmaking capabilities.

Permit me to translate: supposing utility is not about the (portion of map labeled) territory but about the (portion of map labeled) map, we get people who want nothing more than to sabotage their own mapmaking capabilities.

Does that make it any clearer what I'm saying?

This is a "does the tree make a sound" argument, and I'm on the, "no it doesn't" side, due to using a definition of "sound" that means "the representation of audio waves within a human nervous system". You are on the "of course it makes a sound" side, because your definition of sound is "pressure waves in the air."

Make sense?

Comment author: saturn 07 June 2009 12:17:08AM 1 point [-]

I am only saying that the entire stack of concepts you have just mentioned exists only in your map.

As far as I can tell, you're saying that there is no territory, or that the territory is irrelevant. In other words, solipsism. You've overcome the naive map/territory confusion, but only to wind up with a more sophisticated form of confusion.

This isn't a "does the tree make a sound" argument. It's more like a "dude... how do we even really know reality is really real" argument. Rationality is entirely pointless if all we're doing is manipulating completely arbitrary map-symbols. But in that case, why not leave us poor, deluded believers in reality to define the words "map", "territory", and "utility" the way we have always done?

Comment author: pjeby 07 June 2009 03:58:22AM 0 points [-]

In other words, solipsism.

No, general semantics. There's a difference.

Comment author: saturn 07 June 2009 10:49:45PM *  2 points [-]

Can you point out the difference?

Even though "this is not a pipe", the form of a depiction of a pipe is nevertheless highly constrained by the physical properties of actual pipes. Do you deny that? If not, how do you explain it?

Comment author: conchis 06 June 2009 04:19:20PM *  1 point [-]

This is a "does the tree make a sound" argument, and I'm on the, "no it doesn't" side, due to using a definition of "sound" that means "the representation of audio waves within a human nervous system". You are on the "of course it makes a sound" side, because your definition of sound is "pressure waves in the air."

I've been trying to be on the "it depends on your definition and my definition sits within the realm of acceptable definitions" side. Unfortunately, whether this is what you intend or not, most of your comments come across as though you're on the "it depends on the definition, and my (PJ's) defintion is right and yours is wrong" side, which is what seems to be getting people's backs up.

Comment author: Vladimir_Nesov 06 June 2009 04:29:58PM 0 points [-]

This confusion is dissolved in the post Disputing Definitions.

Comment author: conchis 06 June 2009 04:32:50PM *  0 points [-]

Which confusion? I didn't think I was confused. Now I'm confused about whether I'm confused. ;)

Comment author: Vladimir_Nesov 06 June 2009 04:40:14PM 0 points [-]

You mentioned this confusion as possibly playing a role in you and Eby talking past each other, the ambiguous use of the word "utility".

Comment author: Cyan 05 June 2009 06:40:30PM *  2 points [-]

There's two things to say in response to this: first, I can define "liking and respecting me" as "experiencing analogous brain states to mine when I like and respect someone else". That's in the territory (modulo some assumptions about the cognitive unity of humankind): I could verify it in principle, although not in practice.

The second thing is that even if we grant that the example was poor, the point was still valid. For example, one might prefer that one's spouse never cheat to one's spouse cheating but never being aware of that fact. (ETA: but maybe you weren't arguing against the point, only the example.)

Comment author: pjeby 05 June 2009 08:10:08PM -1 points [-]

There's two things to say in response to this: first, I can define "liking and respecting me" as "experiencing analogous brain states to mine when I like and respect someone else". That's in the territory (modulo some assumptions about the cognitive unity of humankind): I could verify it in principle, although not in practice.

But what if they experience that state, and still, say, beat you up and treat you like jerks, because that's what their map says you should do when you feel that way?

This isn't about the example being poor, it's about people thinking things in the map actually exist in the territory. Everything you perceive is mediated by your maps, even if only in the minimal sense of being reduced to human sensory-pattern recognition symbols first, let alone all the judgments about the symbols that we add on top.

For example, one might prefer that one's spouse never cheat to one's spouse cheating but never being aware of that fact. (ETA: but maybe you weren't arguing against the point, only the example.)

How about the case where you absolutely believe the spouse is cheating, but they really aren't?

This is certainly a better example, in that it's easier to show that it's not reality that you value, but the part of your map that you label "reality". If you really truly believe the spouse is cheating, then you will feel exactly the same as if they really are.

IOW, when you say that you value something "in the territory", all you are ever really talking about is the part of your map that you label "the territory", whether that portion of the map actually corresponds to the territory or not.

This is not some sort of hypothetical word-argument, btw. (I have no use for them, which is why I mostly avoid the Omega discussions.) This is a practical point for minimizing one's suffering and unwanted automatic responses to events in the world. To the extent you believe that your map is the territory, you will suffer when it is out-of-sync.

Comment author: Cyan 05 June 2009 08:27:17PM *  2 points [-]

But what if they experience that state, and still, say, beat you up and treat you like jerks, because that's what their map says you should do when you feel that way?

It's still possible to prefer this state of affairs to one where they are beating me because they are contemptuous of me. Remember, we're talking about a function from some set X to the real numbers, and we're trying to figure out what sorts of things are members of X. In general, people do have preferences about the way things actually are.

How about the case where you absolutely believe the spouse is cheating, but they really aren't?... If you really truly believe the spouse is cheating, then you will feel exactly the same as if they really are.

But my spouse won't, and I have preferences about that fact. All other things being equal, my preference ordering is "my spouse never cheats and I believe my spouse never cheats" > "my spouse cheats and I find out" > "my spouse cheats and I believe my spouse never cheats" > "my spouse never cheats but I believe she does". If a utility function exists that captures this preference, it will be a function that takes both reality and my map as arguments.

Comment author: Cyan 05 June 2009 03:29:26PM *  0 points [-]

You're right -- I should have written "necessarily based in the map, not just the territory."

My intended meaning has to do with fundamental shifts in one's understanding of how reality works that make some previous apparent question of fact become a "wrong question" or category error or similar non-issue.

Comment author: dclayh 04 June 2009 05:11:22AM *  5 points [-]

thinking "If I had X, would I take Y instead", and "If I had Y, would I take X instead" very often resulted in a pair of "No"s

It's a well-known result that losing something produces roughly twice the disutility that gaining the same thing would produce in utility. (I.e., we "irrationally" prefer what we already have.)

Comment author: conchis 04 June 2009 07:21:05PM *  1 point [-]

It's a well-known result that losing something produces roughly twice the disutility [as] gaining the same thing

This may depend what you mean by (dis)utility. Kermer et al. (one of the alii is Dan Gilbert) argue that "Loss aversion is an affective forecasting error", caused by a tendency to systematically overestimate negative emotional responses.

Comment author: taw 04 June 2009 07:46:04AM 0 points [-]

I thought I could trivially counter it by thinking about "X vs N Ys" and "Y vs M Xs", and geometrically averaging N with 1/M, but it didn't really work, and N/M values were often much larger than 2.

Comment author: SoullessAutomaton 04 June 2009 09:56:55AM 0 points [-]

Are you assuming your utility function for Xs and Ys is linear?

If X is "houses" and Y is "cars", and someone starts with one of each, how many people would gain utility from trading their only house for more cars or their only car for more houses?

Comment author: Philo 04 June 2009 06:02:58PM 0 points [-]

I'd trade my car for another house: virtually any house would be worth more than my old car; I could sell the house and buy a better car, with something left over!

Comment author: SoullessAutomaton 04 June 2009 08:51:28PM *  2 points [-]

But why would you sell the house and buy a car? Because you place higher utility on having one of each, which is precisely my point.

The fact that two houses can be converted into more fungible resources than two cars is true, but is, as thomblake said, missing the point.

Comment author: Alicorn 04 June 2009 06:06:06PM 1 point [-]

In this housing market? You'd be without a car for months waiting for the house to sell - would that be worth the vehicle upgrade and the leftover money, even assuming the house did eventually sell?

Comment author: Larks 08 August 2009 07:47:05PM 0 points [-]

Only if he tried to sell at the current market price (or what passes for one at the moment). I suspect if he tried to sell his house for something just above the price of a car, it would sell easily.

On the other hand, SoullessAutomaron's response is sound.

Comment author: thomblake 04 June 2009 06:40:06PM 0 points [-]

By "this housing market" did you mean the current one in the real world, or in the thought experiment where everyone already has exactly one house and car?

Either way it seems like an apt point, though it seems to miss the point of Philo's response (which in turn seemed to miss the point of SoullessAutomaton's question)

Comment author: Alicorn 04 June 2009 06:54:36PM 2 points [-]

I meant the real one, although the hypothetical universe where everybody has one house and one car would be hard on real estate sales too.

Comment author: taw 04 June 2009 01:20:19PM 0 points [-]

Another obvious trick of thinking about lotteries was even worse - I cannot get myself to do any kind of high-value one-off lottery thinking because of risk aversion, and low-value many-attempts lotteries are just like having EN Xs with some noise.

Comment author: JulianMorrison 08 June 2009 09:16:28AM 6 points [-]

You want a neuron dump? I don't have a utility function, I embody one, and I don't have read access to my coding.

Comment author: k3nt 28 June 2009 02:51:51PM 2 points [-]

I'm not sure I embody one! I'm not sure that I don't just do whatever seems like the next thing to do at the time, based on a bunch of old habits and tendencies that I've rarely or never examined carefully.

I get up in the morning. I go to work. I come home. I spend more time reading the internets (both at work and at home) than I probably should -- on occasion I spend most of the day reading the internets, one way or another, and while I'm doing so have a vague but very real thought that I would prefer to be doing something else, and yet I continue reading the internets.

I eat more or less the same breakfast and the same lunch most days, just out of habit. Do I enjoy these meals more than other options? Almost certainly not. It's just habit, it's easy, I do it without thinking. Does this mean that I have a utility function that values what's easy and habitual over what would be enjoyable? Or does it mean that I'm not living in accord with my utility function?

In other words, is the sentence "I embody a utility function" intended to be tautological, in that by definition, any person's way of living reveals/embodies their utility function (a la "revealed preferences" in economics), or is it supposed to be something more than that, something to aspire to that many people fail at embodying?

If "I embody a utility function" is aspirational rather than tautological -- something one can fail at -- how many people reading this believe they have succeeded or are succeeding in embodying their utility function?

Comment author: Dagon 04 June 2009 03:50:39PM *  6 points [-]

I've put a bit of thought into this over the years, and don't have a believable theory yet. I have learned quite a bit from the excercise, though.

1) I have many utility functions. Different parts of my identity or different frames of thought engage different preference orders, and there is no consistent winner. I bite this bullet: personal identity is a lie - I am a collective of many distinct algorithms. I also accept that Arrow’s impossibility theorem applies to my own decisions.

2) There are at least three dimensions (time, intensity, and risk) to my utility curves. None of these are anywhere near linear - the time element seems to be hyperbolic in terms of remembered happiness for past events, and while I try to keep it sane for future events, that's not my natural state, and I can't do it for all my pieces with equal effectiveness.

3) They change over time (which is different than the time element within the preference space). Things I prefer now, I will not necessarily prefer later. The meta-utility of balancing this possibly-anticipated change against the timeframe of the expected reward is very high, and I can sometimes even manage it.

Comment deleted 07 June 2009 02:05:09PM *  [-]
Comment author: Dagon 08 June 2009 11:06:28PM 0 points [-]

It's not clear to me that my subpersonal algorithms have the ability to enforce reciprocity well enough, or to reflectively alter themselves with enough control to even make an attempt at unification. Certainly parts of me attempt to modify other parts in an attempt to do so, but that's really more conquest than reciprocity (a conquest "I" pursue, but still clearly conquest).

Unification is a nice theory, but is there any reason to think it's possible for subpersonal evaluation mechanisms any more than it is for interpersonal resource sharing?

Comment author: Vladimir_Nesov 07 June 2009 02:18:37PM 0 points [-]

It is in interest of each and every agent to unify (coordinate) more with other agents, so this glosses over the concept of the individual.

Comment author: Cyan 07 June 2009 02:42:51PM *  1 point [-]

...this glosses over the concept of the individual.

This misses the mark, I think. Here's a mutation:

"It is in interest of each and every cell to unify (coordinate) more with other cells, so this glosses over the concept of the organism."

The coordination of cells is what allows us to speak of an organism as a whole. I won't go so far as to declare that co-ordination of agents justifies the concept of the individual, but I do think the idea expressed in the parent is more wrong than right.

Comment author: Drahflow 04 June 2009 10:07:38AM 4 points [-]

I feel some people here are trying to define their utility functions via linear combinations of sub-functions which only depend on small parts of the world state.

Example: If I own X, that'll give me a utility of 5, if I own Y that'll give me a utility of 3, if I own Z, that'll give me a utility of 1.

Problem: Choose any two of {X, Y, Z}

Apparent Solution: {X, Y} for a total utility of 8.

But human utility functions are not a linear combination of such sub-functions, but functions from global World states into the real numbers. Think about the above example with X = Car, Y = Bike, Z = Car keys.

It seems obvious now, but the interdependencies are much more complicated. Like money utility being dependent on the market situation, food utility being dependent on the stuff you ate recently, material (as in building) utility being dependent on available tools and vice versa, internet availability utility being dependent on available computer, power, and time.

Comment author: loqi 04 June 2009 05:56:56AM 4 points [-]

This leads me to two possible conclusions

A third possibility: Humans aren't in general capable of accurately reflecting on their preferences.

Utility functions are really bad match for human preferences, and one of the major premises we accept is wrong.

If utility functions are a bad match for human preferences, that would seem to imply that humans simply tend not to have very consistent preferences. What major premise does this invalidate?

Comment author: SoullessAutomaton 04 June 2009 10:09:06AM *  6 points [-]

A third possibility: Humans aren't in general capable of accurately reflecting on their preferences.

Humans are obviously capable of perceiving their own preferences at some level, otherwise they'd be unable to act on them. I assume what you propose here is that conscious introspection is unable to access those preferences?

In that case, utility functions could potentially be deduced by the individual placing themselves into situations that require real action based on relevant preferences, recording their choices, and attempting to deduce a consistent basis that explains those choices. I'm pretty sure that someone with a bit of math background who spent a few days taking or refusing various bets could deduce the nonlinearity and approximate shape of their utility function for money without any introspection, for instance.

Comment author: taw 04 June 2009 06:40:41AM 0 points [-]

A third possibility: Humans aren't in general capable of accurately reflecting on their preferences.

Three is pretty much like one. If utility functions work, there must be some way of figuring them out, I hoped someone figured it out already.

If utility functions are a bad match for human preferences, that would seem to imply that humans simply tend not to have very consistent preferences. What major premise does this invalidate?

Utilitarian model being wrong doesn't necessarily mean that a different model based on different assumptions doesn't exist. I don't know which assumptions need to be broken.

Comment author: StanR 04 June 2009 07:31:55AM *  2 points [-]

The general premise in the mind sciences is that there are different selves, somehow coordinated through the cortical midline structures. Plenty of different terms have been used, and hypotheses suggested, but the two "selves" I use for shorthand come from Daniel Gilbert: Socrates and the dog. Socrates is the narrative self, the dog is the experiencing self. If you want something a bit more technical, I suggest the lectures about well-being (lecture 3) here, and to get really technical, this paper on cognitive science exploring the self.

Comment author: conchis 04 June 2009 11:13:31AM *  2 points [-]

What counts as a "successful" utility function?

In general terms there are two, conflicting, ways to come up with utility functions, and these seem to imply different metrics of success.

  1. The first assumes that "utility" corresponds to something real in the world, such as some sort of emotional or cognitive state. On this view, the goal, when specifying your utility function, is to get numbers that reflect this reality as closely as possible. You say "I think x will give me 2 emotilons", and "I think y will give me 3 emotilons"; you test this by giving yourself x, and y; and success is if the results seem to match up.

  2. The second assumes that we already have a set of preferences, and "utility" is just a number we use to represent these, such that xPy <=> u(x)>u(y), where xPy means "x is preferred to y". (More generally, when x and y may be gambles, we want: xPy <=> E[u(x)]>E[u(y)]).

It's less clear what the point of specifying a utility function is supposed to be in the second case. Once you have preferences, specifying the utility function has no additional information content: it's just a way of representing them with a real number. I guess "success" in this case simply consists in coming up with a utility function at all: if your preferences are inconsistent (e.g. incomplete, intransitive, ...) then you won't be able to do it, so being able to do it is a good sign.

Much of the discussion about utility functions on this site seems to me to conflate these two distinct senses of "utility", with the result that it's often difficult to tell what people really mean.

Comment author: bill 07 June 2009 04:24:55PM 1 point [-]

When I teach decision analysis, I don't use the word "utility" for exactly this reason. I separate the "value model" from the "u-curve."

The value model is what translates all the possible outcomes of the world into a number representing value. For example, a business decision analysis might have inputs like volume, price, margin, development costs, etc., and the value model would translate all of those into NPV.

You only use the u-curve when uncertainty is involved. For example, distributions on the inputs lead to a distribution on NPV, and the u-curve would determine how to assign a value that represents the distribution. Some companies are more risk averse than others, so they would value the same distribution on NPV differently.

Without a u-curve, you can't make decisions under uncertainty. If all you have is a value model, then you can't decide e.g. if you would like a deal with a 50-50 shot at winning $100 vs losing $50. That depends on risk aversion, which is encoded into a u-curve, not a value model.

Does this make sense?

Comment author: conchis 07 June 2009 04:29:01PM 0 points [-]

Totally. ;)

Comment author: bill 04 June 2009 05:58:44AM *  5 points [-]

Here's one data point. Some guidelines have been helpful for me when thinking about my utility curve over dollars. This has been helpful to me in business and medical decisions. It would also work, I think, for things that you can treat as equivalent to money (e.g. willingness-to-pay or willingness-to-be-paid).

  1. Over a small range, I am approximately risk neutral. For example, a 50-50 shot at $1 is worth just about $0.50, since the range we are talking about is only between $0 and $1. One way to think about this is that, over a small enough range, there isn't much practical difference between a curve and a straight line approximating that curve. Over the range -$10K and +$20K I am risk neutral.

  2. Over a larger range, my utility curve is approximately exponential. For me, between -$200K and +$400K, my utility curve is fairly close to u(x) = 1 - exp (-x/400K). The reason is that, for me, changing my wealth by a relatively small amount won't radically change my risk preference, and that implies an exponential curve. Give me $1M and my risk preferences might change, but within the above range, I pretty much would make the same decisions.

  3. Outside that range, it gets more complicated than I think I should go into here. In short, I am close to logarithmic for gains and exponential for losses, with many caveats and concerns (e.g. avoiding the zero illusion. My utility curve should not have any sort of "inflection point" around my current wealth; there's nothing special about that particular wealth level).

(1) and (2) can be summarized with one number, my risk tolerance of $400K. One way to assess this for yourself is to ask "Would I like a deal with a 50-50 shot at winning $X versus losing $X/2?" The X that makes you indifferent between having the deal and not having the deal is approximately your risk tolerance. I recommend acting risk neutral for deals between $X/20 and minus $X/40, and use an exponential utility function between $X and minus $X/2. If the numbers get too large, thinking about them in dollars per year instead of total dollars sometimes helps. For example, $400K seems large, but $20K per year forever may be easier to think about.

Long, incomplete answer, but I hope it helps.

Comment author: AndrewKemendo 06 June 2009 06:40:44PM 1 point [-]

How have you come to these conclusions?

For example:

The reason is that, for me, changing my wealth by a relatively small amount won't radically change my risk preference, and that implies an exponential curve

Is that because there have been points in time when you have made 200K and 400K respectively and found that your preferences didn't change much. Or is that simply expected utility?

Comment author: bill 06 June 2009 07:13:45PM 0 points [-]

For the specific quote: I know that, for a small enough change in wealth, I don't need to re-evaluate all the deals I own. They all remain pretty much the same. For example, if you told me a had $100 more in my bank account, I would be happy, but it wouldn't significantly change any of my decisions involving risk. For a utility curve over money, you can prove that that implies an exponential curve. Intuitively, some range of my utility curve can be approximated by an exponential curve.

Now that I know it is exponential over some range, I needed to figure out which exponential and over what range does it apply. I assessed for myself that I am indifferent between having and not having a deal with a 50-50 chance of winning $400K and losing $200K. The way I thought about that was how I thought about decisions around job hunting and whether I should take or not take job offers that had different salaries.

If that is true, you can combine it with the above and show that the exponential curve should look like u(x) = 1 - exp(-x/400K). Testing it against my intuitions, I find it an an okay approximation between $400K and minus $200K. Outside that range, I need better approximations (e.g. if you try it out on a 50-50 shot of $10M, it gives ridiculous answers).

Does this make sense?

Comment author: AndrewKemendo 06 June 2009 10:29:42PM 0 points [-]

It makes sense however you mention that you test it against your intuitions. My first reaction would be to say that this is introducing a biased variable which is not based on a reasonable calculation.

That may not be the case as you may have done so many complicated calculations such that your unconscious "intuitions" may give your conscious the right answer. However from the millionaires biographies I have read and rich people I have talked to a better representation of money and utility according to them is logarithmic rather than exponential. This would indicate to me that the relationship between utility and money would be counter-intuitive for those who have not experienced those levels which are being compared.

I have not had the fortune to experience anything more than a 5 figure income so I cannot reasonably say how my preferences would be modeled. I can reasonably believe that I would be better off at 500K than 50K through simple comparison of lifestyle between myself and a millionaire. I cannot make an accurate enough estimation of my utility and as a result I would not be prepared to make a estimation of what model would best represent it because the probability of that being accurate is likely the same as coin flipping.

Ed: I had a much better written post but an errant click lost the whole thing - time didn't allow the repetition of the better post.

Comment author: bill 06 June 2009 11:41:47PM 0 points [-]

As I said in my original post, for larger ranges, I like logarithmic-type u-curves better than exponential, esp. for gains. The problem with e.g. u(x)=ln(x) where x is your total wealth is that you must be indifferent between your current wealth and a 50-50 shot of doubling vs. halving your wealth. I don't like that deal, so I must not have that curve.

Note that a logarithmic curve can be approximated by a straight line for some small range around your current wealth. It can also be approximated by an exponential for a larger range. So even if I were purely logarithmic, I would still act risk neutral for small deals and would act exponential for somewhat larger deals. Only for very large deals indeed would you be able to identify that I was really logarithmic.

Comment author: conchis 07 June 2009 12:25:54AM *  1 point [-]

Further to this, it's also worth pointing out that, to the extent that Andew's biographies and rich acquaintances are talking about a logarithmic experienced utility function that maps wealth into a mind state something like "satisfaction", this doesn't directly imply anything about the shape of the decision utility function they should use to represent their preferences over gambles.

It's only if they're also risk neutral with respect to experienced utility that the implied decision utility function needs to be log(x). If they're risk averse with respect to experienced utility then their decision utility function will be a concave function of log(x), while if they're risk loving it will be a convex function of it.

P.S. For more on the distinction between experienced and decision utility (which I seem constantly to be harping on about) see: Kahneman, Wakker and Sarin (1997) "Back to Bentham? Explorations of Experienced Utility"

Comment author: AndrewKemendo 07 June 2009 04:33:17AM *  0 points [-]

It's only if they're also risk neutral with respect to experienced utility

I am curious how this would look in terms of decisions under experience. Does this imply that they are expecting to change their risk assessment once they are experienced?

Comment author: conchis 07 June 2009 05:28:02AM *  0 points [-]

I'm afraid I have no idea what you mean, perhaps because I failed to adequately explain the distinction between experienced utility and decision utility, and you've taken it to mean something else entirely. Roughly: experienced utility is something you experience or feel (e.g. positive emotions); decision utility is an abstract function that describes the decisions you make, without necessarily corresponding to anything you actually experience.

Follow the link I gave, or see my earlier comment here (experienced utility is 1., decision utility is 2.)

Apologies if I'm failing to understand you for some other reason, such as not having slept. ;)

Comment author: AndrewKemendo 07 June 2009 04:27:58AM 0 points [-]

Unfortunately the better parts of my post were lost - or rather more of the main point.

I posit that the utility valuation is an impossibility currently. I was not really challenging whether your function was exponential or logarithmic - but questioning how you came to the conclusion; how you decide, for instance where exactly the function changes especially having not experienced the second state. The "logarithmic" point I was making was designed to demonstrate that true utility may differ significantly from expected utility once you are actually at point 2 and thus may not be truly representative.

Mainly I am curious as to what value you place on "intuition" and why.

Testing it against my intuitions

Comment author: bill 07 June 2009 04:12:29PM 1 point [-]

If you wanted to, we could assess at least a part of your u-curve. That might show you why it isn't an impossibility, and show what it means to test it by intuitions.

Would you, right now, accept a deal with a 50-50 chance of winning $100 versus losing $50?

If you answer yes, then we know something about your u-curve. For example, over a range at least as large as (100, -50), it can be approximated by an exponential curve with a risk tolerance parameter of greater than 100 (if it were less that 100, then you wouldn't accept the above deal).

Here, I have assessed something about your u-curve by asking you a question that it seems fairly easy to answer. That's all I mean by "testing against intuitions." By asking a series of similar questions I can assess your u-curve over whatever range you would like.

You also might want to do calculations: for example, $10K per year forever is worth around $300K or so. Thinking about losing or gaining $10K per year for the rest of your life might be easier than thinking about gaining or losing $200-300K.

Comment author: AndrewKemendo 08 June 2009 11:28:28PM 1 point [-]

I think this greatly oversimplifies the issue. Whatever my response to the query is, it is only an estimation as to my preferences. It also assumes that my predicted risk will, upon the enactment of an actual deal, stay the same; if only for the life of the deal.

A model like this, even if correct for right now, could be significantly different tomorrow or the next day. It could be argued that some risk measurements do not change at intervals so fast as would technically prohibit recalculation. Giving a fixed metric puts absolutes on behaviors which are not fixed, or which unpredictably change. Today, because I have lots of money in my account, I might agree to your deal. Tomorrow I may not. This is what I mean by intuitions - I may think I want the deal but I may in reality be significantly underestimating the chance of -50 or any other number of factors which may skew my perception.

I know of quite a few examples of people getting stuck in high load mutual funds or other investments because their risk preferences significantly changed over a much shorter time period than they expected because they really didn't want to take that much risk in their portfolio but could not cognitively comprehend the probability as most people cannot.

This in no way advocates going further to correcting for these mistakes after the fact - however the tendencies for economists and policy makers is to suggest modeling such as this. In fact most consequentialists make the case that modeling this way is accurate however I have yet to see a true epistemic study of a model which reliably demonstrates accurate "utility" or valuation. The closest to accurate models I have seen take stated and reveled preferences together and work towards a micro estimation which still has moderate error variability where not observed (http://ideas.repec.org/a/wly/hlthec/v13y2004i6p563-573.html). Even with observed behavior applied it is still terribly difficult and unreliable to apply broadly - even to an individual.

Comment author: conchis 06 June 2009 11:05:55PM *  0 points [-]

Just to be clear, you know that an exponential utility function (somewhat misleadingly ) doesn't actually imply that utility is exponential in wealth, right? Bill's claimed utility function doesn't exhibit increasing marginal utility, if that's what you're intuitively objecting to. It's 1-exp(-x), not exp(x).

Many people do find the constant absolute risk aversion implied by exponential utility functions unappealing, and prefer isoelastic utility functions that exhibit constant relative risk aversion, but it does have the advantage of tractability, and may be reasonable over some ranges.

Comment author: bill 06 June 2009 11:53:15PM 0 points [-]

Example of the "unappealingness" of constant absolute risk aversion. Say my u-curve were u(x) = 1-exp(-x/400K) over all ranges. What is my value for a 50-50 shot at 10M?

Answer: around $277K. (Note that it is the same for a 50-50 shot at $100M)

Given the choice, I would certainly choose a 50-50 shot at $10M over $277K. This is why over larger ranges, I don't use an exponential u-curve.

However, it is a good approximation over a range that contains almost all the decisions I have to make. Only for huge decisions to I need to drag out a more complicated u-curve, and they are rare.

Comment author: CannibalSmith 04 June 2009 06:36:40PM 2 points [-]

I realize that my utility function is inscrutable and I trust the unconscious par of me to make accurate judgments of what I want. When I've determined what I want, I use the conscious part of me to determine how I'll achieve it.

Comment author: Vladimir_Nesov 04 June 2009 06:43:45PM 4 points [-]

Don't trust the subconscious too much in determining what you want either. Interrogate it relentlessly, ask related questions, find incoherent claims and force the truth about your preference to the surface.

Comment author: CannibalSmith 04 June 2009 07:27:16PM 1 point [-]

You seem to mistakenly think that my subconscious is not me.

Comment author: Cyan 04 June 2009 08:34:36PM *  3 points [-]

I'd say rather that it seems he doesn't trust one's subconscious to be self-consistent -- and that doesn't seem mistaken to me.

Comment author: Vladimir_Nesov 04 June 2009 08:57:12PM 1 point [-]

Correct.

I wasn't very careful by implying that the truth extracted from subconscious is to be accepted: the criteria for acceptance, or trust is also a component of your values (that suggests an additional limit of reflective consistency), closing on itself, to be elicited as well.

Comment author: derekz 04 June 2009 12:12:19PM 2 points [-]

Interesting exercise. After trying for a while I completely failed; I ended up with terms that are completely vague (e.g. "comfort"), and actually didn't even begin to scratch the surface of a real (hypothesized) utility function. If it exists it is either extremely complicated (too complicated to write down perhaps) or needs "scientific" breakthroughs to uncover its simple form.

The result was also laughably self-serving, more like "here's roughly what I'd like the result to be" than an accurate depiction of what I do.

The real heresy is that this result does not particularly frighten or upset me. I probably can't be a "rationalist" when my utility function doesn't place much weight on understanding my utility function.

Can you write your own utility fuinction or adopt the one you think you should have? Is that sort of wholesale tampering wise?

Comment author: DanielLC 29 December 2009 05:09:11AM *  1 point [-]

So, we're just listing how much we'd buy things for? I don't see why it's supposed to be hard.

I guess it gets a bit complicated when you consider combinations of things, rather than just their marginal value. For example, once I have a computer with an internet connection, I care for little else. Still, I just have to figure out what would be about neutral, and decide how much I'd pay an hour (or need to be payed an hour) to go from that to something else.

Playing a vaguely interesting game on the computer = 0.

Doing something interesting = 1-3.

Talking to a cute girl = 5.

Talking to a cute girl I know = 8.

Talking to the girl I really like = 50.

Thinking about a girl I really like if I talked to her within the last couple of days, or probably will within a couple of days = 4.

Having hugged the girl I really like within the last two hours = 50.

Hugging a cute girl I know = 50. Note that this one only lasts for about a second, so it's only about a 7000th as good as the last one.

Hugging a cute girl I don't know = 20.

Hugging anyone else except my brother = 10.

Homework = -2, unless it's interesting.

Eating while hungry = 2.

Asleep = ??? I have no idea how to figure that one out.

I didn't use dollar value because I'm too cheap to actually spend money. Knowing how much I can help people for the same cost will do that to you. Check out the Disease Control Priorities Project (http://tinyurl.com/y9wpk5e). There's one for $3 a QALY. Even the hugging the girl I like I only estimate at 0.02 QALYs.

Using that estimate, one unit is about twice my average happiness. More accurate than I'd expect.

Comment author: Swimmer963 18 April 2012 01:06:59PM 0 points [-]

The link is really interesting!

Comment author: PhilGoetz 09 June 2009 03:48:27AM *  1 point [-]

Your observation is interesting. Note that I can't write down my wave function, either, but that doesn't mean I don't have one.

Comment author: orthonormal 09 June 2009 05:07:29AM 2 points [-]

And without being able to calculate it exactly, you can approximate it usefully, and thus derive some of its most relevant properties for practical purposes.

Comment author: DanielLC 29 December 2009 05:10:52AM 1 point [-]

He didn't say it did. It's just one possible reason for it.

Comment author: timtyler 05 June 2009 05:17:53PM *  1 point [-]

For a thread entitled "Post Your Utility Function" remarkably few people have actually posted what they think their utility function is.

Are people naturally secreteve about what they value? If so, why might that be?

Do people not know what their utility function is? That seems strange for such a basic issue.

Do people find their utility function hard to express? Why might that be?

Comment author: ShardPhoenix 06 June 2009 05:14:04PM *  0 points [-]

I assume that specifying your utility function is difficult for the same reasons that specifying the precise behaviour of any other aspect of your brain is difficult.

If you're talking about a conscious, explicitly evaluated utility function then I doubt even most rationalists have such a thing.

Comment author: timtyler 07 June 2009 08:25:02AM *  0 points [-]

I wasn't expecting the ten significant figures.

To be able to say what a utility function is, you have to be conscious of it. It has to be evaluated - or else it isn't your utility function. However, I am not sure that I know what you mean by "explicitly evaluated".

In many cases, behaviour is produced unconsciously. The idea is more that a utility function should be consistent with most goal-oriented behaviour. If you claim that your goal in life is to convert people to Christianity, then you should show signs of actually trying to do that, as best you are able.

Comment author: pjeby 04 June 2009 06:00:49AM 1 point [-]

Utility functions are really bad match for human preferences, and one of the major premises we accept is wrong.

Human utility functions are relative, contextual, and include semi-independent positive-negative axes. You can't model all that crap with one number.

The study of affective synchrony shows that humans have simultaneously-active positive and negative affect systems. At extreme levels in either system, the other is shut down, but the rest of the time, they can support or oppose each other. (And in positions of opposition, we experience conflict and indecision.)

Meanwhile, the activation of these systems is influenced by current state/context/priming, as well as the envisioned future. So unless your attempt at modeling a utility function includes terms for all these things, you're sunk.

(Personally, this is where I think the idea of CEV has its biggest challenge: I know of no theoretical reason why humans must have convergent or consistent utility functions as individuals, let alone as a species.)

Comment author: conchis 04 June 2009 10:05:54AM *  2 points [-]

Human utility functions are relative, contextual, and include semi-independent positive-negative axes. You can't model all that crap with one number.

I don't really see why not (at least without further argument).

  1. Relativity and contextuality introduce additional arguments into the utility function, they don't imply that the output can't be scalar. Lots of people include relativity and contextual concerns into scalar utility all the time.

  2. Semi-independent positive and negative axes only prevent you from using scalar utility if you think they're incommensurable. If you can assign weights to the positive and negative axes, then you can aggregate them into a single utility index. (How accurately you can do this is a separate question.)

Of course, if you do think there are fundamentally incommensurable values, then scalar utility runs into trouble.* Amartya Sen and others have done interesting work looking at plural/vector utility and how one might go about using it. (I guess if we're sufficiently bad at aggregating different types of value, such methods might even work better in practice than scalar utility.)

* I'm sceptical; though less sceptical than I used to be. Most claims of incommensurability strike me as stemming from unwillingness to make trade-offs rather than inability to make trade-offs, but maybe there are some things that really are fundamentally incomparable.

Comment author: pjeby 04 June 2009 04:24:16PM *  1 point [-]

Most claims of incommensurability strike me as stemming from unwillingness to make trade-offs rather than inability to make trade-offs, but maybe there are some things that really are fundamentally incomparable.

My point was that even if you can make a tradeoff, you're likely to have at least some disutility for Omega making that tradeoff for you, rather than letting you make the tradeoff yourself.

My own personal observation, though, is that people don't usually make good tradeoffs by weighing and combining the utility and disutility for each of their options; they're happier (and their lives are more generally functional), when they work to maximize utility and then satisfice disutility, in that order.

Our hardware doesn't do well at cross-comparison, but it can handle, "Which of these do I like best?", followed by "What am I willing to trade off to get what I like best?" (It can also handle the reverse, but that road leads to a dysfunctional and ever-shrinking "comfort zone".)

I assume that this is because the two affect systems were intended for approach and avoidance of predators, prey, and mates, rather than making rational tradeoffs between a wide array of future options.

Each system is quite capable of ranking threats or opportunities within its own value system, but there doesn't seem to be a register or readout in the system that can hold a "pleasure minus pain" value. What appears to happen instead is that the conscious mind can decide to switch off the negative input, if there's an inner consensus that the worst-case downside is stlil manageable.

This mechanism, however, appears to only operate on one goal at at time; it doesn't seem to work to try to cram all your options into it at once.

In the aggregate, these mechanisms would be really difficult to model, since the disutility/worst-case scenario check often depends on the examination of more than one possible future and contemplating possible mitigations, risks, etc.

I guess what I'm trying to say is that not only is goal+context important, there's also a cross-time or over-time input component as well, and that I don't really see anything that allows a person's preferences to be absolute, because the "tradeoff" part is something that can happen consciously or unconsciously, and is very sensitive to the steps undertaken to make the tradeoff. But despite this sensitivity, the emotional affect of having made the choice is the same - we defend it, because we own it.

In contrast, a rational weighing of ratios and scores can easily produce a different felt-sensation about the decision: one of not really having decided at all!

If a person "decides" based only on the objective/numerical criteria (even if this includes scoring and weighing his or her emotional responses!), this ownership/territory mechanism does not kick in, with resulting negative consequences for that person's persistence and commitment.

For example, if you "decide" to go on a diet because you're 20 pounds overweight, you may stop eating healthily (or at least cease to do so consistently) as you approach your desired weight.

Now, that's not to say you can't weigh all the objective information, and then make a decision that's not conditional upon those facts, or is conditional upon those facts only at the point of time you originally received them. I'm just saying that if you just weigh up the facts and "let the facts decide", you are just begging for an akrasia problem.

This is why, btw, effective decision makers and successful people tend to talk about "listening to all the input first, and then making their own decision".

It's because they need to make it theirs -- and there's no way for the math to do that, because the calculation has to run on the right brain hardware first. And the mathematical part of your brain ain't it.

Comment author: conchis 04 June 2009 06:17:36PM *  3 points [-]

OK, there's a lot of food for thought in there, and I can't possibly hope to clarify everything I'd ideally like to, but what I think you're saying is:

  1. it's theoretically possible to think about utility as a single number; but
  2. it's nonetheless a bad idea to do so, because (a) we're not very good at it, and (b) thinking about things mathematically means we won't "own" the decision, and therefore leads to akrasia problems

(FWIW, I was only claiming 1.) I'm fairly sympathetic to 2(a), although I would have thought we could get better at it with the right training. I can see how 2(b) could be a problem, but I guess I'm not really sure (i) that akrasia is always an issue, and (ii) why (assuming we could overcome 2(a)) we couldn't decide mathematically, and then figure out how to "own" the decision afterwards. (This seems to have worked for me, at least; and stopping to do the math has at sometimes stopped me "owning" the wrong decision, which can be worse than half-heartedly following through on the right one.)

P.S.

My point was that even if you can make a tradeoff, you're likely to have at least some disutility for Omega making that tradeoff for you, rather than letting you make the tradeoff yourself.

I didn't think anyone was suggesting Omega should make the trade-off. I certainly wasn't.

Comment author: pjeby 04 June 2009 10:16:24PM 1 point [-]

(FWIW, I was only claiming 1.) I'm fairly sympathetic to 2(a), although I would have thought we could get better at it with the right training. I can see how 2(b) could be a problem, but I guess I'm not really sure (i) that akrasia is always an issue, and (ii) why (assuming we could overcome 2(a)) we couldn't decide mathematically, and then figure out how to "own" the decision afterwards.

To own it, you'd need to not mathematically decide; the math could only ever be a factor in your decision. There's an enormous gap between, "the math says do this, so I guess I'll do that", and "after considering the math, I have decided to do this." The felt-experience of those two things is very different, and it's not merely an issue of using different words.

Regarding getting better at making decisions off of mathematics, I think perhaps you miss my point. For humans, the process by which decision-making is done, has consequences for how it's implemented, and for the person's experience and satisfaction regarding the decision itself. See more below...

(This seems to have worked for me, at least; and stopping to do the math has at sometimes stopped me "owning" the wrong decision, which can be worse than half-heartedly following through on the right one.)

I'd like to see an actual, non-contrived example of that. Mostly, my experience is that people are generally better off with a 50% plan executed 100% than a 100% plan executed 50%. It's a bit of a cliche -- one that I also used to be skeptical/cynical about -- but it's a cliche because it's true. (Note also that in the absence of catastrophic failure, the primary downside of a bad plan is that you learn something, and you still usually make some progress towards your goals.)

It's one of those places where in theory there's no difference between theory and practice, but in practice there is. We just think differently when we're considering something from when we're committed to it -- our brains just highlight different perceptions and memories for our attention, so much so that it seems like all sorts of fortunate coincidences are coming your way.

Our conscious thought process in System 2 is unchanged, but something on the System 1 level operates differently with respect to a decision that's passed through the full process.

I used to be skeptical about this, before I grasped the system 1/system 2 distinction (which I used to call the "you" (S2) vs. "yourself" (S1) distinction). I assumed that I could make a better plan before deciding to do something or taking any action, and refused to believe otherwise. Now I try to plan just enough to get S1 buy-in, and start taking action so I can get feedback sooner.

Comment author: conchis 04 June 2009 11:47:04PM *  1 point [-]

the math could only ever be a factor in your decision.

Sure. I don't think this is inconsistent with what I was suggesting, which was really just that that the math could start the process off.

For humans, the process by which decision-making is done, has consequences for how it's implemented, and for the person's experience and satisfaction regarding the decision itself.

All of which I agree with; but again, I don't see how this rules out learning to use math better.

Mostly, my experience is that people are generally better off with a 50% plan executed 100% than a 100% plan executed 50%.

Fair enough. The examples I'm thinking of typically involve "owned" decisions that are more accurately characterised as 0% plans (i.e. do nothing) or -X% plans (i.e. do things that are actively counterproductive).

Now I try to plan just enough to get S1 buy-in, and start taking action so I can get feedback sooner.

  1. How do you decide what to get S1 to buy in to?
  2. What do you do in situations where feedback comes too late (long term investments with distant payoffs) or never (e.g. ethical decisions where the world will never let you know whether you're right or not).

P.S. Yes, I'm avoiding the concrete example request. I actually have a few, but they'd take longer to write up than I have time available at the moment, and involve things I'm not sure I'm entirely comfortable sharing.

Comment author: pjeby 05 June 2009 12:49:29AM 0 points [-]

How do you decide what to get S1 to buy in to?

I already explained: you select options by comparing their positive traits. The devil is in the details, of course, but as you might imagine I do entire training CDs on this stuff. I've also written a few blog articles about this in the past.

What do you do in situations where feedback comes too late (long term investments with distant payoffs) or never (e.g. ethical decisions where the world will never let you know whether you're right or not).

I don't understand the question. If you're asking how I'd know whether I made the best possible decision, I wouldn't. Maximizers do very badly at long-term happiness, so I've taught myself to be a satisficer. I assume that the decision to invest something for the long term is better than investing nothing, and that regarding an ethical decision I will know by the consequences and my regrets or lack thereof whether I've done the "right thing"... and I probably won't have to wait very long for that feedback.

Comment author: Cyan 04 June 2009 10:48:46PM *  1 point [-]

..I'm not really sure... why [] we couldn't decide mathematically, and then figure out how to "own" the decision afterwards.

There's an enormous gap between, "the math says do this, so I guess I'll do that", and "after considering the math, I have decided to do this." The felt-experience of those two things is very different, and it's not merely an issue of using different words.

One can imagine a person who has committed emotionally to the maxim "shut up and multiply (when at all possible)" and made it an integral part of their identity. For such an individual, the commitment precedes the act of doing the math, and the enormous gap referred to above does not exist.

Comment author: pjeby 05 June 2009 01:10:25AM 1 point [-]

For such an individual, the commitment precedes the act of doing the math, and the enormous gap referred to above does not exist.

If such an individual existed, they would still have the same problem of shifting decisions, unless they also included a commitment to not recalculate before a certain point.

Consider, e.g. Newcomb's problem. If you do the calculation before, you should one-box. But doing the calculation at the actual time, means you should two-box.

So, to stick to their commitments, human beings need to precommit to not revisiting the math, which is a big part of my point here.

Your hypothetical committed-to-the-math person is not committed to their "decisions", they are committed to doing what the math says to do. This algorithm will not produce the same results as actual commitment will, when run on human hardware.

To put it more specifically, this person will not get the perceptual benefits of a committed decision for decisions which are not processed through the machinery I described earlier. They will be perceptually tuned to the math, not the situation, for example, and will not have the same level of motivation, due to a lack of personal stake in their decision.

In theory there's no difference between theory and practice, but in practice there is. This is because System 2 is very bad at intuitively predicting System 1's behavior, as we don't have a built-in reflective model of our own decision-making and motivation machinery. Thus, we don't know (and can't tell) how bad our theories are without comparing decision-making strategies across different people.

Comment author: Vladimir_Nesov 05 June 2009 09:49:20AM 1 point [-]

Consider, e.g. Newcomb's problem. If you do the calculation before, you should one-box. But doing the calculation at the actual time, means you should two-box.

This is incorrect. You are doing something very wrong if changing the time when you perform a calculation changes the result. That's an important issue in decision theory being reflectively consistent.

Comment author: pjeby 05 June 2009 03:01:24PM 1 point [-]

This is incorrect. You are doing something very wrong if changing the time when you perform a calculation changes the result. That's an important issue in decision theory being reflectively consistent.

That's the major point I'm making: that humans are NOT reflectively consistent without precommitment... and that the precommitment in question must be concretely specified, with the degree of concreteness and specificity required being proportional to the degree of "temptation" involved.

Comment author: Vladimir_Nesov 05 June 2009 03:47:25PM 1 point [-]

That may usually be the case, but this is not a law. Certain people could conceivably precommit to being reflectively consistent, to follow the results of calculations whenever the calculations are available.

Comment author: Cyan 05 June 2009 02:05:03AM *  1 point [-]

Newcomb's problem is a bad example to use here, because it depends on which math the person has committed to, e.g., Eliezer claims to have worked out a general analysis that justifies one-boxing...

They will be perceptually tuned to the math, not the situation, for example, and will not have the same level of motivation, due to a lack of personal stake in their decision.

The personal stake I envision is defending their concept of their own identity. "I will do this because that's the kind of person I am."

Comment author: pjeby 05 June 2009 02:52:24AM 1 point [-]

The personal stake I envision is defending their concept of their own identity. "I will do this because that's the kind of person I am."

Then their perception will be attuned to what kind of person they are, instead of the result. You can't cheat your brain - it tunes in on whatever you've decided your "territory" is, whatever you "own". This is not a generalized abstraction, but a concrete one.

You know how, once you buy a car, you start seeing that model everywhere? That's an example of the principle at work. Notice that it's not that you start noticing cars in general, you notice cars that look like yours. When you "own" a decision, you notice things specifically connected with that particular decision or goal, not "things that match a mathematical model of decision-making". The hardware just isn't built for that.

You also still seem to be ignoring the part where, if your decisions are made solely on the basis of any external data, then your decision is conditional and can change when the circumstances do, which is a bad idea if your real goal or intent is unconditional.

I've already mentioned how a conditional decision based on one's weight leads to stop-and-start dieting, but another good example is when somebody decides to start an exercise program when they're feeling well and happy, without considering what will happen on the days they're running late or feeling depressed. The default response in such cases may be to give up the previous decision, since the conditions it was made under "no longer apply".

What I'm saying is, it doesn't matter what conditions you base a decision on, if it is based solely on conditions, and not on actually going through the emotional decision process to un-conditionalize it, then you don't actually have a commitment to the course of action. You just have a conditional decision to engage in that course, until conditions change.

And the practical difference between a commitment and a conditional decision is huge, when it comes to one's personal and individual goals.

Comment author: Cyan 05 June 2009 04:32:36AM 2 points [-]

Thank you for this interesting discussion. Although I posed the "emotionally committed to math" case as a specific hypothetical, many of the things you've written in response apply more generally, so I've got a lot more material to incorporate into my understanding of the pjeby model of cognition. (I know that's a misnomer, but since you're my main source for this material, that's how I think of it.) I'm going to have to go over this exchange more thoroughly after I get some sleep.

Comment author: conchis 05 June 2009 03:53:58PM *  0 points [-]

Of course, there are presumably situations where one's decision should change with the conditions. (I do get that there's a trade-off between retaining the ability to change with the right conditions and opening yourself up to changing with the wrong conditions though.)

Comment author: taw 06 June 2009 12:46:13AM 1 point [-]

Most claims of incommensurability

I was pretty convinced for commensurability and thought cognitive biases would just introduce noise, but lack of success by me, and apparently by everyone else in this thread, changed my mind quite significantly.

Comment author: conchis 06 June 2009 01:03:34AM *  0 points [-]

Not knowing how to commensurate things doesn't imply they're incommensurable (though obviously, the fact that people have difficulty with this sort of thing is interesting in its own right).

As a (slight) aside, I'm still unclear about what you think would count as "success" here.

Comment author: taw 06 June 2009 03:15:54AM 0 points [-]

It's not a hard implication, but it's a pretty strong evidence against existence of traditional utility functions.

A success would be a list of events or states of reality and their weights, such that you're pretty convinced that your preferences are reasonably consistent with this list, so that you know how many hours of standing in queues is losing 5kg worth and how much money is having one thousand extra readers of your blog worth.

It doesn't sound like much, but I completely fail as soon as it goes out of very narrow domain, I'm surprised by this failure, and I'm surprised that others fail at this too.

Comment author: Cyan 06 June 2009 04:18:49AM *  0 points [-]

I'm surprised at your surprise. Even granting that humans could possibly be innately reflectively self-consistent, there's a huge curse of dimensionality problem in specifying the damn thing. ETA: The problem with the dimensionality is that interactions between the dimensions abound; ceteris paribus assumptions can't get you very far at all.

Comment author: taw 06 June 2009 04:24:18AM 0 points [-]

I was expecting noise, and maybe a few iterations before reaching satisfying results, but it seems we cannot even get that much, and it surprises me.

Comment author: billswift 04 June 2009 07:39:28AM 2 points [-]

This is similar to one problem Austrians have with conventional economics. They think the details of transactions are extremely important and that too much information is lost when they are aggregated in GDP and the like; more information than the weak utility of the aggregates can justify.

Comment author: timtyler 05 June 2009 05:10:46PM 0 points [-]

Re: Human utility functions are relative, contextual, and include semi-independent positive-negative axes. You can't model all that crap with one number.

That is not a coherent criticism of utilitarianism. Do you understand what it is that you are criticising?

Comment author: pjeby 05 June 2009 05:23:38PM 1 point [-]

That is not a coherent criticism of utilitarianism. Do you understand what it is that you are criticising?

Yes, I do... and it's not utilitarianism. ;-)

What I'm criticizing is the built-in System 2 motivation-comprehending model whose function is predicting the actions of others, but which usually fails when applied to self, because it doesn't model all of the relevant System 1 features.

If you try to build a human-values-friendly AI, or decide what would be of benefit to a person (or people), and you base it on System 2's model, you will get mistakes, because System 2's map of System 1 is flawed, in the same way that Newtonian physics is flawed for predicting near-light-speed mechanics: it leaves out important terms.

Comment author: loqi 04 June 2009 06:35:00AM 0 points [-]

Personally, this is where I think the idea of CEV has its biggest challenge: I know of no theoretical reason why humans must have convergent or consistent utility functions as individuals, let alone as a species.

It's been a while since I looked at CEV, but I thought the "coherent" part was meant to account for this. It assumes we have some relatively widespread, fairly unambiguous preferences, which may be easier to see in the light of that tired old example, paperclipping the light cone. If CEV outputs a null utility function, that would seem to imply that human preferences are completely symmetrically distributed, which seems hard to believe.

Comment author: pjeby 04 June 2009 06:53:50AM 1 point [-]

If CEV outputs a null utility function, that would seem to imply that human preferences are completely symmetrically distributed, which seems hard to believe.

If by "null utility function", you mean one that says, "don't DO anything", then do note that it would not require that we all have balanced preferences, depending on how you do the combination.

A global utility function that creates more pleasure for me by creating pain for you would probably not be very useful. Heck, a function that creates pleasure for me by creating pain for me might not be useful. Pain and pleasure are not readily subtractable from each other on real human hardware, and when one is required to subtract them by forces outside one's individual control, there is an additional disutility incurred.

These things being the case, a truly "Friendly" AI might well decide to limit itself to squashing unfriendly AIs and otherwise refusing to meddle in human affairs.

Comment author: loqi 04 June 2009 07:08:32AM 1 point [-]

These things being the case, a truly "Friendly" AI might well decide to limit itself to squashing unfriendly AIs and otherwise refusing to meddle in human affairs.

I wouldn't be particularly surprised by this outcome.

Comment author: SoullessAutomaton 04 June 2009 10:02:50AM 0 points [-]

Human utility functions are relative, contextual, and include semi-independent positive-negative axes. You can't model all that crap with one number.

Of course you can.

It just won't be a very good model.

What do you think would work better as a simplified model of utility, then? It seems you think that having orthogonal utility and disutility values would be a start.

Comment author: Vichy 04 June 2009 05:12:47PM 1 point [-]

"Utility functions are really bad match for human preferences, and one of the major premises we accept is wrong." Given the sheer messiness and mechanical influence involved in human brains, it's not even clear we have real 'values' which could be examined on a utility function, rather than simple dominant-interestedness that happens for largely unconscious and semi-arbitrary reasons.

Comment author: [deleted] 18 April 2012 02:03:08PM 0 points [-]

Suppose individuals have several incommensurable utility functions: would this present a problem for decision theory? If you were presented with Newcomb's problem, but were at the same time worried about accepting money you didn't earn, would these sorts of considerations have to be incorporated into a single algorithm?

If not, how do we understand such ethical concerns as being involved in decisions? If so, how do we incorporate such concerns?

Comment author: [deleted] 01 February 2012 09:36:39AM 0 points [-]

I think I see some other purpose to thinking that you have a numerically well-defined utility function. It's a pet theory of mine, but here we go:

It pays off to do reasoning with the "mathematical" reasoning. This "mathematical" reasoning is the one that kicks in when I ask you what 67 + 49 is, it is the thing that kicks in when i say "if x < y and y < z is x < z?" Even putting your decision problem into just a vague algebraic structure will let you reason comparatively about them, even if you cannot for the life of you assign any concrete values.

This is probably doubly true for a good understanding of Bayesian probability; you can assign a vague feeling of probability a letter, and another vague probability another letter and then, with mathematical reasoning ponder what to do in order to fulfil your vague sense of utility function.

I think I might write some serious articles about mental models and mathematical reasoning sometime.

Comment author: spriteless 08 June 2009 07:13:47PM 0 points [-]

Maximize the # of 'people' who can do 'much' with their lives.

Comment author: saturn 07 June 2009 10:38:26PM 0 points [-]

Some of the difficulty might be because the availability heuristic is causing us to focus on things which are relatively small factors in our global preferences, and ignore larger but more banal factors; e.g. being accepted within a social group, being treated with contempt, receiving positive and negative "strokes", demonstrating our moral superiority, etc.

Another problem is that although we seem to be finely attuned to small changes in social standing, as far as I know there have been no attempts to quantify this.

Comment author: prase 05 June 2009 12:43:05PM 0 points [-]

I vote for the first possibility - that utility functions are not particularly good match for human preferences, for following reasons: 1) I have never seen one, at least valid outside very narrow subject matter. That implies that people are not good at drawing these functions, which may be caused by the fact that these functions could in reality be very complicated, if even they exist. So even if my preferences are consistent with some utility function, any practical application would apply some strongly simplified model of the function, which could differ significantly from my real preferences. 2) As Roko has said, the utility function is defined rather on histories of universe than on its states. Since my preferences change (and I don't have general desire to keep all of them constant), I am not sure how to treat the time entanglement. 3) From a purely practical point of view, assigning numerical values to very improbable possibilities is prone to numerical errors.

For me some mix of deontology with utilitarianism works better than pure utilitarianism as real-life ethical and decision-making theory.

Comment author: Lightwave 04 June 2009 09:31:41AM 0 points [-]

Do people care more about money's absolute value, or more about its relative value to what other people have? Does our utility function have a term for other people in it which is it in conflict with other people's utility functions?

Comment author: taw 06 June 2009 12:43:36AM 0 points [-]

Even without bothering about rest of the world, just things that directly affect me, I totally failed, so I see no reason to think about such complications yet.

Comment author: Annoyance 04 June 2009 04:40:25PM 0 points [-]

Because everyone wants more money,

Why do people keep saying things like this? Intuition suggests, and research confirms, that there's a major diminishing returns factor involved with money, and acquiring lots of it can actually make people unhappy.

I want more money only to a degree, then I wouldn't want more. My utility function does not assign a positive, set value to money.

Comment author: conchis 04 June 2009 06:24:09PM *  4 points [-]

and acquiring lots of it can actually make people unhappy.

I was with you on diminishing returns, but that doesn't contradict the original claim. I haven't seen reliable research suggesting that more money actively makes people unhappy (in a broad-ish, ceteris paribus sense, so that e.g. having to work more to get it doesn't count as "money making you unhappy"). Could you point me to what you're referring to?

Comment author: taw 06 June 2009 12:42:22AM 0 points [-]

Theoretically you should be able to assign marginal values, like your cat getting sick is worth $X, good weather for your barbecue party is worth $Y and so on - these being marginal values. As long as the numbers are pretty small diminishing utilities shouldn't cause any problems.