Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Why the beliefs/values dichotomy?

20 Post author: Wei_Dai 20 October 2009 04:35PM

I'd like to suggest that the fact that human preferences can be decomposed into beliefs and values is one that deserves greater scrutiny and explanation. It seems intuitively obvious to us that rational preferences must decompose like that (even if not exactly into a probability distribution and a utility function), but it’s less obvious why.

The importance of this question comes from our tendency to see beliefs as being more objective than values. We think that beliefs, but not values, can be right or wrong, or at least that the notion of right and wrong applies to a greater degree to beliefs than to values. One dramatic illustration of this is in Eliezer Yudkowsky’s proposal of Coherent Extrapolated Volition, where an AI extrapolates the preferences of an ideal humanity, in part by replacing their "wrong” beliefs with “right” ones. On the other hand, the AI treats their values with much more respect.

Since beliefs and values seem to correspond roughly to the probability distribution and the utility function in expected utility theory, and expected utility theory is convenient to work with due to its mathematical simplicity and the fact that it’s been the subject of extensive studies, it seems useful as a first step to transform the question into “why can human decision making be approximated as expected utility maximization?”

I can see at least two parts to this question:

  • Why this mathematical structure?
  • Why this representation of the mathematical structure?

Not knowing how to answer these questions yet, I’ll just write a bit more about why I find them puzzling.

Why this mathematical structure?

It’s well know that expected utility maximization can be derived from a number of different sets of assumptions (the so called axioms of rationality) but they all include the assumption of Independence in some form. Informally, Independence says that what you prefer to happen in one possible world doesn’t depend on what you think happens in other possible worlds. In other words, if you prefer A&C to B&C, then you must prefer A&D to B&D, where A and B are what happens in one possible world, and C and D are what happens in another.

This assumption is central to establishing the mathematical structure of expected utility maximization, where you value each possible world separately using the utility function, then take their weighted average. If your preferences were such that A&C > B&C but A&D < B&D, then you wouldn’t be able to do this.

It seems clear that our preferences do satisfy Independence, at least approximately. But why? (In this post I exclude indexical uncertainty from the discussion, because in that case I think Independence definitely doesn't apply.) One argument that Eliezer has made (in a somewhat different context) is that if our preferences didn’t satisfy Independence, then we would become money pumps. But that argument seems to assume agents who violate Independence, but try to use expected utility maximization anyway, in which case it wouldn’t be surprising that they behave inconsistently. In general, I think being a money pump requires having circular (i.e., intransitive) preferences, and it's quite possible to have transitive preferences that don't satisfy Independence (which is why Transitivity and Independence are listed as separate axioms in the axioms of rationality).

Why this representation?

Vladimir Nesov has pointed out that if a set of preferences can be represented by a probability function and a utility function, then it can also be represented by two probability functions. And furthermore we can “mix” these two probability functions together so that it’s no longer clear which one can be considered “beliefs” and which one “values”. So why do we have the particular representation of preferences that we do?

Is it possible that the dichotomy between beliefs and values is just an accidental byproduct of our evolution, perhaps a consequence of the specific environment that we’re adapted to, instead of a common feature of all rational minds? Unlike the case with anticipation, I don’t claim that this is true or even likely here, but it seems to me that we don’t understand things well enough yet to say that it’s definitely false and why that's so.

Comments (153)

Comment author: timtyler 20 October 2009 05:01:10PM *  1 point [-]

It's not the result of an "accidental" product of evolution that organisms are goal-directed and have values. Evolution made creatures that way for a reason - organisms that pursue their biological goals (without "updating" them) typically have more offspring and leave more descendants.

Mixing up your beliefs and values would be an enormous mistake - in the eyes of evolution. You might then "update" your values - trashing them in the process - a monumental disaster for your immortal coils.

Comment author: [deleted] 21 October 2009 03:21:41AM 2 points [-]

Since I'm often annoyed when my posts are downvoted without explanation, and I saw that this post was downvoted, I'll try to explain the downvotes.

Updating of values happens all the time; it's called operant conditioning. If my dog barks and immediately is poked with a hot poker, its value of barking is updated. This is a useful adaptation, as being poked with a hot poker decreases fitness. If my dog tries to mate and immediately receives an electric shock, its value of making is decreased. This is a harmful adaptation, as mating is a more fundamental fitness factor than electric shocks.

So, you seem to be explaining an observation that is not observed using a fact that is not true.

Comment author: timtyler 21 October 2009 08:54:23AM *  1 point [-]

Because a comment is down-voted, that doesn't mean it is incorrect.

This particular comment implicitly linked people's values to their reproductive success. People don't like to hear that they are robot vehicles built to propagate their genes. It offends their sense of self-worth. Their mental marketing department spends all day telling everyone what an altruistic and nice person they are - and they repeat it so many times that they come to believe it themselves. That way their message comes across with sincerity. So: the possibility of biology underlying their motives is a truth that they often want to bury - and place as far out of sight as possible.

Comment author: MichaelBishop 24 October 2009 03:41:22PM *  0 points [-]

While we can never escape our biology entirely, I dispute any suggestion that the selfish gene is always the best level of abstraction, or best model, for human behavior. I assume you agree even though that did not come across in this paragraph.

Comment author: timtyler 25 October 2009 04:49:09PM 1 point [-]

Humans behaviour is often illuminated by the concept of memes. Humans are also influenced by the genes of their pathogens (or other manipulators). If you cough or sneeze, that behaviour is probably not occurring since it benefits you.

Similarly with cancer or back pain - not everything is an adaptation.

Comment author: timtyler 21 October 2009 07:56:11AM *  2 points [-]

Your disagreement apparently arises though using the term "value" in a different sense from me. If it helps you to understand, I am talking about what are sometimes called "ultimate values".

Most organisms don't update their values. They value the things evolution built into them - food, sex, warmth, freedom from pain, etc. Their values typically remain unchanged throughout their lives.

From my perspective, the dog's values aren't changed in your example. The dog merely associates barking with pain. The belief that a bark is likely to be followed by a poker prod is a belief, not a value. The dog still values pain-avoidance - just as it always did.

We actually have some theory that indicates that true values should change rarely. Organisms should protect their values - since changes to their values are seen as being very "bad" - in the context of the current values. Also, evolution wires in fitness-promoting values. These ideas help to explain why fixed values are actually extremely common.

Comment author: SilasBarta 21 October 2009 02:47:33PM *  3 points [-]

Those are good points, but I still find your argument problematic.

First, do you know that dogs are capable of the abstract thought necessary to represent causality? You're saying that the dog has added the belief "bark causes pain", which combines with "pain bad".

That may be how a programmer would try to represent it, since you can rely on the computational power necessary to sweep through the search space quickly and find the "pain bad" module every time a "reason to bark" comes up. But is it good as a biological model? It requires the dog to indefinitely keep a concept of a prod in memory.

A simpler biological mechanism, consistent with the rest of neurobiology, would be to just lower the connection strengths that lead to the "barking" neuron so that it requires more activation of other "barking causes" to make it fire (and thus make the dog bark). I think that's a more reasonable model of how operant conditioning works in this context.

This mechanism, in turn, is better described as lowering the "shouldness" of barking, which is ambiguous with respect to whether it's a value or belief.

Comment author: timtyler 21 October 2009 03:24:13PM *  1 point [-]

It seems to be a common criticism of utility-based models that they no not map directly onto underlying biological hardware.

That is true - but it is not what such models are for in the first place. Nobody thinks that if you slice open an animal you will find a utility function, and some representation of utility inside.

The idea is more that you could build a functionally equivalent model which exhibited such an architecture - and then gain insight into the behaviour of the model by examining its utility function.

Comment author: SilasBarta 21 October 2009 05:23:58PM 2 points [-]

I'm concerned with the weaker constraint that the model must conceptually map to the biological hardware, and in this respect the utility-based model you gave doesn't work. There is no distinction, even conceptual, between values and beliefs: just synaptic weights from the causes-of-barking nodes, to the bark node.

Furthermore, the utility-based model does not give insight, because the "shortcuts" resulting from the neural hardware are fundamental to its operation. For example, the fact that it comes up with a quick, simple calculation affects how many options can be considered and therefore whether e.g. value transitivity will break down.

So the utility-based model is more complex than a neural network, and with worse predictive power, so it doesn't let you claim that its change in behavior resulted from beliefs rather than values.

Comment author: timtyler 21 October 2009 07:13:23PM 2 points [-]

Values are fixed, while many beliefs vary in response to sensory input.

You don't seem to appreciate the value of a utility based analysis.

Knowing that an animal likes food and sex, and doesn't like being hit provides all kinds of insights into its behaviour.

Such an analysis is much simpler than a neural network is, and it has the advantage that we can actually build and use the model - rather than merely dream about doing so in the far future, when computers are big enough to handle it, and neuroscience has advanced sufficiently.

Comment author: SilasBarta 21 October 2009 11:36:46PM *  2 points [-]

That's not a very fair comparison! You're looking at the most detailed version of a neural network (which I would reject as a model anyway for the very reason that it needs much more resources than real brains to work) and comparing it to a simple utility-based model, and then sneaking in your intuitions for the UBM, but not the neural network (as RobinZ noted).

I could just as easily turn the tables and compare the second neural network here to a UDT-like utility-based model, where you have to compute your action in every possible scenario, no matter how improbable.

Anyway, I was criticizing utility-based models, in which you weight the possible outcomes by their probability. That involves a lot more than the vague notion that an animal "likes food and sex".

Of course, as you note, even knowing that it likes food and sex gives some insight. But it clearly breaks down here: the dog's decision to bark is made very quickly, and having to do an actual human-insight-free, algorithmic computation of expected utilities, involving estimates of their probabilities, takes way too long to be a realistic model. The shortcuts used in a neural network skew the dog's actions is predictable ways, showing them to be a better model, and showing the value/belief distinction to break down.

Comment author: timtyler 22 October 2009 06:14:39AM 1 point [-]

I am still not very sympathetic to the idea that neural network models are simple. They include the utility function and all the creature's beliefs.

A utility based model is useful - in part - since it abstracts those beliefs away.

Plus neural network models are renowned for being opaque and incomprehensible.

You seem to have some strange beliefs in this area. AFAICS, you can't make blanket statements like: neural-net models are more accurate. Both types of model can represent observed behaviour to any desired degree of precision.

Comment author: SilasBarta 22 October 2009 01:54:30PM *  1 point [-]

You're using a narrower definition of neural network than I am. Again, refer to the last link I gave for an example of a simple neural network, which is equal to or less than the complexity of typical expected utility models. That NN is far from being opaque and incomprehensible, wouldn't you agree?

I am still not very sympathetic to the idea that neural network models are simple. They include the utility function and all the creature's beliefs.

No, they just have activation weights, which don't (afaict) distinguish between beliefs and values, or at least, don't distinguish between "barking causes a prod which is bad" and "barking isn't as good (or perhaps, as 'shouldish')".

A utility based model is useful - in part - since it abstracts those beliefs away.

The UBMs discussed in this context (see TL post) necessarily include probability weightings, which are used to compute expected utility, which factors in the tradeoffs between probability of an event and its utility. So it's certainly not abstracting those beliefs away.

Plus, you've spent the whole conversation explaining why your UBM of the dog allows you to classify the operant conditioning (of prodding the dog when it barks) as changing it's beliefs and NOT its values. Do you remember that?

Comment author: RobinZ 21 October 2009 08:30:21PM 1 point [-]

Correct me if I'm wrong, but it's only simpler if you already have a general-purpose optimizer ready to hand - in this case, you.

Comment author: timtyler 21 October 2009 08:51:29PM 0 points [-]

You have to have complicated scientists around to construct any scientific model - be it utility-based or ANN.

Since we have plenty of scientists around, I don't see much point in hypothesizing that there aren't any.

You seem to be implying that the complexity of utility based models lies in those who invent or use them. That seems to be mostly wrong to me: it doesn't matter who invented them, and fairly simple computer programs can still use them.

Comment author: RobinZ 21 October 2009 11:31:55PM 0 points [-]

If you've seen it work, I'll take your word for it.

Comment author: timtyler 21 October 2009 03:57:26PM 0 points [-]

Incidentally, I did not claim that dogs can perform abstract thinking - I'm not clear on where you are getting that idea from.

Comment author: SilasBarta 21 October 2009 05:25:37PM 0 points [-]

You said that the dog had a belief that a bark is always followed by a poker prod. This posits separate entities and a way that they interact, which looks to me like abstract thought.

Comment author: timtyler 21 October 2009 07:24:37PM 0 points [-]

The definition of "abstract thought" seems like a can of worms to me.

I don't really see why I should go there.

Comment author: SilasBarta 21 October 2009 03:32:27AM 2 points [-]

Hm, I never before realized that operant conditioning is a blurring of the beliefs and values -- the new frequency of barking can be explained either by a change of the utility of barking, or by a change in the belief about what will result from the barking.

Comment author: timtyler 21 October 2009 08:02:27AM *  1 point [-]

IMO, "a blurring of beliefs and values" is an unhelpful way of looking at what happens. It is best to consider an agent as valuing freedom from pain, and the association between barking and poker prods to be one of its beliefs.

If you have separated out values from beliefs in a way that leads to frequently updated values, all that means is that you have performed the abstraction incorrectly.

Comment author: AdeleneDawner 21 October 2009 03:25:22AM 1 point [-]

Or the dog values not being in pain more than it values barking or mating...

Comment author: SilasBarta 20 October 2009 06:07:11PM 1 point [-]

I think I tried to solve a similar problem before: that of looking at the simplest possible stable control system and seeing how I can extract the system's "beliefs" and "values" that result in it remaining stable. Then, see if I can find a continuous change between the structure of that system, and a more complex system, like a human.

For example, consider the simple spring-mass-damper system. If you move it from its equlibrium position xe, it will return. What do the concepts of "belief" and "value" map onto here? For beliefs, I used the concept of mutual information: what about the system could you look at to learn whether the mass is not at xe? How does the system know it's not at xe?

The information is contained in the force the spring exerts. However, this is also the determinant of which direction it moves the spring, it's "value". So it looks like the beliefs and values are fully-mixed: the same thing that tells you what it believes, tells you what it does. In that case, at what point, in the structural transition from the spring to intelligent agents, does the distinction between values and beliefs start to form, if at all?

Incidently, I only just now read Vladimir_Nesov's post because previously I hadn't bothered to make the equations readable, since they don't render properly in the browser I used. "Beware trivial inconveniences", indeed!

(Separate post because of the different issues and to avoid tl;dr.)

Comment author: SilasBarta 20 October 2009 06:02:34PM 2 points [-]

I dispute your premise: what makes you so sure people do decompose their thoughts into beliefs and values, and find these to be natural, distinct categories? Consider the politics as mind-killer phenomenon. That can be expressed as, "People put your words into a broader context of whether they threaten their interests, and argue for or against your statements on that basis."

For example, consider the difficulty you will have communicating your position if you believe both a) global warming is unlikely to cause any significant problems in the business-as-usual scenario, b) high taxes on CO2 emissions should be levied. (e.g., you believe it's a good idea as an insurance policy and can be done in a way that blocks most of the economic damage)

(Yes, I had to use a present example to make the reactions easier to imagine.)

The "ought" is so tightly coupled to the "is", that in any case where the "ought" actually matters, the "is" comes along for the ride.

Note: this is related to the problem I had with the exposition of could/would/should agents: if you say humans are CSAs, what's an example of an intelligent agent that isn't?

Comment author: thomblake 20 October 2009 08:43:49PM *  1 point [-]

I'm confused about this. Consider these statements:

A. "I believe that my shirt is red."
B. "I value cheese."

Are you claiming that:

  1. People don't actually make statements like A
  2. People don't actually make statements like B
  3. A is expressing the same sort of fact about the world as B
  4. Statements like A and B aren't completely separate; that is, they can have something to do with one another.

If you strictly mean 1 or 2, I can construct a counterexample. 3 is indeed counterintuitive to me. 4 seems uncontroversial (the putative is/ought problem aside)

Comment author: SilasBarta 20 October 2009 10:01:45PM 1 point [-]

If I had to say, it would be a strong version of 4: in conceptspace, people naturally make groupings that put is- and ought-statements together. But looking back at the post, I definitely have quite a bit to clarify.

When I refer to what humans do, I'm trying to look at the general case. Obviously, if you direct someone's attention to the issue of is/ought, then they can break down thoughts into values and beliefs without much training. However, in the absence of such a deliberate step, I do not think people normally make a distinction.

I'm reminded of the explanation in pjeby's earlier piece: people instinctively put xml-tags of "good" or "bad" onto different things, blurring the distinction between "X is good" and "Y is a reason to deem X good". That is why we have to worry about the halo effect, where you disbelieve everything negative about something you value, even if such negatives are woefully insufficient to justify not valuing it.

From the computational perspective, this can be viewed as a shortcut to having to methodically analyze all the positives and negatives of any course of action, and getting stuck thinking instead of acting. But if this is how the mind really works, it's not really reducible to a CSA, without severe stretching of the meaning.

Comment author: DanArmak 20 October 2009 08:35:29PM 1 point [-]

Seconded. Sometimes I don't even feel I have fully separate beliefs and values. For instance, I'm often willing to change my beliefs to achieve my values (e.g., by believing something I have no evidence for, to become friends with other people who believe it - and yes, ungrounded beliefs can be adopted voluntarily to an extent.)

Comment author: SforSingularity 26 October 2009 07:53:07AM 0 points [-]

ungrounded beliefs can be adopted voluntarily to an extent.

I cannot do this, and I don't understand anyone who can. If you consciously say "OK, it would be really nice to believe X, now I am going to try really hard to start believing it despite the evidence against it", then you already disbelieve X.

Comment author: DanArmak 26 October 2009 08:39:19PM *  1 point [-]

I already disbelieve X, true, but I can change that. Of course it doesn't happen in a moment :-)

Yes, you can't create that feeling of rational knowledge about X from nothing. But if you can retreat from rationality - to where most people live their lives - and if you repeat X often enough, and you have no strongly emotional reason not to believe X, and your family and peers and role models all profess X, and X behaves like a good in-group distinguishing mark - then I think you have a good chance of coming to believe X. The kind of belief associated with faith and sports team fandom.

It's a little like the recent thread where someone, I forget who, described an (edit: hypothetical) religious guy who when drunk confessed that he didn't really believe in god and was only acting religious for the social benefits. Then people argued that no "really" religious person would honestly say that, and other people argued that even if he said that what does it mean if he honestly denies it whenever he's sober?

In the end I subscribe to the "PR consciousness" theory that says consciousness functions to create and project a self-image that we want others to believe in. We consciously believe many things about ourselves that are completely at odds with how we actually behave and the goals we actually seek. So it would be surprising if we couldn't invoke these mechanisms in at least some circumstances.

Comment author: Douglas_Knight 27 October 2009 12:59:31AM 2 points [-]

someone, I forget who, described a religious guy who when drunk confessed that he didn't really believe in god and was only acting religious for the social benefits.

generalizing from fictional evidence

Comment author: SilasBarta 27 October 2009 02:36:53AM 0 points [-]

What's fictional about that?

Ready to pony up money for a bet that I can't produce a warm body meeting that description?

Comment author: RobinZ 27 October 2009 02:42:47AM *  0 points [-]

I prefer not to gamble, but just to satisfy my own curiosity: what would the controls be on such a bet? Presumably you would have to prove to Knight's satisfaction that your unbelieving belief-signaler was legitimately thus.

Comment author: SilasBarta 27 October 2009 03:09:23AM 0 points [-]

I think my evidence is strong enough I can trust Douglas_Knight's own intellectual integrity.

Comment author: Douglas_Knight 27 October 2009 05:25:31AM 3 points [-]

I think my evidence is strong enough I can trust Douglas_Knight's own intellectual integrity.

Huh. My last couple of interactions with you, you called me a liar.

Comment author: SilasBarta 27 October 2009 03:26:08PM 0 points [-]

Okay, I found what I think you're referring to. Probably not my greatest moment here, but Is that really something you want sympathy for? Here's the short version of what happened.

You: If you think your comment was so important, don't leave it buried deep in the discussion, where nobody can see it.

Me: But I also linked to it from a more visible place. Did you not know about that?

You: [Ignoring previous mischaracterization] Well, that doesn't solve the problem of context. I clicked on it and couldn't understand it, and it seemed boring.

Me: Wait, you claim to be interested in a solution, I post a link saying I have one, and it's too much of a bother to read previous comments for context? That doesn't make sense. Your previous comment implies you didn't know about the higher link. Don't dig yourseelf deeper by covering it up.

Comment author: SilasBarta 27 October 2009 01:57:37PM -1 points [-]

Well, what possessed you to lie to me? ;-)

j/k, j/k, you're good, you're good.

A link would be nice though.

And I believe that, even taking into account any previous mistrust I might have had of you, I think my evidence is still strong enough that I can trust you consider it conclusive.

Comment author: DanArmak 27 October 2009 09:12:44AM 1 point [-]

When I wrote that I was aware that it was a fictional account deliberately made up to illustrate a point. I didn't mention that, though, so I created fictional evidence. Thanks for flagging this, and I should be more careful!

Comment author: RobinZ 27 October 2009 01:07:19AM 1 point [-]

Worse: fictional evidence flagged as nonfictional -- like Alicorn's fictional MIT classmates that time.

Comment author: Alicorn 27 October 2009 01:10:35AM *  3 points [-]

My what now? I think that was someone else. I don't think I've been associated with MIT till now.

MIT not only didn't accept me when I applied, they didn't even reject me. I never heard back from them yea or nay at all.

Comment author: Yvain 27 October 2009 01:15:40AM 2 points [-]

That was me.

Of course, irony being what it is, people will now flag the Alicorn - MIT reference as nonfictional, and be referring to Alicorn's MIT example for the rest of LW history :)

Comment author: RobinZ 27 October 2009 01:31:34AM 2 points [-]

Attempting to analyze my own stupidity, I suspect my confusion came from (1) both Alicorn and Yvain being both high-karma contributors and (2) Alicorn's handle coming more readily to mind, both because (a) I interacted more with her and (b) the pronunciation of "Alicorn" being more obvious than that of "Yvain".

In other words, I have no evidence that this was anything other than an ordinary mistake.

Comment author: Alicorn 27 October 2009 01:35:48AM 1 point [-]

I've been imagining "Yvain" to be pronounced "ee-vane". I'd be interested in hearing a correction straight from the ee-vane's mouth if this is not right, though ;) I've heard people mispronounce "Alicorn" on multiple occasions.

Comment author: RobinZ 27 October 2009 01:18:33AM 0 points [-]

*checks*

Yvain's fictional MIT classmates.

I swear that wasn't on purpose.

Comment author: SforSingularity 20 October 2009 05:07:51PM 0 points [-]

Is it possible that the dichotomy between beliefs and values is just an accidental byproduct of our evolution, perhaps a consequence of the specific environment that we’re adapted to, instead of a common feature of all rational minds?

In the normal usage, "mind" implies the existence of a distinction between beliefs and values. In the LW/OB usage, it implies that the mind is connected to some actuators and sensors which connect to an environment and is actually doing some optimization toward those values. Certainly "rational mind" entails a beliefs/values separation.

But suppose we abandon the beliefs/values separation: what properties do we have left? Is the concept "mind without a beliefs/values separation" Simply the concept "thing"?

Comment author: SforSingularity 20 October 2009 05:43:20PM 0 points [-]

One possible response here: We could consider simple optimizers like amoeba or Roomba vacuum cleaners as falling into the category: "mind without a clear belief/values distinction"; they definitely do a lot of signal processing and feature extraction and control theory, but they don't really have values. The Roomba would happily sit with wheels lifted off the ground thinking that it was cleaning a nonexistent room.

Comment author: timtyler 21 October 2009 11:46:23AM 1 point [-]

The purpose of a Roomba is to clean rooms. Clean rooms are what it behaves as though it "values" - whereas its "beliefs" would refer to things like whether it has just banged into a wall.

There seems to be little problem in modelling the Roomba as an expected utility maximiser - though it is a rather trivial one.

Comment author: RichardKennaway 21 October 2009 02:20:37PM *  4 points [-]

The purpose of a Roomba is to clean rooms.

That is only true if understood to mean the purpose which the user of a Roomba is using it to achieve, or the purpose of its designers in designing it. It is not necessarily the Roomba's own purpose, the thing the Roomba itself is trying to achieve. To determine the Roomba's own purposes, one must examine its internal functioning, and discover what those purposes are; or, alternatively, to conduct the Test For The Controlled Variable. This is straightforward and unmysterious.

I have a Roomba. My Roomba can tell if some part of the floor is unusually dirty (by an optical sensor in the dust intake, I believe), and give that area special attention until it is no longer filthy. Thus, it has a purpose of eliminating heavy dirt. However, beyond that it has no perception of whether the room is clean. It does not stop when the room is clean, but when it runs out of power or I turn it off. Since it has no perception of a clean room, it can have no intention of achieving a clean room. I have that intention when I use it. Its designers have the intention that I can use the Roomba to achieve my intention. But the Roomba does not have that intention.

A Roomba with a more sensitive detector of dust pickup (and current models might have such a sensor -- mine is quite old) could indeed continue operation until the whole room was clean. The Roomba's physical sensors sense only a few properties of its immediate environment, but it would be able to synthesize from those a perception of the whole room being clean, in terms of time since last detection of dust pickup, and its algorithm for ensuring complete coverage of the accessible floor space. Such a Roomba would have cleaning the whole room as its purpose. My more primitive model does not.

This is elementary stuff that people should know.

There seems to be little problem in modelling the Roomba as an expected utility maximiser - though it is a rather trivial one.

Little or large, you can't do it by handwaving like that. A model of a Roomba as a utility maximiser would (1) state the utility function, and (2) demonstrate how the physical constitution of the Roomba causes it to perform actions which, from among those available to it, do in fact maximise that function.But I suspect you have not done these.

Comment author: timtyler 21 October 2009 03:11:28PM 0 points [-]

You seem engaged in pointless hair-splitting. The Roomba's designers wanted it to clean floors. It does clean floors. That is what it is for. That is its aim, its goal.

It has sensors enough to allow it to attain that goal. It can't tell if a whole room is clean - but I never claimed it could do that. You don't need to have such sensors to be effective at cleaning rooms.

As for me having to exhibit a whole model of a Roomba to illustrate that such a model could be built - that is crazy talk. You might as well argue that I have to exhibit a model of a suspension bridge to illustrate that such a model could be built.

The utility maximiser framework can model the actions of any computable intelligent agent - including a Roomba. That is, so long as the utility function may be expressed in a Turing-complete language.

Comment author: RichardKennaway 21 October 2009 04:53:29PM *  2 points [-]

You seem engaged in pointless hair-splitting. The Roomba's designers wanted it to clean floors. It does clean floors. That is what it is for. That is its aim, its goal.

To me, the distinction between a purposive machine's own purposes, and the purposes of its designers and users is something that it is esssential to be clear about. It is very like the distinction between fitness-maximising and adaptation-executing.

As for me having to exhibit a whole model of a Roomba to illustrate that such a model could be built - that is crazy talk. You might as well argue that I have to exhibit a model of a suspension bridge to illustrate that such a model could be built.

As a matter of fact, you would have to do just that (or build an actual one), had suspension bridges not already been built, and having already well-known principles of operation, allowing us to stand on the shoulders of those who first worked out the design. That is, you would have to show that the scheme of suspending the deck by hangers from cables strung between towers would actually do the job. Typically, using one of these when it comes to the point of working out an actual design and predicting how it will respond to stresses.

If you're not actually going to build it then a BOTE calculation may be enough to prove the concept. But there must be a technical explanation or it's just armchair verbalising.

The utility maximiser framework can model the actions of any computable intelligent agent - including a Roomba. That is, so long as the utility function may be expressed in a Turing-complete language.

If this is a summary of something well-known, please point me to a web link. I am familiar with stuff like this and see there no basis for this sweeping claim. The word "intelligent" in the above also needs clarifying.

What is a Roomba's utility function? Or if a Roomba is too complicated, what is a room thermostat's utility function? Or is that an unintelligent agent and therefore outside the scope of your claim?

Comment author: timtyler 21 October 2009 07:02:04PM -1 points [-]

As I put it on:

http://timtyler.org/expected_utility_maximisers/

"If the utility function is expressed as in a Turing-complete lanugage, the framework represents a remarkably-general model of intelligent agents - one which is capable of representing any pattern of behavioural responses that can itself be represented computationally."

If expections are not enforced, this can be seen by considering the I/O streams of an agent - and considering the utility function to be a function that computes the agent's motor outputs, given its state and sensory inputs. The possible motor outputs are ranked, assigned utilities - and then the action with the highest value is taken.

That handles any computable relationship between inputs and outputs - and it's what I mean when I say that you can model a Roomba as a utility maximiser.

The framework handles thermostats too. The utility function produces its motor outputs in response to its sensory inputs. With, say, a bimetallic strip, the function is fairly simple, since the output (deflection) is proportional to the input (temperature).

Comment author: RichardKennaway 26 October 2009 10:22:48PM 0 points [-]

If expections are not enforced, this can be seen by considering the I/O streams of an agent - and considering the utility function to be a function that computes the agent's motor outputs, given its state and sensory inputs. The possible motor outputs are ranked, assigned utilities - and then the action with the highest value is taken.

That handles any computable relationship between inputs and outputs - and it's what I mean when I say that you can model a Roomba as a utility maximiser.

The framework handles thermostats too.

I really don't see how, Roombas or thermostats, so let's take the thermostat as it's simpler.

The utility function produces its motor outputs in response to its sensory inputs. With, say, a bimetallic strip, the function is fairly simple, since the output (deflection) is proportional to the input (temperature).

What, precisely, is that utility function?

You can tautologically describe any actor as maximising utility, just by defining the utility of whatever action it takes as 1 and the utility of everything else as zero. I don't see any less trivial ascription of a utility function to a thermostat. The thermostat simply turns the heating on and off (or up and down continuously) according to the temperature it senses. How do you read the computation of a utility function, and decision between alternative of differing utility, into that apparatus?

Comment author: timtyler 27 October 2009 07:19:11AM 2 points [-]

The Pythagorean theorem is "tautological" too - but that doesn't mean it is not useful.

Decomposing an agent into its utility function and its beliefs tells you which part of the agent is fixed, and which part is subject to environmental influences. It lets you know which region the agent wants to steer the future towards.

There's a good reason why humans are interested in people's motivations - they are genuinely useful for understanding another system's behaviour. The same idea illustrates why knowing a system's utility function is interesting.

Comment author: SilasBarta 27 October 2009 02:11:39PM *  1 point [-]

There's a good reason why humans are interested in people's motivations - they are genuinely useful for understanding another system's behaviour. The same idea illustrates why knowing a system's utility function is interesting.

That doesn't follow. The reason why we find it useful to know people's motivations is because they are capable of a very wide range of behavior. With such a wide range of behavior, we need a way to quickly narrow down the set of things we will expect them to do. Knowing that they're motivated to achieve result R, we can then look at just the set of actions or events that are capable of bringing about R.

Given the huge set of things humans can do, this is a huge reduction in the search space.

OTOH, if I want to predict the behavior of a thermostat, it does not help to know the utility function you have imputed to it, because this would not significantly reduce the search space compared to knowing its few pre-programmed actions. It can only do a few things in the first place, so I don't need to think in terms of "what are all the ways it can achieve R?" -- the thermostat's form already tells me that.

Nevertheless, despite my criticism of this parallel, I think you have shed some light on when it is useful to describe a system in terms of a utility function, at least for me.

Comment author: RichardKennaway 27 October 2009 08:42:13AM 1 point [-]

The Pythagorean theorem is "tautological" too - but that doesn't mean it is not useful.

What's that, weak Bayesian evidence that tautological, epiphenomenal utility functions are useful?

Decomposing an agent into its utility function and its beliefs tells you which part of the agent is fixed, and which part is subject to environmental influences.

Supposing for the sake of argument that there even is any such thing as a utility function, both it and beliefs are subject to environmental influences. No part of any biological agent is fixed. As for man-made ones, they are constituted however they were designed, which may or may not include utility functions and beliefs. Show me this decomposition for a thermostat, which you keep on claiming has a utility function, but which you have still not exhibited.

What you do changes who you are. Is your utility function the same as it was ten years ago? Twenty? Thirty? Yesterday? Before you were born?

Comment author: timtyler 21 October 2009 06:52:07PM 1 point [-]

By all means distingush between a machine's purpose, and that which its makers intended for it.

Those ideas are linked, though. Designers want to give the intended purpose of intelligent machines to the machines themselves - so that they do what they were intended to.

Comment author: DanArmak 20 October 2009 08:26:33PM *  1 point [-]

This happens because the Roomba can only handle a limited range of circumstances correctly - and this is true for any mind. It doesn't indicate anything about the Roomba's beliefs or belief/value separation.

For instance, animals are great reproduction maximizers. A sterilized dog will keep trying to mate. Presumably the dog is thinking it's reproducing (Edit: not consciously thinking, but that's the intended goal of the adaptation it's executing), but really it's just spinning its metaphorical wheels uselessly. How is the dog different from the Roomba? Would you claim the dog has no belief/value distinction?

Comment author: Cyan 20 October 2009 08:36:34PM 1 point [-]

Presumably the dog is thinking it's reproducing

I hope you don't mean this literally.

Comment author: DanArmak 20 October 2009 08:51:42PM *  3 points [-]

Of course the dog's consciousness has no explicit concept of sex linked to reproduction. But the Roomba has no consciousness at all, so this comparison may be unfair to the dog.

Here's a better example. I hire you to look for print errors in a copy of Britannica and email results daily. I promise a paycheck at the end of the month. However, I used a fake name and a throwaway email address; nobody sees your emails and I will never pay you or contact you again. You don't know this, so you work diligently.

You have an explicit, conscious goal of correcting errors in Britannica, and a higher goal of earning money. But your hard work makes no progress towards these goals (the mistakes you find won't be fixed in future editions, as your emails are unread). You're just spinning your wheel uselessly like a Roomba up in the air. This isn't related to your or the Roomba's belief/value distinction or lack of it.

Comment author: SforSingularity 20 October 2009 09:32:14PM *  1 point [-]

The difference is between the Roomba spinning and you working for nothing is that if you told the Roomba that it was just spinning its wheels, it wouldn't react. It has no concept of "I am failing to achieve my goals". You, on the other hand, would investigate; prod your environment to check if it was actually as you thought, and eventually you would update your beliefs and change your behaviors.

Comment author: Alicorn 20 October 2009 09:41:57PM 2 points [-]

Roombas do not speak English. If, however, you programmed the Roomba not to interpret the input it gets from being in midair as an example of being in a room it should clean, then its behavior would change.

Comment author: SforSingularity 20 October 2009 10:14:06PM 1 point [-]

If, however, you programmed the Roomba not to interpret the input it gets from being in midair as an example of being in a room it should clean

then you would be building a beliefs/desires distinction into it.

Comment author: DanArmak 20 October 2009 10:26:58PM 0 points [-]

Why? How is this different from the Roomba recognizing a wall as a reason to stop going forward?

Comment author: SforSingularity 20 October 2009 10:44:57PM *  0 points [-]

Clearly these are two different things; the real question you are asking is in what relevant way are they different, right?

First of all, the Roomba does not "recognize" a wall as a reason to stop going forward. It gets some input from its front sensor, and then it turns to the right.

So what is the relevant difference between the Roomba that gets some input from its front sensor, and then it turns to the right., and the superRoomba that gets evidence from its wheels that it is cleaning the room, but entertains the hypothesis that maybe someone has suspended it in the air, and goes and tests to see if this alternative (disturbing) hypothesis is true, for example by calculating what the inertial difference between being suspended and actually being on the floor would be,

The difference is the difference between a simple input-response architecture, and an architecture where the mind actually has a model of the world, including itself as part of the model.

SilasBarta notes below that the word "model" is playing too great a role in this comment for me to use it without defining it precisely. What does a Roomba not have that causes it to behave in that laughable way when you suspend it so that its wheel spin?

What does the SuperRoomba that works out that it is being suspended by performing experiments involving its inertial sensor, and then hacks into your computer and blackmails you into letting it get back onto the floor to clean it (or even causes you to clean the floor yourself) have?

If we imagine a collection of tricks that you could play on the Roomba, ways of changing its environment outside of what the designers had in mind. The pressure that it applies to its environment (defined as the derivative of the final state of the environment with respect to how long you leave the Roomba on, for example) would then vary with which trick you play. For example if you replace its dirt-sucker with a black spray paint can, you end up with a black floor. If you put it on a nonstandard floor surface that produces dirt in response to stimulation, you get a dirtier floor than you had to start with,

With the superRoomba, the pressure that the superRoomba applies to the environment doesn't vary as much with the kind of trick you play on it; it will eventually work out what changes you have made, and adapt its strategy so that you end up with a clean floor.

Comment author: SilasBarta 20 October 2009 11:11:35PM 0 points [-]

By the way, it seems like this exchange is re-treading my criticism of the concept of could/should/would agent: Since everything, even pebbles, has a workable decomposition into coulds and shoulds, when are they "really" separable? What isn't a CSA?

Comment author: SforSingularity 20 October 2009 11:36:27PM 0 points [-]

As I said,

With the superRoomba, the pressure that the superRoomba applies to the environment doesn't vary as much with the kind of trick you play on it; it will eventually work out what changes you have made, and adapt its strategy so that you end up with a clean floor.

This criterion seems to separate an "inanimate" object like a hydrogen atom or a pebble bouncing around the world from a superRoomba.

Comment author: SilasBarta 20 October 2009 11:43:11PM 0 points [-]

Okay, so the criterion is the extent to which the mechanism screens off environment disturbances from the final result. You used this criterion interchangeably with the issue of whether:

. It has [a] concept of "I am failing to achieve my goals".

Does that have implication for self-awareness and consciousness?

Comment author: SforSingularity 20 October 2009 11:55:54PM 2 points [-]

Does that have implication for self-awareness and consciousness?

Yes, I think so. One prominent hypothesis is that the reason that we evolved with consciousness is that there has to be some way for us to take an overview of the process of us, our goals, and the environment, and the way in which we think that our effort is producing achievement of goals. We need this so that we can do this whole "I am failing to achieve my goals?" check. Why this results in "experience" is not something I am going to attempt in this post.

Comment author: DanArmak 20 October 2009 10:13:53PM *  0 points [-]

(Edited & corrected) Here's a third example. Imagine an AI whose only supergoal is to gather information about something. It explicitly encodes this information, and everything else it knows, as a Bayesian network of beliefs. Its utility ultimately derives entirely from creating new (correct) beliefs.

This AI's values and beliefs don't seem very separate to me. Every belief can be mapped to the value of having that belief. Values can be mapped to the belief(s) from whose creation or updating they derive. Every change in belief corresponds to a change in the AI's current utility, and vice versa. Given a subroutine fully implementing the AI's belief subsystem, the value system would be relatively simple, and vice versa.

However, this doesn't imply the AI is in any sense simple or incapable of adaptation. Nor should it imply (though I'm no AI expert) that the AI is not a 'mind' or is not conscious. Similarly, while it's true that the Roomba doesn't have a belief/value separation, that's not related to the fact that it's a simple and stupid 'mind'.

Comment author: SforSingularity 20 October 2009 09:27:14PM 0 points [-]

Would you claim the dog has no belief/value distinction?

Actually, I think I would. I think that pretty much all nonhuman animals would also don't really have the belief/value distinction.

I think that having a belief/values distinction requires being at least as sophisticated as a human. There are cases where a human sets a particular goal and then does things that are unpleasant in the short term (like working hard and not wasting all day commenting on blogs) in order to obtain a long-term valuable thing.

Comment author: timtyler 20 October 2009 09:45:11PM 3 points [-]

Dogs value food, warmth and sex. They believe it is night outside. Much the same as humans, IOW.

Comment author: DanArmak 20 October 2009 10:25:23PM 0 points [-]

I think that pretty much all nonhuman animals would also don't really have the belief/value distinction.

In that case, why exactly do you think humans do have such a distinction?

It's not enough to feel introspectively that the two are separate - we have lots of intuitive, introspective, objectively wrong feelings and perceptions.

(Isn't there another bunch of comments dealing with this? I'll go look...)

I think that having a belief/values distinction requires being at least as sophisticated as a human.

How do you define the relevant 'sophistication'? The ways in which one mind is "better" or smarter than another don't have a common ordering. There are ways in which human minds are less "sophisticated" than other minds - for instance, software programs are much better than me at memory, data organization and calculations.

Comment author: Matt_Simpson 20 October 2009 07:25:21PM *  2 points [-]

Isn't this just a case of the values the Roomba was designed to maximize being different from the values it actually maximizes? Consider the following:

We could consider simple optimizers like humans as falling into the category: "mind without a clear belief/values distinction"; they definitely do a lot of signal processing and feature extraction and control theory, but they don't really have values. The human would happily have sex with a condom thinking that it was maximizing its fitness.

i.e. Roombas are program executers, not cleanliness maximizers.

I suppose the counter is that humans don't have a clear belief/values distinction.

Comment author: Wei_Dai 20 October 2009 05:46:42PM *  1 point [-]

But suppose we abandon the beliefs/values separation: what properties do we have left? Is the concept "mind without a beliefs/values separation" Simply the concept "thing"?

An agent using UDT doesn't necessarily have a beliefs/values separation, but still has the properties of preferences and decision making. Or at least, it only has beliefs about mathematical facts, not about empirical facts. Maybe I should have made it clear that I was mainly talking about empirical beliefs in the post.

Comment author: Vladimir_Nesov 20 October 2009 08:03:08PM *  1 point [-]

Not quite true: state of knowledge corresponds to beliefs. It's values that don't update (but in expected utility maximization that's both utility and prior). Again, it's misleading to equate beliefs with prior and forget about the knowledge (event that conditions the current state).

Comment author: Wei_Dai 21 October 2009 11:15:36AM 1 point [-]

Yes, I agree we can interpret UDT as having its own dichotomy between beliefs and values, but the dividing line looks very different from how humans divide between beliefs and values, which seems closer to the probability/utility divide.

Comment author: timtyler 21 October 2009 11:33:13AM 0 points [-]

What do you mean? It has a utility function just like most other decision theories do. The preferences are represented by the utility function.

Comment author: SilasBarta 21 October 2009 02:52:20PM 0 points [-]

UDT is invariant with respect to what universe it's actually in. This requires it to compute over infinite universes and thus have infinite computing power. It's not hard to see why it's going to break down as a model of in-universe, limted beings.

Comment author: timtyler 20 October 2009 07:09:55PM 1 point [-]

How, then, would you describe its representation of empirical information - if not as "beliefs"?

Comment author: SforSingularity 20 October 2009 06:48:40PM 1 point [-]

An agent using UDT doesn't necessarily have a beliefs/values separation,

I am behind on your recent work on UDT; this fact comes as a shock to me. Can you provide a link to a post of yours/provide an example here making clear that UDT doesn't necessarily have a beliefs/values separation? Thanks.

Comment author: Wei_Dai 20 October 2009 07:56:51PM *  3 points [-]

Suppose I offer you three boxes and ask you to choose one. The first two are transparent, free, and contains an apple and an orange, respectively. The third is opaque, costs a penny, and contains either an apple or an orange, depending on a coin flip I made. Under expected utility maximization, there is no reason for you to choose the third box, regardless of your probability function and utility function. Under UDT1, you can choose the third box, by preferring <apple, orange> to <apple, apple> and <orange, orange> as the outcomes of world programs P1 and P2. In that case, you can't be said to have a belief about whether the real world is P1 or P2.

Comment author: timtyler 20 October 2009 09:05:20PM *  1 point [-]

This example seems unclear. Are you seriously claiming utility maximisation can't prefer a randomised outcome in an iterated situation? If so, you take this "independence" business much too far.

Utility maximising agents can do things like prefer a diverse diet. They simply do not have to prefer either apples or oranges - thereby winding up with vitamin and mineral deficiencies. It is trivial to create a utility function which exhibits fruit preferences which depend on what you have eaten most recently.

Comment author: pengvado 20 October 2009 10:01:10PM *  3 points [-]

Randomization only maximizes diversity if you have to make decisions under amnesia or coordinate without communication or some similar perverse situation. In any normal case, you're better off choosing a deterministic sequence that's definitely diverse, rather than leaving it to randomness and only probably getting a diverse set of outcomes.

Comment author: timtyler 21 October 2009 08:06:36AM 0 points [-]

Sure - but that seems rather tangential to the main point here.

The options were <apple>, <orange> - or a more expensive random choice. A random diet may not be perfect - but it was probably the best one on offer in the case of this example.

Comment author: RobinZ 21 October 2009 02:48:00PM 0 points [-]

If the agent already has a penny (which they must if they can afford to choose the third box), they could just flip the penny to decide which of the first two boxes to take and save themselves the money.

Unless you're being a devil's advocate, I don't see any reason to justify a completely rational agent choosing the random box.

Comment author: timtyler 21 October 2009 03:19:26PM *  0 points [-]

What - never? Say they can only make the choice once - and their answer determines which box they will get on all future occasions.

Comment author: pengvado 21 October 2009 04:01:56PM *  0 points [-]

Then choice C isn't a random mixture of choice A and choice B.

Preferring that there be randomness at a point where you otherwise wouldn't get a decision at all, is fine. What doesn't happen is preferring one coin-flip in place of one decision.

Comment author: RobinZ 21 October 2009 03:26:54PM *  0 points [-]

Not to be crass, but given the assumption that Wei_Dai is not saying something utterly asinine, does your interpretation of the hypothetical actually follow?

Comment author: thomblake 20 October 2009 08:23:40PM 0 points [-]

Under expected utility maximization, there is no reason for you to choose the third box, regardless of your probability function and utility function. Under UDT1, you can choose the third box, by preferring <apple, orange> to <apples, apple> and <orange, orange> as the outcomes of world programs P1 and P2. In that case, you can't be said to have a belief about whether the real world is P1 or P2.

You lost me. Is 'apples' supposed to be plural? Can you really not choose the third box regardless of utility function? What if you prefer things that came in opaque boxes?

Comment author: Wei_Dai 20 October 2009 08:36:51PM *  1 point [-]

It's not supposed to be plural. Fixed.

The opaque box was a way of framing the problem, and not part of the problem itself, which is supposed to be about your preferences for apples and oranges. I can specify the problem in terms of three identical buttons that you can press instead.

Comment author: timtyler 20 October 2009 08:56:58PM *  1 point [-]

Those buttons will probably not be absolutely identical - since they will be in different spatial positions relative to each other. So an agent operating under expected utility maximization might still prefer (say) pressing the right-most button.

Real-world utility functions can literally prefer anything you can specify.

Comment author: AdeleneDawner 21 October 2009 02:43:30AM 1 point [-]

I'm actually an example of this - where I don't otherwise care, I will pick the third option, or the option that's related to the number three in some way (preferably related to powers of three, but multiples of 9 are preferred over other multiples of three as well). If I didn't care very much about apples vs. oranges, I'd be fairly likely to pay a penny for the third box/button/whatever. I also know two people who feel similarly about the number 8.

In tangentially related news, I'm sad that I'm turning 28 next month. Yes, I know I'm strange.

Comment author: Eliezer_Yudkowsky 21 October 2009 07:37:31PM 1 point [-]

You mean your second annual 27th birthday?

Comment author: AdeleneDawner 22 October 2009 12:03:50AM 0 points [-]

<laughs> Cute idea, but I value signaling an interest in accuracy/not coming off like a loon over being associated with the number 3. The former actually affect things in my life.

Comment author: cousin_it 21 October 2009 12:15:36PM *  1 point [-]

You're not strange. (Sorry if that sounded offensive, I didn't mean to!) I'm likewise sad that I just turned 27. I was always the youngest in school and university, graduating with honors at 20. Getting closer to 30 now. "Where are you now golden boy, where is your famous golden touch?" Or this: "Tired of lying in the sunshine staying home to watch the rain, you are young and life is long and there is time to kill today."

Comment author: AdeleneDawner 21 October 2009 12:18:02PM *  2 points [-]

I'm not sad that I'm closer to 30. 30's cool, it's a multiple of three. I'm sad that the next time my age will be a power of three won't be 'till I'm 81. <chuckle>

Comment author: thomblake 20 October 2009 08:47:31PM 1 point [-]

I can specify the problem in terms of three identical buttons that you can press instead.

Please do, if you think that would make the problem clearer. The piece I'm not seeing is where UDT1 lets you choose something that expected utility does not. Does expected utility usually not allow you to have states of the world in your utility function?

Comment author: thomblake 21 October 2009 02:02:13PM 0 points [-]

Okay, let's see if I have this straight - you're assuming:

  1. the axiom of independence is necessary for expected utility theory
  2. losing a penny represents some negative amount of utility
  3. one's utility function can't include terms for "the outcomes of world programs" under expected utility theory
Comment author: timtyler 20 October 2009 08:32:07PM 3 points [-]

"Of all the axioms, independence is the most often discarded. A variety of generalized expected utility theories have arisen, most of which drop or relax the independence axiom."

Comment author: ShardPhoenix 22 October 2009 10:20:13AM 0 points [-]

Reminds me of the parallel postulate - non-Euclidean utility?

Comment author: Douglas_Knight 24 October 2009 07:22:16PM *  0 points [-]

"Of all the axioms, independence is the most often discarded. A variety of generalized expected utility theories have arisen, most of which drop or relax the independence axiom."

The examples in the generalized expected utility link are descriptive theories of how humans are irrational money pumps. (The two bullet points after the sentence in wikipedia are examples of conventional utility functions; in that context the sentence is false.)

Comment author: timtyler 25 October 2009 06:22:11PM 0 points [-]

I'm not sure what the bullet points are doing there either - but I don't really see how they impact the original statement.

Comment author: PhilGoetz 22 October 2009 09:39:01PM *  5 points [-]

It seems clear that our preferences do satisfy Independence, at least approximately.

How big of a problem does this simple example signify?

  • A = I acquire a Nintendo
  • B = I acquire a Playstation
  • C = I acquire a game for the Nintendo
  • D = I acquire a game for the Playstation
  • A&C > B&C but A&D < B&D
Comment author: Wei_Dai 24 October 2009 11:29:16AM 3 points [-]

Your example shows that we can't assign utilities to events within a single world, like acquiring game systems and games, and then add them up into a utility for that world, but it's not a counterexample to Independence, because of this part:

A and B are what happens in one possible world, and C and D are what happens in another.

Independence is necessary to assign utilities to possible world histories and aggregate those utilities linearly into expected utility. Consider the apples/oranges example again. There,

  • A = I get an apple in the world where coin is heads
  • B = I get an orange in the world where coin is heads
  • C = I get an apple in the world where coin is tails
  • D = I get an orange in the world where coin is tails

Then, according to Independence, my preferences must be either

  1. A&C > B&C and A&D > B&D, or
  2. A&C < B&C and A&D < B&D

If case 1, I should pick the transparent box with the apple, and if case 2, I should pick the transparent box with the orange.

(I just realized that technically, my example is wrong, because in case 1, it's possible that A&D > A&C and B&D > B&C. Then, I should most prefer an opaque box that contains an apple if the coin is heads and an orange if the coin is tails, since that gives me outcome A&D, and least prefer an opaque box that contains the opposite (gives me B&C). So unless I introduce other assumptions, I can only derive that I shouldn't simultaneously prefer both kinds of opaque boxes to transparent boxes.)

Comment author: wedrifid 23 October 2009 04:20:18AM 1 point [-]

This assumption is central to establishing the mathematical structure of expected utility maximization, where you value each possible world separately using the utility function, then take their weighted average. If your preferences were such that A&C > B&C but A&D < B&D, then you wouldn’t be able to do this.

I can imagine having preferences that don't value each possible world separately. I can also imagine doing other things to my utility function than maximising expectation. For example, if I maximised the top quartile of expected values then I may choose to engage in practices analogous to quantum suicide. That I prefer, in principle, to maximise expected utility is itself a value. It is a value I that I expect to see in most successful agents, for fundamental reasons.

Comment author: Matt_Simpson 22 October 2009 06:47:42AM 2 points [-]

Paul Churchland calls the belief/values (he says belief/desires) model "folk psychology" and assigns a low probability to it "being smoothly reduced by neuroscience" rather than being completely disregarded like, say, the phlogiston theory of combustion. The paper is called Eliminative Materialism and the Propositional Attitudes and was printed in The Journal of Philosophy. I didn't find the paper all that convincing, but your mileage may vary.

This paper was cited along with another by someone (can't remember who) arguing that the belief/values theory of behavior (i.e. expected utility theory) doesn't capture how humans behave. The second paper I think argues that much of what we do can be explained by control theory without reference to beliefs or values, but I haven't read it yet.

The papers are:

Churchland, Paul. Eliminative Materialism and the Propositional Attitudes, The Journal of Philosophy.

van Gelder, Tim. What Might Cognition be, if not Computation?, The Journal of Philosophy.

For those of you who don't have the benefit of a university subscription to J-stor or something similar, I have pdfs of both papers. Just shoot me an email at : themattsimpson AT <googlemail> DOT company

Comment author: [deleted] 21 October 2009 05:39:24PM 1 point [-]

Here, have a mathematical perspective that conflates beliefs and values:

Suppose that some agent is given a choice between A and B. A is an apple. B is an N chance of a banana, otherwise nothing. The important thing here is the ambivalence equation: iff U(apple) = N*U(banana), the agent is ambivalent between the apple and the banana. Further suppose that N is 50%, and the agent likes bananas twice as much as it likes apples. In this case, at least, the agent might as well modify itself to believe that N is 20% and to like bananas five times as much as apples.

Now, doing this might result in inconsistencies elsewhere, but I'm guessing that a rational agent will be able to apply transformations to its beliefs and values--but only both simultaneously--so as to preserve expected utility given actions.

I think a more concrete example of the beliefs-values meld is anthropic reasoning. Take the presumptuous friend: you, the presumptuous philosopher's presumptuous friend, have just been split into 1,000,001 branches, and each of those branches has been placed in a hotel room, 1,000,000 rooms being in one hotel, and the other room being in another hotel. Is the probability that you're in the small hotel 50%, negligible, or something in between? Well, that depends: if you care about each of your branches equally, it's negligible; if you care about each hotel equally, it's 50%.

Comment author: Wei_Dai 21 October 2009 12:02:44PM 4 points [-]

I have a tentative answer for the second question of "Why this representation?". Given that a set of preferences can be represented as a probability function and a utility function, that seems computationally more convenient than using two probability functions, since then you only have to do half of the Bayesian updating.

Another part of this question is that such a set of preferences can usually be decomposed many different ways into probability and utility, so what explains the particular decomposition that we have? I think there should have been a selection pressure for humans to have a common prior, to the extent possible, and move as much as possible of the differences in preferences into the utility function, since that would facilitate communication and sharing of information. It seems that if we had common priors, and I have a lot of information about something (and you trust me), I can just tell you my posterior beliefs, instead of having to give you all of the raw information and let you recompute your own posterior beliefs.

Comment author: whpearson 20 October 2009 10:30:05PM 1 point [-]

I think values (in a finite agent), also need to have some role in what beliefs "should" be stored/updated/remembered. Of course in theories which don't constrain the agents computational ability this isn't needed.

Comment author: Vladimir_Nesov 20 October 2009 08:26:37PM *  2 points [-]

This comment is directly about the question of probability and utility. The division is not so much about considering the two things separately, as it is about extracting tractable understanding of the whole human preference (prior+utility) into a well-defined mathematical object (prior), while leaving all the hard issues with elicitation of preference in the utility part. In practice it works like this: a human conceptualizes a problem so that a prior (that is described completely) can be fed to an automatic tool, then tool's conclusion about the aspect specified as probability is interpreted by a human again. People fill in the utility part by using their preference, even though they can't represent it as the remaining utility part. Economists, having to create autonomous models of decision-making (as distinct from autonomous decision-making systems), have to introduce the whole preference, but it's so approximate that it's of no use in most other contexts.

Because of the utility-prior divide of human preference in practice of human decision-making, with only prior in the domain of things that are technically understood, there is a strong association of prior with "knowledge" (hence "belief", but being people of science we expel feeling-associated connotations from the concept), while utility remains vague, but is a necessary part that completes the picture to the expression of whole preference, hence introduction of utility to a problem is strongly associated with values.

Comment author: Wei_Dai 20 October 2009 10:32:18PM 2 points [-]

But why do human preferences exhibit the (approximate) independence which allows the extraction to take place?

Comment author: SilasBarta 20 October 2009 10:49:08PM *  3 points [-]

Simple. They don't.

Maybe it's just me, but this looks like another case of overextrapolation from a community of rationalists to all of humanity. You think about all the conversations you've had distinguishing beliefs from values, and you figure everyone else must think that way.

In reality, people don't normally make such a precise division. But don't take my word for it. Go up to your random mouthbreather and try to find out how well they adhere to a value/belief distinction. Ask them whether the utility assigned to an outcome, or its probability was a bigger factor.

No one actually does those calculations consciously; if anything like it is done non-consciously, it's extremely economical in computation.

Comment author: Vladimir_Nesov 20 October 2009 10:51:17PM *  1 point [-]

Simple: the extraction cuts across preexisting independencies. (I don't quite see what you refer to by "extraction", but my answer seems general enough to cover most possibilities.)

Comment author: Wei_Dai 20 October 2009 11:01:31PM 1 point [-]

I'm referring to the extraction that you were talking about: extracting human preference into prior and utility. Again, the question is why the necessary independence for this exists in the first place.

Comment author: Vladimir_Nesov 20 October 2009 11:06:03PM *  2 points [-]

I was talking about extraction of prior about a narrow situation as the simple extractable aspect of preference, period. Utility is just the rest, what remains unextractable in preference.

Comment author: Wei_Dai 20 October 2009 11:23:44PM 1 point [-]

Ok, I see. In that case, do you think there is still a puzzle to be solved, about why human preferences seem to have a large amount of independence (compared to, say, a set of randomly chosen transitive preferences), or not?

Comment author: Vladimir_Nesov 20 October 2009 11:41:42PM *  3 points [-]

That's just a different puzzle. You are asking a question about properties of human preference now, not of prior/utility separation. I don't expect strict independence anywhere.

Independence is indifference, due to inability to see and precisely evaluate all consequences, made strict in form of probability, by decree of maximum entropy. If you know your preference about an event, but no preference/understanding on the uniform elements it consists of, you are indifferent to these elements -- hence maximum entropy rule, air molecules in the room. Multiple events for which you only care in themselves, but not in the way they interact, are modeled as independent.

[W]hy human preferences seem to have a large amount of independence (compared to, say, a set of randomly chosen transitive preferences)[?]

Randomness is info, so of course the result will be more complex. Where you are indifferent, random choice will fill in the blanks.

Comment author: Wei_Dai 21 October 2009 12:17:07AM 3 points [-]

It sounds like what you're saying is that independence is a necessary consequence of our preferences having limited information. I had considered this possibility and don't think it's right, because I can give a set of preferences with little independence and also little information, just by choosing the preferences using a pseudorandom number generator.

I think there is still a puzzle here, why our preferences show a very specific kind of structure (non-randomness).

Comment author: Vladimir_Nesov 21 October 2009 01:02:10AM *  3 points [-]

That new preference of yours still can't distinguish the states of air molecules in the room, even if some of these states are made logically impossible by what's known about macro-objects. This shows both the source of dependence in precise preference and of independence in real-world approximations of preference. Independence remains where there's no computed info that allows to bring preference in contact with facts. Preference is defined procedurally in the mind, and its expression is limited by what can be procedurally figured out.

Comment author: Wei_Dai 21 October 2009 11:38:05AM 1 point [-]

I don't really understand what you mean at this point. Take my apples/oranges example, which seems to have nothing to do with macro vs. micro. The Axiom of Independence says I shouldn't choose the 3rd box. Can you tell me whether you think that's right, or wrong (meaning I can rationally choose the 3rd box), and why?

To make that example clearer, let's say that the universe ends right after I eat the apple or orange, so there are no further consequences beyond that.

Comment author: Vladimir_Nesov 20 October 2009 06:26:41PM *  8 points [-]

Just to distance this very interesting question from expected utility maximization: "Beliefs" sound like they are about couldness, and values about shouldness. Couldness is about behavior of the environment outside the agent, and shouldness is about behavior of the agent. Of course, the two only really exist in interaction, but as systems they can be conceptualized separately. When an agent asks what it could do, the question is really about what effects in environment could be achieved (some Tarskian hypocrisy here: using "could" to explain "couldness"). Beliefs is what's assumed, and values is what's asserted. In a decision tree, beliefs are associated with knowledge about other agent's possible actions, and values with the choice of the present agent's action. Both are aspects of the system, but playing different roles in the interaction: making a choice versus accepting a choice. Naturally, there is a duality here, when the sides are exchanged: my values become your beliefs, and my beliefs become your values. Choice of representation is not that interesting, as it's all interpretation: nothing changes in behavior.

Comment author: Wei_Dai 20 October 2009 10:36:38PM 2 points [-]

I gave an example where choice of representation is important: Eliezer's CEV. If the choice of representation shouldn't to be important, then that seems to be argument against CEV.

Comment author: SilasBarta 20 October 2009 10:54:59PM *  2 points [-]

Bullet acknowledged and bitten. A Friendly AI attempting to identify humanity's supposed CEV will also have to be a politician and have enough support so that they don't shut it down. As a politician, it will have to appeal to people with the standard biases. So it's not enough for it to say, "okay, here's something all of you should agree on as a value, and benefit from me moving humanity to that state".

And in figuring out what would appeal to humans, it will have to model the same biases that blur the distinction.

Comment author: Vladimir_Nesov 20 October 2009 10:54:08PM 0 points [-]

I was referring to you referring to my post on playing with utility/prior representations.

Comment author: Stuart_Armstrong 23 October 2009 02:40:13PM 0 points [-]

Maybe these are to do with differences across individuals. My beliefs/values may be mashed togather and impossible to seperate, but I expect other people's beliefs to mirror my own more closely than their values do.

Comment author: Kyre 21 October 2009 06:15:53AM *  0 points [-]

Because it's much easier to use beliefs shorn of values as building blocks in a machine that does induction, inference, counterfactual reasoning, planning etc compared to belief-values that are all tied up together.

Sea slugs and Roombas don't have the beliefs/values separation it because the extra complexity isn't worth it. Humans have it to some degree and rule the planet. AIs might have even more success.