= 27d567bc2d48d9f40e9ccc20ea370b15)
Cyan comments on The Domain of Your Utility Function - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (94)
Utility as a technical term in decision theory isn't equivalent to happiness and disutility isn't equivalent to unhappiness. Rather, the idea is to find some behaviorally descriptive function which takes things like negative affectivity and appetite satisfaction levels as arguments and return a summary, which for lack of a better term we call utility. The existence of such a function is required by certain axioms of consistency -- the thought is that if one's behavior cannot be described by a utility function, then they will have intransitive preferences.
As a descriptive statement, human beings probably do have circular preferences; the prescriptive question is whether there is a legitimate utility function we can extrapolate from that mess without discarding too much.
You inevitably draw specific actions, so there is no escaping forming a preference over actions (a decision procedure, not necessarily preference over things that won't play), and "discarding too much" can't be an argument against the inevitable. (Not that I particularly espouse the form of preference being utility+prior.)
Sorry, I meant something like "whether there is a relatively simple decision algorithm with consistent preferences that we can extrapolate from that mess without discarding too much". If not, then a superintelligence might be able to extrapolate us, but until then we'll be stymied in our attempts to think rationally about large unfamiliar decisions.
Fair enough. Note that the superintelligence itself must be a simple decision algorithm for it to be knowably good, if that's at all possible (at the outset, before starting to process the particular data from observations), which kinda defeats the purpose of your statement. :-)
Well, the code for the seed should be pretty simple, at least. But I don't see how that defeats the purpose of my statement; it may be that short of enlisting a superintelligence to help, all current attempts to approximate and extrapolate human preferences in a consistent fashion (e.g. explicit ethical or political theories) might be too crude to have any chance of success (by the standard of actual human preferences) in novel scenarios. I don't believe this will be the case, but it's a possibility worth keeping an eye on.
Oh, indeed. I just want to distinguish between things that humans really experience and the technical meaning of the term "utility". In particular, I wanted to avoid a conversation in which disutility, which sounds like a euphemism for discomfort, is juxtaposed with decision theoretic utility.
Nitpick: if one's behavior cannot be described by a utility function, then one will have preferences that are intransitive, incomplete, or violate continuity.
I'm with you on "incomplete" (thanks for the catch!) but I'm not so sure about "violate continuity". Can you give an example of preferences that are transitive and complete but violate continuity and are therefore not encodable in a utility function?
Lexicographic preferences are the standard example: they are complete and transitive but violate continuity, and are therefore not encodable in a standard utility function (i.e. if the utility function is required to be real-valued; I confess I don't know enough about surreals/hyperreals etc. to know whether they will allow a representation).
I'd heard that mentioned before around these parts, but I didn't recall it because I don't really understand it. I think I must be making a false assumption, because I'm thinking of lexicographic ordering as the ordering of words in a dictionary, and the function that maps words to their ordinal position in the list ought to qualify. Maybe the assumption I'm missing is a countably infinite alphabet? English lacks that.
The wikipedia entry on lexicographic preferences isn't great, but gives the basic flavour:
That entry says,
So my intuition above was not correct -- an uncountably infinite alphabet is what's required.
Intransitive preferences don't mean that you can't describe an agent's actions with a utitilty function. So what if an agent prefers A to B, B to C and C to A? It might mean they will drive in circles and waste their energy - but it doesn't mean you can't describe their preferences with a utility function. All it means is that their utility function will not be as simple as it could be.
In the standard definition, the domain of the utility function is the set of states of the world and the range is the set of real numbers; the preferences among states of the world are encoded as inequalities in the utility of those states. I read your comment as asserting that there exists real numbers a, b, c, such that a > b, b > c, and c > a. I conclude that you must have something other than the standard definition in mind.
If A is Alaska, B is Boston, and C is California, the preferences involve preferring being in Alaska if you are in Boston, preferring being in Boston if you are in California, and preferring being in California if you are in Alaska. The act of expressing those preferences using a utility function does not imply any false statements about the set of real numbers.
Preferring A to B means that, given the choice between A and B, you will pick A, regardless of where you currently are (you might be in California but have to leave). This is not the same thing as choosing A over B, contingent on being in B.
You can indeed express the latter set of preferences you describe using a standard utility function, but that's because you've redefined them so that they're no longer intransitive.
Its not clear you're contradicting Cyan. You describe the converse of what he describes.
Even if a utility function can be written down which allows intransitive preferences, its worth noting that transitive preferences is a standard assumption.
ISTM that if an agent's preferences cannot be described by a utility function, then it is because the agent is either spatially or temporally infinite - or because it is uncomputable.
I'm struggling to see how such a utility function could work. Could you give an example of a utility function that describes the preferences you just set out, and has the implication that u(x)>u(y) <=> xPy?
It’s not difficult to code (if A:B,if B:C,if C:A) into a utilitarian system. If A is Alaska, B is Boston, and C is California, that would cause driving in circles.
With respect, that doesn't seem to meet my request. Like Cyan, I'm tempted to conclude that you are using a non-standard definition of "utility function".
ETA: Oh, wait... perhaps I've misunderstood you. Are you trying to say that you can represent these preferences with a function that assigns: u(A:B)>u(x:B) for x in {B,C}; u(B:C)>u(x:C) for x in {A,C} etc? If so, then you're right that you can encode these preferences into a utility function; but you've done so by redefining things such that the preferences no longer violate transitivity; so Cyan's original point stands.
Cyan claimed some agent's behaviour corresponded to intransitive preferences. My example is the one that is most frequently given as an example of circular preferences. If this doesn't qualify, then what behaviour are we talking about?
What is this behaviour pattern that supposedly can't be represented by a utility function due to intransitive preferences?
Suppose I am in Alaska. If told I can either stay or go to Boston, I choose to stay. If told I can either stay or go to California, I choose California. If told I must leave for either Boston or California, I choose Boston. These preferences are intransitive, and AFAICT, cannot be represented by a utility function. To do so would require u(A:A)>u(B:A)>u(C:A)>u(A:A).
More generally, it is true that one can often redefine states of the world such that apparently intransitive preferences can be rendered transitive, and thus amenable to a utility representation. Whether it's wise or useful to do so will depend on the context.
You are not getting this :-( You have just given me a description of the agents preferences. From there you are not far from an algorithm that describes them.
Your agent just chooses differently depending on the options it is presented with. Obviously, the sense data relating to what it was told about its options is one of the inputs to its utility function - something like this:
If O=(A,C) then u(C)=1; else if O=(B,C) then u(B)=1.
So your position isn't so much "intransitive preferences are representable in utility functions" as it is "all preferences are transitive because we can always make them contingent on the choice offered".
I think the point is that any decision algorithm, even one which has intransitive preferences over world-states, can be described as optimization of a utility function. However, the objects to which utility are assigned may be ridiculously complicated constructs rather than the things we think should determine our actions.
To show this is trivially true, take your decision algorithm and consider the utility function "1 for acting in accordance with this algorithm, 0 for not doing so". Tim is giving an example where it doesn't have to be this ridiculous, but still has to be meta compared to object-level preferences.
Still (I say), if it's less complicated to describe the full range of human behavior by an algorithm that doesn't break down into utility function plus optimizer, then we're better off doing so (as a descriptive strategy).
Which pretty much mauls the definition of transitive beyond recognition.
I think "circular preferences" is a useful concept - but I deny that it means that a utilitarian explanation is impossible. See my A, B, C example of what are conventionally referred to as being circular preferences - and then see how that can still be represented within a utilitarian framework.
This really is the conventional example of circular preferences - e.g. see:
"If you drive from San Jose to San Francisco to Oakland to San Jose, over and over again, you may have fun driving, but you aren't going anywhere." - http://lesswrong.com/lw/n3/circular_altruism/
"This almost inevitably leads to circular preferences wherein you prefer Spain to Greece, Greece to Turkey but Turkey to Spain." - http://www.cparish.co.uk/cpapriover.html
Circular preferences in agents are often cited as something utilitarianism can't deal with - but it's simply a fallacy.
Sure, you can do that (though you'll also need to specify what happens when O=(A,B,C) or any larger set of options, which will probably get pretty cumbersome pretty quickly). But the resulting algorithm doesn't fall within the standard definition of a utility function, the whole point of which is to enable us to describe preferences without needing to refer to a specific choice set.
If you want to use a different definition of "utility function" that's fine. But you should probably (a) be aware that you're departing from the standard technical usage, and (b) avoid disputing claims put forward by others that are perfectly valid on the basis of that standard technical usage.
P.S. Just because someone disagrees with you, doesn't mean they don't get it. ;)
A utility function just maps states down to a one-dimensional spectrum of utility.
That is a simple-enough concept, and I doubt it is the source of disagreement.
The difference boils down to what the utility function is applied to. If the inputs to the utility function are "Alaska", "Boston" and "California", then a utilitarian representation of circular driving behaviour is impossible.
However, in practice, agents know more than just what they want. They know what they have got. Also, they know how bored they are. So, expanding the set of inputs to the utility function to include other aspects of the agent's state provides a utilitarian resolution. This does not represent a non-standard definition or theory - it is just including more of the agent's state in the inputs to the utility function.