= 27d567bc2d48d9f40e9ccc20ea370b15)
pjeby comments on Morality as Parfitian-filtered Decision Theory? - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (270)
Humans don't operate by maximizing utiltiy, for any definition of "utility" that isn't hideously tortured. Mostly, we simply act in ways that keep the expected value of relevant perceptual variables (such as our own feelings) within our personally-defined tolerances.
(Corollary: creating autonomous systems that are utility-maximizing is a Really Bad Idea, as they will fail in ways that humans wouldn't intuitively expect. A superhuman FAI might be capable of constructing a friendly maximizer, but a human would be an idiot to try.)
I appreciate that you're criticizing the ad-hoc assumptions needed to salvage the utility function model in certain contexts, as one of my points was that several utility functions can equally well explain the same actions.
Still, could you please limit your comments about Perceptual Control Theory to points directly relevant to the issues I raised? Just link one of your previous expositions of PCT rather than use this discussion as a platform to argue for it anew.
Actually, the definition of "utility" is pretty simple. It is simply "that thing that gets maximized in any particular person's decision making". Perhaps you think that humans do not maximize utility because you have a preferred definition of utility that is different from this one.
Ok, that is a plausible sounding alternative to the idea of maximizing something. But the maximizing theory has been under scrutiny for 150 years, and under strong scrutiny for the past 50. It only seems fair to give your idea some scrutiny too. Two questions jump out at me:
Looking forward to hearing your answer on these points. If we can turn your idea into a consistent and plausible theory of human decision making, I'm sure we can publish it.
Whatever occurs to us first. ;-)
We waffle, or try to avoid making the decision in the first place. ;-) (See, e.g., typical people's reactions to "trolley problems", or other no-win scenarios.)
What I'm saying is that the above construction leads to error if you assume that "utility" is a function of the state of the world outside the human, rather than a function of the difference between the human's perceptions of the outside world, and the human's internal reference values or tolerance ranges for those perceptions.
Maximizing a utility function over the state of the external world inherently tends to create results that would be considered undesirable by most humans. (See, for example, the various tortured insanities that come about when you try to maximize such a conception of "utility" over a population of humans.)
It's important to understand that the representation you use to compute something is not value-neutral. Roman numerals, for example, make division much more complicated than Arabic ones.
So, I'm not saying that you can't create some sort of "utility" function to represent human values. We have no reason to assume that human values aren't Turing-computable, and if they're Turing-computable, we should be able to use whatever stupidly complex representation we want to compute them.
However, to use world-state-utility as your basis for computation is just plain silly, like using Roman numerals for long division. Your own intuition will make it harder for you to see the Friendliness-failures that are sitting right under your nose, because utility maximization is utterly foreign to normal human cognitive processes. (Externality-maximizing processes in human behavior are generally the result of pathology, rather than normal brain function.)
Eliezer hasn't been alive that long, has he? ;-)
Seriously, though, external-utility-maximizing thinking is the very essence of Unfriendly AI, and the history of discussions of world-state-based utility is that models based on it lead to counterintuitive results unless you torture the utility function hard enough, and/or carefully avoid the sort of creative thinking that an unfettered superintelligence might come up with.
It's a losing battle to describe humans as utility maximizers. Utility, as applied to people, is more useful in the normative sense, as a way to formulate one's wishes, allowing to infer the way one should act in order to follow them.
Nevertheless, standard economic game theory frequently involves an assumption that it is common knowledge that all players are rational utility maximizers. And the reason it does so is the belief that on the really important decisions, people work extra hard to be rational.
For this reason, on the really important decisions, utility maximization probably is not too far wrong as a descriptive theory.
The reason it does so is because it is convenient.
I don't entirely agree with pgeby. Being unable to adequately approximate human preferences to a single utility function is not something that is a property of the 'real world'. It is something that is a property of our rather significant limitations when it comes to making such evaluations. Nevertheless, having a textbook prescribe official status to certain mechanisms for deriving a utility function does not make that process at all reliable.
I'll be sure to remember that line, for when the people promoting other models of rationality start citing textbooks too. Well, no, I probably won't, since I doubt I will live long enough to see that. ;)
But, if I recall correctly, I have mostly cited the standard textbook thought-experiments when responding to claims that utility maximization is conceptually incoherent - so absurd that no one in their right mind would propose it.
I see that you are trying to be snide, but it took a while to figure out why you would believe this to be incisive. I had to reconstruct a model of what you think other people here believe from your previous rants.
Yes. That would be a crazy thing to believe. (Mind you, I don't think pjeby believes crazy things - he just isn't listening closely enough to what you are saying to notice anything other than a nail upon which to use one of his favourite hammers.)
It seems to me that what has actually been shown is that when people think abstractly (i.e. "far") about these kinds of decisions, they attempt to calculate some sort of (local and extremely context-dependent) maximum utility.
However, when people actually act (using "near" thinking), they tend to do so based on the kind of perceptual filtering discussed in this thread.
What's more, even their "far" calculations tend to be biased and filtered by the same sort of perceptual filtering processes, even when they are (theoretically) calculating "utility" according to a contextually-chosen definition of utility. (What a person decides to weigh into a calculation of "best car" is going to vary from one day to the next, based on priming and other factors.)
In the very best case scenario for utility maximization, we aren't even all that motivated to go out and maximize utility: it's still more like playing, "pick the best perceived-available option", which is really not the same thing as operating to maximize utility (e.g. the number of paperclips in the world). Even the most paperclip-obsessed human being wouldn't be able to do a good job of intuiting the likely behavior of a true paperclip-maximizing agent -- even if said agent were of only-human intelligence.
For me, I'm not sure that "rational" and "utility maximizer" belong in the same sentence. ;-)
In simplified economic games (think: spherical cows on a frictionless plane), you can perhaps get away with such silliness, but instrumental rationality and fungible utility don't mix under real world conditions. You can't measure a human's perception of "utility" on just a single axis!
You have successfully communicated your scorn. You were much less successful at convincing anyone of your understanding of the facts.
And you can't (consistently) make a decision without comparing the alternatives along a single axis. And there are dozens of textbooks with a chapter explaining in detail exactly how you go about doing it.
And what makes you think humans are any good at making consistent decisions?
The experimental evidence clearly says we're not: frame a problem in two different ways, you get two different answers. Give us larger dishes of food, and we eat more of it, even if we don't like the taste! Prime us with a number, and it changes what we'll say we're willing to pay for something utterly unrelated to the number.
Human beings are inconsistent by default.
Of course. But that's not how human beings generally make decisions, and there is experimental evidence that shows such linearized decision algorithms are abysmal at making people happy with their decisions! The more "rationally" you weigh a decision, the less likely you are to be happy with the results.
(Which is probably a factor in why smarter, more "rational" people are often less happy than their less-rational counterparts.)
In addition, other experiments show that people who make choices in "maximizer" style (people who are unwilling to choose until they are convinced they have the best choice) are consistently less satisfied than people who are satisficers for the same decision context.
It seems there is some criteria by which you are evaluating various strategies for making decisions. Assuming you are not merely trying to enforce your deontological whims upon your fellow humans I can infer that there is some kind of rough utility function by which you are giving your advice and advocating decision making mechanisms. While it is certainly not what we would find in Perplexed's text books it is this function which can be appropriately described as a 'rational utility function'.
I am glad that you included the scare quotes around 'rationally'. It is 'rational' to do what is going to get the best results. It is important to realise the difference between 'sucking at making linearized spock-like decisions' and for good decisions being in principle uncomputable in a linearized manner. If you can say that one decision sucks more than another one then you have criteria by which to sort them in a linearized manner.
Not at all. Even in pure computational systems, being able to compare two things is not the same as having a total ordering.
For example, in predicate dispatching, priority is based on logical implication relationships between conditions, but an arbitrary set of applicable conditions isn't guaranteed to have a total (i.e. linear) ordering.
What I'm saying is that human preferences generally express only a partial ordering, which means that mapping to a linearizable "utility" function necessarily loses information from that preference ordering.
That's why building an AI that makes decisions on such a basis is a really, really Bad Idea. Why build that kind of information loss into your ground rules? It's insane.
Am I correct thinking that you welcome money pumps?
True enough.
But the information loss is "just in time" - it doesn't take place until actually making a decision. The information about utilities that is "stored" is a mapping from states-of-the-world to ordinal utilities of each "result". That is, in effect, a partial order of result utilities. Result A is better than result B in some states of the world, but the preference is reversed in other states.
You don't convert that partial order into a total order until you form a weighted average of utilities using your subjective estimates of the state-of-the-world probability distribution. That takes place at the last possible moment - the moment when you have to make the decision.
It doesn't literally lose information - since the information inputs are sensory, and they can be archived as well as ever.
The short answer is that human cognition is a mess. We don't want to reproduce all the screw-ups in an intelligent machine - and what you are talking about lookss like one of the mistakes.
Perplexed answered this question well.
Nothing make me think that. I don't even care. That is the business of people like Tversky and Kahneman.
They can give us a nice descriptive theory of what idiots people really are. I am more interested in a nice normative theory of what geniuses people ought to be.
What you seem to have not noticed is that one key reason human preferences can be inconsistent is because they are represented in a more expressive formal system than a single utility value.
Or that conversely, the very fact that utility functions are linearizable means that they are inherently less expressive.
Now, I'm not saying "more expressiveness is always better", because, being human, I have the ability to value things non-fungibly. ;-)
However, in any context where we wish to be able to mathematically represent human preferences -- and where lives are on the line by doing so -- we would be throwing away important, valuable information by pretending we can map a partial ordering to a total ordering.
That's why I consider the "economic games assumption" to be a spherical cow assumption. It works nicely enough for toy problems, but not for real-world ones.
Heck, I'll go so far as to suggest that unless one has done programming or mathematics work involving partial orderings, that one is unlikely to really understand just how non-linearizable the world is. (Though I imagine there may be other domains where one might encounter similar experiences.)
Programming and math are definitely the fields where most of my experience with partial orders comes from. Particularly domain theory and denotational semantics. Complete partial orders and all that. But the concepts also show up in economics textbooks. The whole concept of Pareto optimality is based on partial orders. As is demand theory in micro-economics. Indifference curves.
Theorists are not as ignorant or mathematically naive as you seem to imagine.
You are talking about the independence axiom...?
You can just drop that, you know:
"Of all the axioms, independence is the most often discarded. A variety of generalized expected utility theories have arisen, most of which drop or relax the independence axiom."
Ah, "the advantage of theft over honest toil". Writing down a definite noun phrase does not guarantee the existence of a thing in reality that it names.
Some specific references would help in discerning what, specifically, you are alluding to here. You say in another comment in this thread:
but you have not done this at all, merely made vague allusions to "the last 150 years" and "standard economic game theory".
Well, you can't get much more standard than Von Neumann and Morgenstern's "Theory of Games and Economic Behaviour". This book does not attempt to justify the hypothesis that we maximise something when we make decisions. That is an assumption that they adopt as part of the customary background for the questions they want to address. Historically, the assumption goes back to the questions about gambling that got probability theory started, in which there is a definite thing -- money -- that people can reasonably be regarded as maximising. Splitting utility from money eliminates complications due to diminishing marginal utility of money. The Utility Theorem does not prove, or attempt to prove, that we are maximisers. It is a not very deep mathematical theorem demonstrating that certain axioms on a set imply that it is isomorphic to an interval of the real line. The hypothesis that human preferences are accurately modelled as a function from choices to a set satisfying those axioms is nowhere addressed in the text.
I shall name this the Utility Hypothesis. What evidence are you depending on for asserting it?
That isn't a particularly good example. There are advantages to theft over honest toil. It is just considered inappropriate to acknowledge them.
I have a whole stash of audio books that I purchased with the fruit of 'honest toil'. I can no longer use them because they are crippled with DRM. I may be able to sift around and find the password somewhere but to be honest I suspect it would be far easier to go and 'steal' a copy.
Oh, then there's the bit where you can get a whole lot of money and stuff for free. That's an advantage!
It's a metaphor.
My point being that it is a bad metaphor.
I liked the metaphor. Russell was a smart man. But so was von Neumann, and Aumann and Myerson must have gotten their Nobel prizes for doing something useful.
Axiomatic "theft" has its place along side empirical "toil"
So, am I to understand that you like people with Nobel prizes? If I start writing the names of impressive people can I claim some of their status for myself too? How many times will I be able to do it before the claims start to wear thin?
Before I broke down and hit the Kibitz button I had a strong hunch that Clippy had written the above. Interesting. ;)
Only if you are endorsing their ideas in the face of an opposition which cannot cite such names. ;)
Sorry if it is wearing thin, but I am also tired of being attacked as if the ideas I am promoting mark me as some kind of crank.
I haven't observed other people referencing those same names both before and after your appearance having all that much impact on you. Nor have I taken seriously your attempts to present a battle between "Perplexed and all Nobel prize winners" vs "others". I'd be very surprised if the guys behind the names really had your back in these fights, even if you are convinced you are fighting in their honour.
Sure. Happy to help. I too sometimes have days when I can't remember how to work that "Google" thing.
You mention Von Neumann and Morgenstern's "Theory of Games and Economic Behaviour" yourself - as you can see, I have added an Amazon link. The relevant chapter is #3.
Improvements to this version have been made by Savage and by Anscombe and Aumann. You can get a useful survey of the field from wikipedia. Wikipedia is an amazing resource, by the way. I strongly recommend it.
Two texts from my own bookshelf that contain expositions of this material are Chapter 1 of Myerson and Chapter 2 of Luce and Raiffa. I would recommend the Myerson. Luce and Raiffa is cheaper, but it is somewhat dated and doesn't prove much coverage at all of the more advanced topics such as correlated equilibria and the revelation principle. It does have some good material on Nash's program though.
And finally, for a bit of fun in the spirit of Project Steve, I offer this online bibliography of some of the ways this body of theory has been applied in one particular field.
Did I assert it? Where? I apologize profusely if I did anything more than to suggest that it provides a useful model for the more important and carefully considered economic decisions. I explicitly state here that the justification of the theory is not empirical. The theory is about rational decision making, not human decision making.
It is not. As I said, the authors do not attempt to justify the Utility Hypothesis, they assume it. Chapter 2 (not 3), page 8: "This problem [of what to assume about individuals in economic theory] has been stated traditionally by assuming that the consumer desires to obtain a maximum of utility or satisfaction and the entrepreneur a maximum of profits." The entire book is about the implications of that assumption, not its justificaation, of which it says nothing.
Neither do these authors attempt to justify the Utility Hypothesis; they too assume it. I can find Luce and Raiffa in my library and Myerson through inter-library loan, but as none of the first three works you've cited provide evidence for the claim that people have utility functions, rather than postulating it as an axiom, I doubt that these would either.
But now you deny having asserted any such thing:
Here you claim that people have utility functions:
And also here:
Here you assume that people must be talking about utility functions:
Referring to the message from which the last three quotes are taken, you say
and yet here you expand the phrase "prefer to assume" as :
These are weasel words to let you talk about utility functions while denying you think there are any such things.
How would you set about finding a model that is closer to reality, rather than one which merely makes better predictions?
I would undertake an arduous self-education in neuroscience. Thankfully, I have no interest in cognitive models which are close to reality but make bad predictions. I'm no longer as good at learning whole new fields as I was when I was younger, so I would find neuroscience a tough slog.
It looks as though it can be rearranged into a utility-maximization representation pretty easily. Set utility equal to minus the extent to which the "personally-defined tolerances" are exceeded. Presto!
Not quite - this would imply that tolerance-difference is fungible, and it's not. We can make trade-offs in our decision-making, but that requires conscious effort and it's a process more akin to barter than to money-trading.
Diamonds are not fungible - and yet they have prices. Same difference here, I figure.
What's the price of one red paperclip? Is it the same price as a house?
That seems to be of questionable relevance - since utilities in decision theory are all inside a single agent. Different agents having different values is not an issue in such contexts.
That's a big part of the problem right there: humans aren't "single agents" in this sense.
Humans are single agents in a number of senses - and are individual enough for the idea of revealed preference to be useful.
From the page you linked (emphasis added):
However, even if you ignore that, WARP is trivially proven false by actual human behavior: people demonstrably do sometimes choose differently based on context. That's what makes ordinal utilities a "spherical cow" abstraction.
(WARP's inapplicability when applied to real (non-spherical) humans, in one sentence: "I feel like having an apple today, instead of an orange." QED: humans are not "economic agents" under WARP, since they don't consistently choose A over B in environments where both A and B are available.)
The obvious way to combine the two systems -- tolerance and utility -- is to say that stimuli that exceed our tolerances prompt us to ask questions about how to solve a problem, and utility calculations answer those questions. This is not an original idea on my part, but I do not remember where I read about it.
The one that appears to maximize utility after a brief period of analysis. For example, I want ice cream; my ice cream satisfaction index is well below tolerance. Fortunately, I am in an ice cream parlor, which carries several flavors. I will briefly reflect on which variety maximizes my utility, which in this case is mostly defined by price, taste, and nutrition, and then pick a flavor that returns a high (not necessarily optimal) value for that utility.
A lack of acceptable alternatives leads to stress, which (a) broadens the range of acceptable outcomes, and (b) motivates future analysis about how to avoid similar situations in the future. For example, I want ice cream; my ice cream satisfaction index is well below tolerance; unfortunately, I am in the desert. I find this situation unpleasant, and eventually reconcile myself to the fact that my ice cream satisfaction level will remain below what was previously thought of as 'minimum' tolerance for some time, however, upon returning to civilization, I will have a lower tolerance for 'desert-related excursions' and may attempt to avoid further trips through the desert.
Note that 'minimum' tolerance refers to the minimum level that will lead to casual selection of an acceptable alternative, rather than the minimum level that allows my decision system to continue functioning.
Actually, I'd tend to say that you are not so much maximizing the utility of your ice cream choice, as you are ensuring that your expected satisfaction with your choice is within tolerance.
To put it another way, it's unlikely that you'll actually weigh price, cost, and taste, in some sort of unified scoring system.
Instead, what will happen is that you'll consider options that aren't already ruled out by cached memories (e.g. you hate that flavor), and then predict whether that choice will throw any other variables out of tolerance. i.e., "this one costs too much... those nuts will give me indigestion... that's way too big for my appetite... this one would taste good, but it just doesn't seem like what I really want..."
Yes, some people do search for the "best" choice in certain circumstances, and would need to exhaustively consider the options in those cases. But this is not a matter of maximizing some world-state-utility, it is simply that each choice is also being checked against a, "can I be certain I've made the best choice yet?" perception.
Even when we heavily engage our logical minds in search of "optimum" solutions, this cognition is still primarily guided by these kinds of asynchronous perceptual checks, just ones like, "Is this formula really as elegant as I want it to be?" instead.
Very interesting. There's a lot of truth in what you say. If anyone reading this can link to experiments or even experimental designs that try to figure out when people typically rely on tolerances vs. utilities, I'd greatly appreciate it.
Y'know, most people probably don't, and at times I certainly do take actions based entirely on nested tolerance-satisfaction. When I'm consciously aware that I'm making a decision, though, I tend to weigh the utilities, even for a minor choice like ice cream flavor. This may be part of why I felt estranged enough from modern society in the first place to want to participate in a blog like Less Wrong.
OK, so you've hit on the behavioral mechanism that helps me decide how much time I want to spend on a decision...90 seconds or so is usually the upper bound on how much time I will comfortably and casually spend on selecting an ice cream flavor. If I take too much time to decide, then my "overthinking" tolerance is exceeded and alarm bells go off; if I feel too uncertain about my decision, then my "uncertainty" tolerance is exceeded and alarm bells go off; if neither continuing to think about ice cream nor ending my thoughts about ice cream will silence both alarm bells, then I feel stress and broaden my tolerance and try to avoid the situation in the future, probably by hiring a really good psychotherapist.
But that's just the criteria for how long to think...not for what to think about. While I'm thinking about ice cream, I really am trying to maximize my ice-cream-related world-state-utility. I suspect that other people, for somewhat more important decisions, e.g., what car shall I buy, behave the same way -- it seems a bit cynical to me to say that people make the decision to buy a car because they've concluded that their car-buying analysis is sufficiently elegant; they probably buy the car or walk out of the dealership when they've concluded that the action will very probably significantly improve their car-related world-state-utility.
And how often, while doing this, do you invent new ice cream options in an effort to increase the utility beyond that offered by the available choices?
How many new ice cream flavors have you invented, or decided to ask for mixed together?
So now you say, "Ah, but it would take too long to do those things." And I say, "Yep, there goes another asynchronous prediction of an exceeded perceptual tolerance."
"Okay," you say, "so, I'm a bounded utility calculator."
"Really? Okay, what scoring system do you use to arrive at a combined rating on all these criteria that you're using? Do you even know what criteria you're using?"
Is this utility fungible? I mean, would you eat garlic ice cream if it were free? Would you eat it if they paid you? How much would they need to pay you?
The experimental data says that when it comes to making these estimates, your brain is massively subject to priming and anchoring effects -- so your "utility" being some kind of rational calculation is probably illusory to start with.
I was referring to the perceptions involved in a task like computer programming, not car-buying.
Part of the point is that every task has its own set of regulating perceptions.
They do it when they find a car that leads to an"acceptable "satisfaction" level.
Part of my point about things like time, elegance, "best"-ness, etc. though, is that they ALL factor into what "acceptable" means.
"Satisfaction", in other words, is a semi-prioritized measurement against tolerances on ALL car-buying-related perceptual predictions that get loaded into a person's "working memory" during the process.
Aside: I have partaken of the garlic ice-cream, and lo, it is good.
Are you joking? I'm curious!
I'm not joking, either about its existence or its gustatory virtues. I'm trying to remember where the devil I had it; ah yes, these fine folks served it at Taste of Edmonton (a sort of outdoor food-fair with samples from local restaurants).
Theory: you don't actually enjoy garlic ice cream. You just pretend to in order to send an expensive signal that you are not a vampire.
If I ever encounter it I shall be sure to have a taste!
I'm not going to respond point for point, because my interest in whether we make decisions based on tolerances or utilities is waning, because I believe that the distinction is largely one of semantics. You might possibly convince me that more than semantics are at stake, but so far your arguments have been of the wrong kind in order to do so.
Obviously we aren't rational utility-maximizers in any straightforward early-20th-century sense; there is a large literature on heuristics and biases, and I don't dispute its validity. Still, there's no reason that I can see why it must be the case that we exclusively weigh options in terms of tolerances and feedback rather than a (flawed) approach to maximizing utility. Either procedure can be reframed, without loss, in terms of the other, or at least so it seems to me. Your fluid and persuasive and persistent rephrasing of utility in terms of tolerance does not really change my opinion here.
As for ice cream flavors, I find that the ingenuity of chefs in manufacturing new ice cream flavors generally keeps pace with my ability to conceive of new flavors; I have not had to invent recipes for Lychee sorbet or Honey Mustard ice cream because there are already people out there trying to sell it to me. I often mix multiple flavors, syrups, and toppings. I would be glad to taste garlic ice cream if it were free, but expect that it would be unpleasant enough that I would have to be paid roughly $5 an ounce to eat it, mainly because I am counting calories and would have to cut out other foods that I enjoy more to make room for the garlic. As I've already admitted, though, I am probably not a typical example. The fact that my estimate of $5/oz is almost certainly biased, and is made with so little confidence that a better estimate of what you would have to pay me to eat it might be negative $0.50/oz to positive $30/oz, does not in any way convince me that my attempt to consult my own utility is "illusory."
It does not seem so to me, unless you recapitulate/encapsulate the tolerance framework into the utility function, at which point the notion of a utility function has become superfluous.
The point here isn't that humans can't do utility-maximization, it's merely that we don't, unless we have made it one of our perceptual-tolerance goals. So, in weighing the two models, we see one model that humans in principle can do (but mostly don't) and one that models what we mostly do, and can also model the flawed way of doing the other, that we actually do as well.
Seems like a slam dunk to me, at least if you're looking to understand or model humans' actual preferences with the simplest possible model.
The only thing I'm saying is illusory is the idea that utility is context-independent, and totally ordered without reflection.
(One bit of non-"semantic" relevance here is that we don't know whether it's even possible for a superintelligence to compute your "utility" for something without actually running a calculation that amounts to simulating your consciousness! There are vast spaces in all our "utility functions" which are indeterminate until we actually do the computations to disambiguate them.)
You confuse descriptive with normative.
Actually, in fairness to pjeby, I did a pretty good job of confusing them in my comment. If you look again, you will see that I was saying that standard utility maximization does a pretty good job on both the descriptive and the normative tasks.
And, of course as the whole structure of LW teaches us, utility maximization is only an approximation to the correct descriptive theory. I would claim that it is a good approximation - an approximation which keeps getting better as more and more cognitive resources are invested in any particular decision by the decision maker. But an approximation nonetheless.
So, what I am saying is that pjeby criticized me on descriptive grounds because that is where it seemed I had pitched my camp.
He made a "corollary" about the normative sense of utility maximization, right after an argument about its descriptive sense. Hence, confusion.
The choice of how you represent a computation is not value-neutral, even if all you care about is the computation speed.
The notion of a single utility function is computationally much better suited to machines than humans -- but that's because it's a much poorer representation of human values!
Conversely, single utility functions are much more poorly-suited to processing on humans' cognifitive architecture, because our brains don't really work that way.
Ergo, if you want to think about how humans will behave and what they will prefer, you are doing it suboptimally by using utility functions. You will have to think much harder to get worse answers than you would by thinking in terms of satisficing perceptual differences.
(IOW, the descriptive and normative aspects are pretty thoroughly intertwined, because the thing being described is also the thing that needs to be used to do the computation!)