Interesting article, but these really bugged me:
1) Using the environment as an example of false revealed preference. One person's pollution never ruins "the environment", at least not "their environment". The environment is only ruined by the aggregate effects of many people's pollution; or, the person is massively polluting a different environment.
Environmental solutions require collective agreement and enforcement, not unilateral disarmament. So polluting while claiming to value the environment is not hypocrisy even in the conventional sense of the term (that you criticize here).
And this is at least the second time I've explained this to you. Please stop using it as an example.
2) This phrasing:
There are enough connections between them that there's a big correlation in their activity, but the correlation isn't one ...
That initially reads like you're saying "the correlation isn't a correlation" so I had to re-read it. I recommend using any of the following terms as a replacement for the bolded word: perfect, unity, 1.0, 1, or "equal to one", any of which would have been clearer.
(Btw, I agree with your disrecommendation of Landsburg!)
Thanks for bringing that up. I've actually argued the opposite in the case of voting. Using timeless decision theory, you can justify voting (even without causing a bunch of people to go along with you) on the grounds that, if you would make this decision, the like-minded would reason the same way. (See my post "real-world newcomb-like problems".)
I think a crucial difference between the two cases is that non-pollution makes it even more profitable for others to pollute, which would make collective non-pollution (in the absence of a collective agreement) an unstable node. (For example, using less oil bids down the price and extends the scope of profitable uses.)
I've seen a fair amount of happiness research, and happiness tends towards the "liking" end of the scale. What makes people happy is giving to charity, meditating, long walks, and so on; what makes people unhappy is commuting, work stress, and child-rearing. Religion, old age, and living in Utah also make people happy.
A life designed to maximize happiness, according to happiness researchers, would not be a hedonistic orgy, as one might imagine. You are actually happier with a fair degree of self-restraint. But it would have a lot more peaceful hobbies and fewer grand, stressful goals (like strenuous careers and parenthood.) To me, the happiness-optimized life does not sound fun. It is not something I would look forward to with anticipation and eagerness. Statistically speaking, we'd like such a life, but we wouldn't want it. Myself, I'd rather be given what I want than what would make me happy.
Wow, this is unexpected in so many levels for me. You have access to happiness research yet you would stick to what you want instead. I don't mean to insult or think there is anything wrong, I'm just genuinely staggered at the fact.
I have read some thousands of pgs in happiness research, and started to follow advice. I'm more generous, I take long walks, I cherish friendships, I care very little for a long career, I go to evolutionary envinroments all the time (the park, swimming pools and beaches) I pursue objectives which really ought to make me say "I was doing something I consider important" and ignore money, having children, and some parts of familial obligations.
We had the same info, and we took such different paths...... this is awesome.
So I suppose I am much happier but am in a constant struggle not to want lots of things that I naturally would. So I'm in a kind of strenuous effort of self-control leading to constant bliss. I suppose that you are less happier (though probably not in any way perceivable from a first person perspective) but way more relaxed, prone to be guided by your desires and wishes, and willing to actually go there and do that thing you feel like doing.....
I wish I was you for two weeks or something, if only that were possible, and then I came back....
Thus wanting (motivation) is near, liking (enjoyment) is far (dopamine is near, opioids are far!). If liking doesn't have the power to make you actually do things, its role is primarily in forming your beliefs about what you want, which leads to presenting good images of yourself to others with sincerity.
So far, this is not a disagreement with "revealed preferences" thought. The disagreement would come in value judgment, where instead of taking the side of wanting (as economists seem to), or the side of liking (naive view, or one of the many varieties of moral ideologies), one carefully considers the virtues on case-to-case basis, being open to discard parts from either category. True preference is neither revealed nor felt.
By definition, if you choose X over Y, then X is a higher utility option than Y. That means utility represents wanting and not liking. But good utilitarians (and, presumably, artificial intelligences) try to maximize utility. This correlates contingently with maximizing happiness, but not necessarily
You are equivocating on the term 'utility' here, as have so many other commenters before in this forum. In the first sentence above, 'utility' is used in the sense given to that term by axiomatic utility theory. When the preferences of an individual conform to a set of axioms, they can be represented by a 'utility function'. The 'utilities' of this individual are the values of that function. By contrast, when ethicists discuss utilitarianism, what they mean by 'utility' is either pleasure or good. The empirical studies you cite, therefore, do not pose problems for utility theory or utilitarianism. They only pose problems for the muddled view on which utility functions represent that which hedonistic utilitarians think we ought to maximize.
I accept Benthamite's criticism as valid. It may not be obvious from the text, but in my mind I was definitely equivocating.
If we can't use preference to determine ethical utility, it makes ethical utilitarianism a lot harder, but that might be something we have to live with. I don't remember very much about Coherent Extrapolated Volition, but my vague memories say it makes that a lot harder too.
I observe that you might have caught this mistake earlier via this heuristic: "Using the phrase "by definition", anywhere outside of math, is among the most alarming signals of flawed argument I've ever found. It's right up there with "Hitler", "God", "absolutely certain" and "can't prove that"." I should probably rewrite "math" as "pure math" just to make this clearer.
Great post. It raised a question for me: why did evolution give us the pleasure mechanism at all, if the urge mechanism is sufficient to make us do stuff?
I agree that pleasure has something to do with learning, but I don't see why the "urge" or "desire" mechanism couldn't help us learn to do rewarding things without the existence of pleasure.
Without pleasure, things could work like this: If X is good for the animal, make the animal do X more often.
With pleasure, like this: If X is good for the animal, make the animal feel pleasure. Make the animal seek pleasure. (Therefore the animal will do X more often.)
So pleasure would seem to be a kind of buffer. My guess is that its purpose is to reduce the number of modifications to the animal's desires, thereby reducing the likelihood of mistaken modifications, which would be impossible to override.
I noticed the distinction between wanting and liking as a result of my meditation practice. I began to derive great pleasure from very simple things, like the quality of an intake of breath, or the color combination of trees and sky.
And, I began to notice a significant decrease in compulsive wanting, such as for excess food, and for any amount of alcohol.
I also noticed a significant decrease in my startle reflex.
Similar results have been reported from Davidson's lab at the University of Wisconsin. http://psyphz.psych.wisc.edu/
The first category, "things you do even though you don't like them very much" sounds like many drug addictions.
It's not limited to drugs or even similar physical stimuli like tasty food; according to my personal experience you can get the same effect with computer games. There's games that can be plenty of fun in the beginning (while you're learning what works), but stop being so once you abstract from that to a simple set of rules by which you can (usually) win, but nevertheless stay quite addictive in the latter phase. Whenever I play Dungeon Crawl Stone Soup for more than a few hours, I inevitably end up at a point where I don't even need to verbally think about what I'm doing for 95%+ of the wall-clock time spent playing, but that doesn't make it much easier to quit.
Popular vocabulary suggests that this is a fairly common effect.
Perhaps it is true that our modest technology for altering brain states (simple wireheading, recreational drugs, magnetic stimulation, etc.) leads only to stimulation of the "wanting" centers of the brain and to simple (though at times intense) pleasurable sensations. On the other hand though, it seems almost inevitable that as the secrets of the brain are progressively unlocked, and as our ability to manipulate the brain grows, it will be possible to generate all sorts of brain states, including those "higher" ones associated with love, accomplishment, fulfillment, joy, religious experiences, insight, bliss, tranquility and so on. Hence, while your analysis appears to be quite relevant with regard to wireheading today, I am skeptical that it is likely to apply much to the brain technology that could exist 50 years from now.
Man this post is horrifying to me. It seems to imply (or at least suggest) a world where everyone, individually and collectively, is in the grip of mind controlling parasites, shifting people's choices and redirecting societies resources, against our best interests. We're all trapped in these mental snares where we can't even choose something different.
I've never really taken akrasia, per se, seriously, because I basically believed the claim about revealed preferences. If you're engaging in some behavior that's because some part of you wants (and (implicitly) likes) the resulting consequences.
This new view is dystopic.
I will need to go back through this again, but as a DD person, I know that my ability to motivate myself to learn new things was astronomical compared to after I destroyed most of the dopaminergic systems in my head with Drug Abuse.
The largest area I have noticed is in painting and sculpting. Two areas where I used to spend inordinate amounts of time practicing/doing. I used to have the vast majority of my work-spaces covered with miniatures and sculptures that I was working on. Now... I have a hard time getting motivated to just get them out (which is I t...
Mostly it was Heroin, but there was a modest amount of Amphetamine usage involved as well (for completely patriotic reasons as well - /rolls eyes), and Cocaine became a problem for a few years, but strangely, I just stopped doing it one day like I would decide to throw out an old pair of underwear.
No alcohol was involved, which was mostly how I managed to get my brain into so many ƒMRI tunnels. I have never had any impairment from alcohol use, nor any dysfunction usage or abuse of alcohol either. Then, when several doctors found out about my anomalous cessation of cocaine, I got even more attention. That attention helped to free me from Heroin without the usual entanglement with a 12-step group or AA/NA (which at this point in time I have rather low opinions of).
I often wonder if I would still be alive if I hadn't started using these drugs though (which is contrary to what most people expect to hear). They do give a person a certain cognitive augmentation for each different drug, each of which can be highly useful depending upon the situation. I happened to be in a situation, during the 80s where amphetamines were indicated. I began to use the heroin because the amphetamines made ...
...A University of Michigan study analyzed the brains of rats eating a favorite food. They found separate circuits for "wanting" and "liking", and were able to knock out either circuit without affecting the other (it was actually kind of cute - they measured the number of times the rats licked their lips as a proxy for "liking", though of course they had a highly technical rationale behind it).
One could come up with a story about how people are motivated to act selfishly but enjoy acting morally, which allows them to tell other
The other problem here is distinguishing pleasure, fun, and happiness.
As I understand wireheading, it's equivalent to experiencing a lengthy orgasm. I would describe an orgasm as pleasurable, but it seems inaccurate to call it "fun", or to call the state of experiencing an orgasm as "happiness".
Just as a note, I don't experience orgasms as pleasurable. In fact, orgasms seem to me an excellent case in point of the difference between wanting and liking, at least in my case.
For the record, the actual Landsburg quote is
In one recent survey, 39 percent of New Yorkers said they would leave the city "if they could"! Every one of them was in New York on the day of the interview, so we know that at a minimum, 39 percent of New Yorkers lie to pollsters.
page 63 of his latest book The Big Questions.
Although I'm generally a big fan of Landsburg, this seems much more a case of confusion over what "leave the city" and "if you can" mean than one of lying.
On a related note, it seems people do not use 'happy' and 'unhappy' as opposite, at least when they're referring to a whole life. Rather, happiness involves normative notions (a good life) whereas being unhappy is simply about endorphins.
http://experimentalphilosophy.typepad.com/experimental_philosophy/2009/12/can-.html
There is a very simple explanation to the seeming discrepancy between wanting and liking, and that is that a person is always experiencing a tension between wanting a bit of pleasure now, versus a lot of pleasure later. Yes, spending time with your family may give you some pleasure now, but staying in NY and putting money aside will give you a lot of security later on. This may not explain the whole difference but perhaps a good chunk of it.
I think wireheading is dismissed far too quickly here and elsewhere. Like it or not (pun intended), the only reason ...
(Response to old post)
According to a recent poll, two out of three New Yorkers say that, given the choice, they would rather live somewhere else. But all of them have the choice, and none of them live anywhere else. A proper summary of the results of this poll would be: two out of three New Yorkers lie on polls.
Ordinary people do not interpret the statement "given the choice" to mean "under at least one circumstance where it is not physically impossible". That's not an example of revealed preferences or inconsistency--it's an example of real people not acting like Internet geeks.
I was reading a free on line book "The Authoritarians" by Robert Altemeyer one of his many findings of studying fundamentalists authoritarian follower types is that many deal with the guilt of doing some thing morally wrong by asking God for forgiveness and then feel closer to "Much less guilty" then "Appreciably less guilty" it may not be a wire in the head but it should keep one from some suffering.
I also very much like Dan Gilbert's Ted talk on synthesizing happiness I use it all the time because "it really is not so bad" and "it turned out for the best"
Great post! I completely agree with the criticism of revealed preferences in economics.
As a hedonistic utilitarian, I can't quite understand why we would favor anything other than the "liking" response. Converting the universe to utilitronium producing real pleasure is my preferred outcome. (And fortunately, there's enough of a connection between my "wanting" and "liking" systems that I want this to happen!)
The big point I took away from this article is that wanting and liking are different, and thus we should be skeptical of "revealed preferences".
But the title seemed to imply that the article wanted to address the question of whether or not we should wirehead. The last paragraph seems to argue that we should be really careful with wireheading, because we could get it wrong and not really know that we got it wrong.
...Go too far toward the liking direction, and you risk something different from wireheading only in that the probe is stuck in a differe
I'm a neuroscience major and have known about the different circuits for liking vs. wanting. And it's always been a belief of mine that people's revealed preferences are often times just wrong, and that this is huge problem with our economy. But somehow I never connected this to the liking/wanting circuits being different. Thanks!
they measured the number of times the rats licked their lips as a proxy for "liking", though of course they had a highly technical rationale behind it
Could you give a quick summary of this rationale?
I don't smoke, but I made the mistake of starting a can of Pringles yesterday. If you >asked me my favorite food, there are dozens of things I would say before "Pringles". >Right now, and for the vast majority of my life, I feel no desire to go and get Pringles. >But once I've had that first chip, my motivation for a second chip goes through the >roof, without my subjective assessment of how tasty Pringles are changing one bit.
What is missing from this is the effort (which eats up the limited willpower budget) required to get the s...
It's hard to track down specific things from that wireheading.com site, but this seems to be a good overview. Of particular note are a couple of excerpts:
...The results of these experiments indicate that reinforcing brain stimulation may have two distinct effects: (a) it activates pathways related to natural drives, and (b) it stimulates reinforcement pathways normally activated by natural rewards. The empirical observations seem to contradict classic "drive-reduction" theories of reinforcement (reinforcement appears to be associated with increase
So which form of good should altruists, governments, FAIs, and other agencies in the helping people business respect?
Somehow, trying to figure out the policy an FAI or paternalist government should have by examining our addictive reactions strikes me as like doing transportation planning by looking at people's reactions to red, yellow, and green lights. Not that people's reactions to these things are irrelevant to traffic planning, but rather that figuring out where people are trying to go is even more important than figuring out their reactions to tr...
My grandma always assumes that if I don't want to have [some kind of food] right now, that means I don't like it.
Problem: large chunks of philosophy and economics are based upon wanting and liking being the same thing.
I don't think that is true.
"Wanting" maps onto expected utility; "liking" is the reward signal - the actual utility.
That framing surely makes it seem like pretty standard economics.
There are some minor footnotes about how the reward signal can sometimes be self-generated - e.g. when you know you should have got the reward, but were just unlucky.
as a person who plans to wirehead themselves if other positive futures don't work out I find this very interesting but unconvincing.
“The wanting system is activated by dopamine, and the liking system is activated by opioids. There are enough connections between them that there's a big correlation in their activity” But are they orthogonal in principle?
This summarizes a common strain of thought in economics, the idea of "revealed preferences". People tend to say they like a lot of things, like family or the environment or a friendly workplace. Many of the same people who say these things then go and ignore their families, pollute, and take high-paying but stressful jobs. The traditional economic explanation is that the people's actions reveal their true preferences, and that all the talk about caring about family and the environment is just stuff people say to look good and gain status.
I thi...
To me, the question of whether wireheading is good is one where wireheading is defined as stimulating actual liking (not wanting). Because to me, the question is trying to get at whether or not happiness is reducible in theory, not in practice.
And so, I sense that this shouldn't be titled "Are wireheads happy?". It's more about the distinction between wanting and liking.
"One could come up with a story about how people are motivated to act selfishly but enjoy acting morally..."
Actually, I think a lot of stories are like that.
Because "CONFLICT IS DRAMA!!!!!1!!!one!!!!", a whole lot of stories I've been reading involve the characters having an arc that goes like this:
-Problem occurs, everyone has different ideas about how to solve it. -Ignored dissenting character, perhaps with prodding by certain outside forces, blows up, acts like a jerk, and storms off. -Dissenting character realizes that regardless of...
For research on human happiness that really does a great job of presenting non-intuitive results in a compelling fashion, I recommend Daniel Gilbert's Stumbling on Happiness.
For a great book in a lot of ways, I recommend Robert Frank's "Darwin Economy." I read this because in his interview on Russ Robert's "Econtalk" podcast, Robert Frank made the claim that 100 years from now Darwin will be recognized as the greatest economist.
Some of their points relevant to wireheading: happiness seems mostly relative, relative to where you wer...
I wonder. I grew up with experience in multiple systems of meditation, and found a way that works for me. Without electrodes or drugs or Nobel Prizes, I can choose to feel happy and relaxed and whatever. When I think about it, meditation can feel more pleasing and satisfying than every other experience in my life. Yet (luckily?) I do not feel any compulsion to do that in place of many other things, or try to advocate it. This is not because of willpower. While it lasts I like and want it, as if there's fulfillment of purpose, and when it's over I cannot r...
So which form of good should altruists, governments, FAIs, and other agencies in the helping people business respect?
Governments should give people what people say they want, rather than giving people what the governments think will make people happier, whenever they can't do both. But this is not because it's intrinsically better for people to get what they want than to get what makes them happier (it isn't), it's because people will resent what they percieve as paternalism in governments and because they won't pay taxes and obey laws in general if th...
I certainly find that I like creative work but don't want to work, like music but don't want to listen to music, like exercise but (sometimes) don't want to exercise. I like volunteering, but don't want to volunteer. (Perhaps tautologically) I like being in a good mood but sometimes don't want to be in a good mood.
From that short list, it seems that one ought to give more credence to "like" than "want." What I like doing, in the moment, correlates fairly well with conventional judgments of good behavior. (To be fair, some of what I ...
I realize that I'm late to the game on this post, but I have to say that as economist, I found the take home point about revealed preference to be quite interesting, and it makes me wonder about the extent to which further neuroscience research will find systematic disjunctions in everyday circumstances between what motivates us and what gives us pleasure. Undoubtedly this would be leveraged into new sorts of paternalistic arguments... I'm guessing we'll need another decade or two before we have the neuropaternalist's equivalent of Nudge, however.
...A University of Michigan study analyzed the brains of rats eating a favorite food. They found separate circuits for "wanting" and "liking", and were able to knock out either circuit without affecting the other (it was actually kind of cute - they measured the number of times the rats licked their lips as a proxy for "liking", though of course they had a highly technical rationale behind it).
One could come up with a story about how people are motivated to act selfishly but enjoy acting morally, which allows them to tell other
...A University of Michigan study analyzed the brains of rats eating a favorite food. They found separate circuits for "wanting" and "liking", and were able to knock out either circuit without affecting the other (it was actually kind of cute - they measured the number of times the rats licked their lips as a proxy for "liking", though of course they had a highly technical rationale behind it).
One could come up with a story about how people are motivated to act selfishly but enjoy acting morally, which allows them to tell other
...A University of Michigan study analyzed the brains of rats eating a favorite food. They found separate circuits for "wanting" and "liking", and were able to knock out either circuit without affecting the other (it was actually kind of cute - they measured the number of times the rats licked their lips as a proxy for "liking", though of course they had a highly technical rationale behind it).
One could come up with a story about how people are motivated to act selfishly but enjoy acting morally, which allows them to tell other
...A University of Michigan study analyzed the brains of rats eating a favorite food. They found separate circuits for "wanting" and "liking", and were able to knock out either circuit without affecting the other (it was actually kind of cute - they measured the number of times the rats licked their lips as a proxy for "liking", though of course they had a highly technical rationale behind it).
One could come up with a story about how people are motivated to act selfishly but enjoy acting morally, which allows them to tell other
Seems like a pretty large leap from certain simple behaviours of rats to the natural-language meaning of "wanting" and "liking". Far-reaching claims such as this one should have strong evidence. Why not give humans drugs selective for either system and ask them? (Incidentally, at least with the dopamine system, this has been done millions of times ;) The opioids are a bit trickier because activating mu receptors (e.g. by means of opiates) will in turn cause a dopamine surge, too)
(Yes, I should just read the paper for their rationale, but can't be bothered right now...)
Edit: Sorry, editing messed up the formatting. Trying to fix...
Seems like a pretty large leap from certain behaviours of rats to the natural-language meaning of "wanting" and "liking". Far-reaching claims such as this one should have strong evidence. Why not give humans drugs selective for either system and ask them? (Incidentally, at least with the dopamine system, this has been done millions of times ;) The opioids are a bit trickier because activating mu receptors (e.g. by means of opiates) will in turn cause a dopamine surge, too)
(Yes, I should just read the paper for their rationale, but can't be bothered right now...)
One could come up with a story about how people are motivated to act selfishly but enjoy acting morally, which allows them to tell others a story about how virtuous they are while still pursuing their own selfish gain.
Surely that should read "while still pursuing their genes' selfish gain.", because if you do something that makes you less happy, you have not gained...
Related to: Utilons vs. Hedons, Would Your Real Preferences Please Stand Up
And I don't mean that question in the semantic "but what is happiness?" sense, or in the deep philosophical "but can anyone not facing struggle and adversity truly be happy?" sense. I mean it in the totally literal sense. Are wireheads having fun?
They look like they are. People and animals connected to wireheading devices get upset when the wireheading is taken away and will do anything to get it back. And it's electricity shot directly into the reward center of the brain. What's not to like?
Only now neuroscientists are starting to recognize a difference between "reward" and "pleasure", or call it "wanting" and "liking". The two are usually closely correlated. You want something, you get it, then you feel happy. The simple principle behind our entire consumer culture. But do neuroscience and our own experience really support that?
It would be too easy to point out times when people want things, get them, and then later realize they weren't so great. That could be a simple case of misunderstanding the object's true utility. What about wanting something, getting it, realizing it's not so great, and then wanting it just as much the next day? Or what about not wanting something, getting it, realizing it makes you very happy, and then continuing not to want it?
The first category, "things you do even though you don't like them very much" sounds like many drug addictions. Smokers may enjoy smoking, and they may want to avoid the physiological signs of withdrawl, but neither of those is enough to explain their reluctance to quit smoking. I don't smoke, but I made the mistake of starting a can of Pringles yesterday. If you asked me my favorite food, there are dozens of things I would say before "Pringles". Right now, and for the vast majority of my life, I feel no desire to go and get Pringles. But once I've had that first chip, my motivation for a second chip goes through the roof, without my subjective assessment of how tasty Pringles are changing one bit.
Think of the second category as "things you procrastinate even though you like them." I used to think procrastination applied only to things you disliked but did anyway. Then I tried to write a novel. I loved writing. Every second I was writing, I was thinking "This is so much fun". And I never got past the second chapter, because I just couldn't motivate myself to sit down and start writing. Other things in this category for me: going on long walks, doing yoga, reading fiction. I can know with near certainty that I will be happier doing X than Y, and still go and do Y.
Neuroscience provides some basis for this. A University of Michigan study analyzed the brains of rats eating a favorite food. They found separate circuits for "wanting" and "liking", and were able to knock out either circuit without affecting the other (it was actually kind of cute - they measured the number of times the rats licked their lips as a proxy for "liking", though of course they had a highly technical rationale behind it). When they knocked out the "liking" system, the rats would eat exactly as much of the food without making any of the satisifed lip-licking expression, and areas of the brain thought to be correlated with pleasure wouldn't show up in the MRI. Knock out "wanting", and the rats seem to enjoy the food as much when they get it but not be especially motivated to seek it out. To quote the science1:
Pleasure and desire circuitry have intimately connected but distinguishable neural substrates. Some investigators believe that the role of the mesolimbic dopamine system is not primarily to encode pleasure, but "wanting" i.e. incentive-motivation. On this analysis, endomorphins and enkephalins - which activate mu and delta opioid receptors most especially in the ventral pallidum - are most directly implicated in pleasure itself. Mesolimbic dopamine, signalling to the ventral pallidum, mediates desire. Thus "dopamine overdrive", whether natural or drug-induced, promotes a sense of urgency and a motivation to engage with the world, whereas direct activation of mu opioid receptors in the ventral pallidum induces emotionally self-sufficient bliss.
The wanting system is activated by dopamine, and the liking system is activated by opioids. There are enough connections between them that there's a big correlation in their activity, but the correlation isn't one and in fact activation of the opioids is less common than the dopamine. Another quote:
It's relatively hard for a brain to generate pleasure, because it needs to activate different opioid sites together to make you like something more. It's easier to activate desire, because a brain has several 'wanting' pathways available for the task. Sometimes a brain will like the rewards it wants. But other times it just wants them.
So you could go through all that trouble to find a black market brain surgeon who'll wirehead you, and you'll end up not even being happy. You'll just really really want to keep the wirehead circuit running.
Problem: large chunks of philosophy and economics are based upon wanting and liking being the same thing.
By definition, if you choose X over Y, then X is a higher utility option than Y. That means utility represents wanting and not liking. But good utilitarians (and, presumably, artificial intelligences) try to maximize utility (or do they?). This correlates contingently with maximizing happiness, but not necessarily. In a worst-case scenario, it might not correlate at all - two possible such scenarios being wireheading and an AI without the appropriate common sense.
Thus the deep and heavy ramifications. A more down-to-earth example came to mind when I was reading something by Steven Landsburg recently (not recommended). I don't have the exact quote, but it was something along the lines of:
According to a recent poll, two out of three New Yorkers say that, given the choice, they would rather live somewhere else. But all of them have the choice, and none of them live anywhere else. A proper summary of the results of this poll would be: two out of three New Yorkers lie on polls.
This summarizes a common strain of thought in economics, the idea of "revealed preferences". People tend to say they like a lot of things, like family or the environment or a friendly workplace. Many of the same people who say these things then go and ignore their families, pollute, and take high-paying but stressful jobs. The traditional economic explanation is that the people's actions reveal their true preferences, and that all the talk about caring about family and the environment is just stuff people say to look good and gain status. If a person works hard to get lots of money, spends it on an iPhone, and doesn't have time for their family, the economist will say that this proves that they value iPhones more than their family, no matter what they may say to the contrary.
The difference between enjoyment and motivation provides an argument that could rescue these people. It may be that a person really does enjoy spending time with their family more than they enjoy their iPhone, but they're more motivated to work and buy iPhones than they are to spend time with their family. If this were true, people's introspective beliefs and public statements about their values would be true as far as it goes, and their tendency to work overtime for an iPhone would be as much a "hijacking" of their "true preferences" as a revelation of them. This accords better with my introspective experience, with happiness research, and with common sense than the alternative.
Not that the two explanations are necessarily entirely contradictory. One could come up with a story about how people are motivated to act selfishly but enjoy acting morally, which allows them to tell others a story about how virtuous they are while still pursuing their own selfish gain.
Go too far toward the liking direction, and you risk something different from wireheading only in that the probe is stuck in a different part of the brain. Go too far in the wanting direction, and you risk people getting lots of shiny stuff they thought they wanted but don't actually enjoy. So which form of good should altruists, governments, FAIs, and other agencies in the helping people business respect?
Sources/Further Reading:
1. Wireheading.com, especially on a particular University of Michigan study
2. New York Times: A Molecule of Motivation, Dopamine Excels at its Task