So which form of good should altruists, governments, FAIs, and other agencies in the helping people business respect?
Governments should give people what people say they want, rather than giving people what the governments think will make people happier, whenever they can't do both. But this is not because it's intrinsically better for people to get what they want than to get what makes them happier (it isn't), it's because people will resent what they percieve as paternalism in governments and because they won't pay taxes and obey laws in general if they resent their governments. Without taxes and law-abiding citizens, there will not be much happiness in the long run. So, simply for the sake of happiness maximizing, governments should (except, possibly, in some very, very extreme situations) just do what people want.
It's understandable that people want others to respect what they want, rather than wanting others to try to make them happier: even if we are not all experts ourselves on what will make us happier (not all people know about happiness research), we may need to make our own mistakes in order to really come to trust that what people say works works and that what people say doesn't work doesn't work. Also, some of government's alleged benevolent paternalism "for people's own good" (for example Orwellian surveillance in the name of the "war on terror") may even be part of a plan to enslave or otherwise exploit the people. We may know these things subconsciously, and that may explain why some of us are so reluctant to conclude that what we want has no intrinsic value and that pleasure is the only thing that has intrinsical value. The instrumental value of letting people have what they want (rather than paternalistically giving them what some government thinks they need) is so huge, that saying it has "mere" instrumental value feels like neglecting how huge a value it has. However, it doesn't really have intrinsic value, it just feels that way, because we are not accostumed to thinking that something that has only instrumental value can have such a huge instrumental value.
For example, freedom of speech is of huge importance, but not primarily because people want it, but primarily because it provides happiness and prevents too much suffering from happening. If it were the case that freedom of speech didn't provide any happiness and didn't prevent any suffering, but people still eagerly wanted it, there would be no point in letting anybody have freedom of speech. However, this would imply either that being denied freedom of speech in no way caused any form of suffering in people, or that, if it caused suffering, then getting freedom of speech wouldn't relieve any of that suffering. That is a hypothetical scenario so hard to imagine that I think the fact that it is so hard to imagine is the reason why people have difficulties accepting the truth that freedom of speech has merely instrumental value.
Related to: Utilons vs. Hedons, Would Your Real Preferences Please Stand Up
And I don't mean that question in the semantic "but what is happiness?" sense, or in the deep philosophical "but can anyone not facing struggle and adversity truly be happy?" sense. I mean it in the totally literal sense. Are wireheads having fun?
They look like they are. People and animals connected to wireheading devices get upset when the wireheading is taken away and will do anything to get it back. And it's electricity shot directly into the reward center of the brain. What's not to like?
Only now neuroscientists are starting to recognize a difference between "reward" and "pleasure", or call it "wanting" and "liking". The two are usually closely correlated. You want something, you get it, then you feel happy. The simple principle behind our entire consumer culture. But do neuroscience and our own experience really support that?
It would be too easy to point out times when people want things, get them, and then later realize they weren't so great. That could be a simple case of misunderstanding the object's true utility. What about wanting something, getting it, realizing it's not so great, and then wanting it just as much the next day? Or what about not wanting something, getting it, realizing it makes you very happy, and then continuing not to want it?
The first category, "things you do even though you don't like them very much" sounds like many drug addictions. Smokers may enjoy smoking, and they may want to avoid the physiological signs of withdrawl, but neither of those is enough to explain their reluctance to quit smoking. I don't smoke, but I made the mistake of starting a can of Pringles yesterday. If you asked me my favorite food, there are dozens of things I would say before "Pringles". Right now, and for the vast majority of my life, I feel no desire to go and get Pringles. But once I've had that first chip, my motivation for a second chip goes through the roof, without my subjective assessment of how tasty Pringles are changing one bit.
Think of the second category as "things you procrastinate even though you like them." I used to think procrastination applied only to things you disliked but did anyway. Then I tried to write a novel. I loved writing. Every second I was writing, I was thinking "This is so much fun". And I never got past the second chapter, because I just couldn't motivate myself to sit down and start writing. Other things in this category for me: going on long walks, doing yoga, reading fiction. I can know with near certainty that I will be happier doing X than Y, and still go and do Y.
Neuroscience provides some basis for this. A University of Michigan study analyzed the brains of rats eating a favorite food. They found separate circuits for "wanting" and "liking", and were able to knock out either circuit without affecting the other (it was actually kind of cute - they measured the number of times the rats licked their lips as a proxy for "liking", though of course they had a highly technical rationale behind it). When they knocked out the "liking" system, the rats would eat exactly as much of the food without making any of the satisifed lip-licking expression, and areas of the brain thought to be correlated with pleasure wouldn't show up in the MRI. Knock out "wanting", and the rats seem to enjoy the food as much when they get it but not be especially motivated to seek it out. To quote the science1:
The wanting system is activated by dopamine, and the liking system is activated by opioids. There are enough connections between them that there's a big correlation in their activity, but the correlation isn't one and in fact activation of the opioids is less common than the dopamine. Another quote:
So you could go through all that trouble to find a black market brain surgeon who'll wirehead you, and you'll end up not even being happy. You'll just really really want to keep the wirehead circuit running.
Problem: large chunks of philosophy and economics are based upon wanting and liking being the same thing.
By definition, if you choose X over Y, then X is a higher utility option than Y. That means utility represents wanting and not liking.
But good utilitarians (and, presumably, artificial intelligences) try to maximize utility(or do they?). This correlates contingently with maximizing happiness, but not necessarily. In a worst-case scenario, it might not correlate at all - two possible such scenarios being wireheading and an AI without the appropriate common sense.Thus the deep and heavy ramifications. A more down-to-earth example came to mind when I was reading something by Steven Landsburg recently (not recommended). I don't have the exact quote, but it was something along the lines of:
This summarizes a common strain of thought in economics, the idea of "revealed preferences". People tend to say they like a lot of things, like family or the environment or a friendly workplace. Many of the same people who say these things then go and ignore their families, pollute, and take high-paying but stressful jobs. The traditional economic explanation is that the people's actions reveal their true preferences, and that all the talk about caring about family and the environment is just stuff people say to look good and gain status. If a person works hard to get lots of money, spends it on an iPhone, and doesn't have time for their family, the economist will say that this proves that they value iPhones more than their family, no matter what they may say to the contrary.
The difference between enjoyment and motivation provides an argument that could rescue these people. It may be that a person really does enjoy spending time with their family more than they enjoy their iPhone, but they're more motivated to work and buy iPhones than they are to spend time with their family. If this were true, people's introspective beliefs and public statements about their values would be true as far as it goes, and their tendency to work overtime for an iPhone would be as much a "hijacking" of their "true preferences" as a revelation of them. This accords better with my introspective experience, with happiness research, and with common sense than the alternative.
Not that the two explanations are necessarily entirely contradictory. One could come up with a story about how people are motivated to act selfishly but enjoy acting morally, which allows them to tell others a story about how virtuous they are while still pursuing their own selfish gain.
Go too far toward the liking direction, and you risk something different from wireheading only in that the probe is stuck in a different part of the brain. Go too far in the wanting direction, and you risk people getting lots of shiny stuff they thought they wanted but don't actually enjoy. So which form of good should altruists, governments, FAIs, and other agencies in the helping people business respect?
Sources/Further Reading:
1. Wireheading.com, especially on a particular University of Michigan study
2. New York Times: A Molecule of Motivation, Dopamine Excels at its Task
3. Slate: The Powerful and Mysterious Brain Circuitry...
4. Related journal articles (1, 2, 3)