This is a big problem with utilitarianism, period. Status is a problem. So are other-regarding values ("I want my neighbors to be Christian.") You're in deep trouble trying to reconcile differing values globally. (Which is why, for myself, I'm not messing with it; I'm starting to believe in the idea of cultivating our own gardens.)
That said, I think this is a distracting and inaccurate example. Status and stuff aren't the only things we value, and I don't see that they split by gender in the way you say. Personally, I don't want to be Elizabeth Taylor.
Your initial observation is interesting; your analysis is somewhat problematic.
First, there is, as mentioned elsewhere, a distinct misconception of status that seems to recur here. Status is not a one-dimensional line upon which each person can be objectively placed. Status is multi-dimensional. For example, your person with a second-rate Faberge egg collection may simply decide that what really counts is how nice someone's roses are, and thus look at better Faberge eggs with sympathy (they should have spent all this money on roses!) rather than jealousy. By competing on multiple dimensions, it's possible for many people to win, potentially even everyone.
You are also using a non-representative sample. The type of person who's likely to become successful through celebrity seems disproportionately likely to be status-obsessed. (This is likely also true of people who inherit fortunes, as they have few other ways to distinguish their self-worth.) Generalizing about the ability of an AI to serve all people based on these few may be inappropriate. This is even more true when we consider that a world with an FAI may look very different in terms of how people are raised and educated.
Furth...
Actually, there's a pretty easy fix for the status problem, because we actually do have independently-valuable feelings that the ability to boss people around correspond to -- feelings like respectability, importance, and pride being among the main ones.
(Often, however, people who are outwardly concerned with their position are actually motivated by perceived threats to safety or affiliation instead -- believing that they are unworthy or unlovable unless they're important enough, or that you're only safe if you're in charge, etc.)
Anyway, your analysis here (as with many others on LW) conflates feelings of status with some sort of actual position in some kind of dominance hierarchy. But this is a classification error. There are people who feel quite respectable, important, and proud, without needing to outwardly be "superior" in some fashion. One's estimate of one's respectability, prideworthiness, or importance in the scheme of things is not intrinsically linked to any value scheme other than one's own.
Given that the idea of CEV is to get to what the more-grown, wiser version of "you" would choose for your value scheme, it's pretty much a given that unsupport...
Just one minor quibble: I think you should have put the explanation for why you chose the male/female examples ("The male/female distinction isn't rigid; it just helped organize the data in a way that made this distinction pop out for me") much earlier in the post. Since agreeing or disagreeing with your generalization has nothing to do with agreeing or disagreeing with your conclusion, you should say so before the beginning of the second section.
"You may argue that the extremely wealthy and famous don't represent the desires of ordinary humans. I say the opposite: Non-wealthy, non-famous people, being more constrained by need and by social convention, and having no hope of ever attaining their desires, don't represent, or even allow themselves to acknowledge, the actual desires of humans."
I have a huge problem with this statement. This is taking one subset of the population where you can measure what they value by their actions, and saying without evidence that they represent the general population whom you can't measure because resources limit the ability of their actions to reflect their values.
You are assuming that the experience of being rich or being famous doesn't change ones values.
I suspect that the value of reclusion for instance is a direct result of being so famous that one is hounded in public, and that a relatively unknown middle class male wouldn't place near as much value on it.
The problem of value aggregation has at least one obvious lower bound: divide the universe on equal parts, and have each part optimized to given person's preference, including game-theoretic trade between the parts to take into account preferences of each of the parts for the structure of the other parts. Even if values of each person have little in common, this would be a great improvement over status quo.
This discussion of "Friendly AI" is hopelessly anthropomorphic. It seems to be an attempt to imagine what a FAI optimizing the world to given person's values will do, and these are destined to fail, if you bring up specific details, which you do. A FAI is a system expected to do something good, not a specific good thing known in advance. You won't see it coming.
(More generally, see the Fun theory sequence.)
The closest thing I can think of as a solution for the status-FAI is domain-specific status. Let Fred be a high-status pianist, let Jim be a high-status computer engineer, let Sheila be a high-status chef, and let the status ordering shift with context.
But that does seem like a problem for FAI, given the appearance of these preferences.
Human values differ a lot based on the surrounding. If you dump all humans into fairy tale land they might react very differently than now.
You seem to assume that the goal structure of a human is stable. But when i look around I see all kinds of manipulations happening. Religion && Politics being one, advertisement being another. An AI doesn't have to rewire brains the hard way. It could just buy a major entertainment company and implement the values it prefers humans to have into a soap opera, and then buy some ads at the super-bowl. Allowing the ...
It was good to be explicit that these are generalizations. Nonetheless, it was still a mistake to label these two views "female" and "male", rather than the more neutral "positional" and "non-positional". That you notice this correlation is interesting, but not the main point. Given the likely effects, it seems better not to over-emphasize this with your choice nomenclature.
Any links made to this article before this comment will now be broken.
Actually, the title in the URL doesn't matter.
I agree that friendly AI is probably doomed if the goal is to maximize human values. But what if we, as AI designers (I'm not an AI designer, but bear with me), don't care about what you've defined as "human values?" As I've alluded to before, what matters isn't the entire system Phil Goetz, but a specific subsystem. This is an important metaethical point. There are algorithms running in your brain that you have no control over, yet they do things that you simply don't want them to be doing. For example, I don't want to have to purchase fuzz...
Can the AI be Friendly if it creates human-comparable minds with abitrary nonhuman values, provided the resulting homunculi are not, themselves, significantly hostile or in any way self-improving?
If so, I think there's a simple fix for this: have the AI give everyone a volcano lair (or equivalent) and two dozen catgirl minions. That puts all humans solidly in the top 5% of humanoids, which is a nice status kick; they can still compete with other humans for the last few percentiles, but there's no real risk of ending up at the bottom of the heap... unless y...
Well, the "female AI" can create a lot of human-seeming robots to play the low status roles, or something...
Google romance novel formula, and you'll find web pages by romance novel authors patiently explaining that there is no such thing as a romance novel formula.
Funny thing is, google science fiction novel formula, or fantasy novel formula, and you won't find that.
Look up the Harlequin Romance author's guidelines, and you won't find anything formulaic.
I read a book called "Dangerous men, adventurous women", by romance novelists for romance novelists, also to find out what women looked for in romance novels. And not only is there a formula for romance novels; there's a formula for articles about romance novels:
Spend the first half of the article complaining about the idea that romance novels are formulaic.
Spend the second half of the article describing the formula, and warning would-be authors that they won't sell books if they deviate from it.
The formula is basically to teach women as many dysfunctional, self-destructive ideas about romance as possible. Start with a young, never-married, beautiful, rebellious woman. Find her a dangerous, out-of-control, rakish, tall, dark, brooding, handsome man with many faults but a heart of gold, who has extensive sexual experien...
This is 'fictional evidence', but one way to satisfy positional values would be something like the Matrix, with suitably convincing NPCs for people to be 'better' than.
All this stuff about values is giving me a proverbial headache. When the Rapture of the Nerds happens, I'll take the easy way out and become a wirehead, and let the rest of the world actually do things. ;)
Do we need FAI that does as good a job of satisfying human desires as possible, or would an FAI which protects humanity against devastating threats enough?
Even devastating threats can be a little hard to define.... if people want to transform themselves into Something Very Different, is that the end of the human race, or just an extension to human history?
Still, most devastating threats (uFAI, asteroid strike) aren't such a hard challenge to identify.
The "headline" is probably inaccurate. I don't know what metric best measures similarity of value - but by the ones that popped into my head, the various human heavens mostly seem relatively clumped together - compared to what kind of future other creatures might dream of having. It's the close genetic relatedness that does it.
To achieve high status for the max amount of people you could invent more categories in which to achieve status. There is not just one best actress, but many in different types of acting. There can be the best person to make some specific dish. The best dancer of type X. Then you can seclude it by region any come out with many many more things, till everyone is famous who desires so. Alternatively the AI could find out if there is any desire beyond the wish for status and fulfill that instead, or offer a self modification for people that desire status, but desire to change that to something else.
I think a shorter term for what you're describing is positionality, the state where the quality of the good depends on its ranking relative to other goods. The problem you're claiming is that women's values are positional (making them non-mutually-satisfiable), while men's aren't (making them mutually-satisfiable).
But in any case, thanks for saving me the ordeal of telling women how the positionality of their values throws a big monkey wrench in everything ;-)
But, since the FAI's top-level goal is just to preserve human top-level goals, it would be pointless to make a lot of fuss making sure the FAI held its own top-level goals constant, if you're going to "correct" human goals first.)
Well, part of the sleight-of-hand here is that the FAI preserves the goals we would have if we were wiser, better people.
...If changing top-level goals is allowed in this instance, or this top-level goal is considered "not really a top-level goal", I would become alarmed and demand an explanation of how a FAI
Reality check: evolutionary theory suggests people's desires should be nailed down as hard as possible to those things that lead to raising good quality babies. Almost 7 billion humans shows how well this theory works.
So: men can be expected to desire status to the extent that it increases their access to young, fertile mates - while women can be expected to desire attention to the extent that it gives them access to a good selection of prospective partners and their genes.
The maternal instict is strong - and it has little to do with attention - and a lot...
Phil, you make a good point here. However, note that when the people you are talking about are extrapolated in a CEV-like FAI, they will also understand this point. Elizabeth Taylor will understand that not everyone can have high status, and therefore people will have to settle on some solution, which could involve "fake people" as low status entities, editing of everyone's top-level goals (a sort of decision-theoretic compromise), etc.
One corollary of this is that for existing high status people, CEV would be a terrible thing. [EDIT: I was thin...
As evidence for someone like this, consider dictators like Kim Jong Il. Opening up North Korea would result in much greater wealth for both him and his people, but it comes with a loss of power and status for Kim Jong. No one thinks he's opening those borders anytime soon. The comparison isn't as drastic, however - Kim Jong's comforts are probably only a decade or two behind modern (I'm speculating).
George Hamilton's autobiography Don't Mind if I Do, and the very similar book by Bob Evans, The Kid Stays in the Picture, give a lot of insight into human nature and values. For instance: What do people really want? When people have the money and fame to travel around the world and do anything that they want, what do they do? And what is it that they value most about the experience afterward?
You may argue that the extremely wealthy and famous don't represent the desires of ordinary humans. I say the opposite: Non-wealthy, non-famous people, being more constrained by need and by social convention, and having no hope of ever attaining their desires, don't represent, or even allow themselves to acknowledge, the actual desires of humans.
I noticed a pattern in these books: The men in them value social status primarily as an ends to a means; while the women value social status as an end in itself.
This is a generalization; but, at least at the very upper levels of society depicted in these books, and a few others like them that I've read, it's frequently borne out. (Perhaps a culture chooses celebrities who reinforce its stereotypes.) Women and men alike appreciate expensive cars and clothing. But the impression I get is that the flamboyantly extravagant are surprisingly non-materialistic. Other than food (and, oddly, clothing), the very wealthy themselves consistently refer to these trappings as things that they need in order to signal their importance to other people. They don't have an opinion on how long or how tall a yacht "ought" to be; they just want theirs to be the longest or tallest. The persistent phenomenon whereby the more wealthy someone appears, the more likely they are to go into debt, is not because these people are too stupid or impulsive to hold on to their money (as in popular depictions of the wealthy, eg., A New Leaf) . It's because they are deliberately trading monetary capital for the social capital that they actually desire (and expect to be able to trade it back later if they wish to, even making a profit on the "transaction", as Donald Trump has done so well).
With most of the women in these books, that's where it ends. What they want is to be the center of attention. They want to walk into a famous night-club and see everyone's heads turn. They want the papers to talk about them. They want to be able to check into a famous hotel at 3 in the morning and demand that the head chef be called at home, woken up, and brought in immediately to cook them a five-course meal. Some of the women in these stories, like Elizabeth Taylor, routinely make outrageous demands just to prove that they're more important than other people.
What the men want is women. Quantity and quality. They like social status, and they like to butt heads with other men and beat them; but once they've acquired a bevy of beautiful women, they are often happy to retire to their mansion or yacht and enjoy them in private for a while. And they're capable of forming deep, private attachments to things, in a way the women are less likely to. A man can obsess over his collection of antique cars as beautiful things in and of themselves. A woman will not enjoy her collection of Faberge eggs unless she has someone to show it to. (Preferably someone with a slightly less-impressive collection of Faberge eggs.) Reclusive celebrities are more likely to be men than women.
Some people mostly like having things. Some people mostly like having status. Do you see the key game-theoretic distinction?
Neither value is very amenable to the creation of wealth. Give everybody a Rolls-Royce; and the women still have the same social status, and the men don't have any more women. But the "male" value is more amenable to it. Men compete, but perhaps mainly because the distribution of quality of women is normal. The status-related desires of the men described above are, in theory, capable of being mutually satisfied. The women's are not.
No real person implements pure mutually-satisfiable or non-mutually-satisfiable values. I have not done a study or taken a survey, and don't claim that these views correlate with sex in general. I just wanted to make accessible the evidence I saw that these two types of values exist in humans. The male/female distinction isn't what I want to talk about; it just helped organize the data in a way that made this distinction pop out for me. I could also have told a story about how men and women play sports, and claim that men are more likely to want to win (a non-mutually-satisfiable value), and women are more likely to just want to have fun (a mutually-satisfiable value). Let's not get distracted by sexual politics. I'm not trying to say something about women or about men; I'm trying to say something about FAI.
I will now rename them "non-positional" and "positional" (as suggested by SilasBarta and wnoise), where "non-positional" means assigning a value to something from category X according to its properties, and "positional" means assigning a new value to something from category X according to the rank of its non-positional value in the set of all X (non-mutually-satisfiable).
Now imagine two friendly AIs, one non-positional and one positional.
The non-positional FAI has a tough task. It wants to give everyone what it imagines they want.
But the positional FAI has an impossible task. It wants to give everyone what it is that it thinks they value, which is to be considered better than other people, or at least better than other people of the same sex. But it's a zero-sum value. It's very hard to give more status to one person without taking the same amount of status away from other people. There might be some clever solution involving sending people on trips at relativistic speeds so that the time each person is high-status seems longer to them than the time they are low-status, or using drugs to heighten their perceptions of high status and diminish the pain of low status. For an average utilitarian, the best solution is probably to kill off everyone except one man and one woman. (Painlessly, of course.)
A FAI trying to satisfy one of these preferences would take society in a completely different direction than a FAI trying to satisfy the other. From the perspective of someone with the job of trying to satisfy these preferences for everyone, they are as different as it is possible for preferences to be, even though they are taken (in the books mentioned above) from members of the same species at the same time in the same place in the same strata of the same profession.
This is not a problem that can be resolved by popping up a level. If you say, "But what people who want status REALLY want is something else that they can use status to obtain," you're just denying the existence of status as a value. It's a value. When given the chance to either use their status to attain something else, or keep pressing the lever that gives them a "You've got status!" hit, some people choose to keep pressing the lever.
If you claim that these people have formed bad habits, and improperly short-circuited a connection from value to stimulus; and can be re-educated to instead see status as a means, rather than as an ends... I might agree with you. But you'd make a bad, unfriendly AI. If there's one thing FAIers have been clear about, it's that changing top-level goals is not allowed. (That's usually said with respect to the FAI's top-level goals, not wrt the human top-level goals. But, since the FAI's top-level goal is just to preserve human top-level goals, it would be pointless to make a lot of fuss making sure the FAI held its own top-level goals constant, if you're going to "correct" human goals first.)
If changing top-level goals is allowed in this instance, or this top-level goal is considered "not really a top-level goal", I would become alarmed and demand an explanation of how a FAI distinguishes such pseudo-top-level-goals from real top-level goals.
If you believe that computer programs can be conscious, then unless you have a new philosophical position that you haven't told anyone about, you believe that consciousness can be a by-product of computation. This means that the formal, computational properties of peoples' values are not just critical, they're the only thing that matters. This means that there is no way to abstract away the bad property of being zero-sum from a value without destroying the value.
In other words, it isn't valid to analyze the sensations that people get when their higher status is affirmed by others, and then recreate those sensations directly in everyone, without anyone needing to have low status. If you did that, I can think of only 3 possible interpretations of what you would have done, and I find none of them acceptable:
This discussion has uncovered several problems for an AI trying to give people what they value without changing what they value. In increasing order of importance: