Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

muflax comments on The curse of identity - Less Wrong

125 Post author: Kaj_Sotala 17 November 2011 07:28PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (298)

You are viewing a single comment's thread.

Comment author: [deleted] 17 November 2011 01:48:17PM *  2 points [-]

I don't understand why you call this a problem. If I understand you correctly, you are proposing that people constantly and strongly optimize to obtain signalling advantages. They do so without becoming directly aware of it, which further increases their efficiency. So we have a situation where people want something and choose an efficient way to get it. Isn't that good?

More directly, I'm confused how you can look at an organism, see that it uses its optimization power in a goal-oriented and efficient way (status gains in this case) and call that problematic, merely because some of these organisms disagree that this is their actual goal. What would you want them to do - be honest and thus handicap their status seeking?

Say you play many games of Diplomacy against an AI, and the AI often promised you to be loyal, but backstabbed you many times to its advantage. You look at the AI's source code and find out that it has backstabbing as a major goal, but the part that talks to people isn't aware of that so that it can lie better. Would you say that the AI is faulty? That it is wrong and should make the talking module aware of its goals, even though this causes it to make more mistakes and thus lose more? If not, why do you think humans are broken?

Comment author: Oscar_Cunningham 17 November 2011 02:33:26PM 4 points [-]

I want people to work toward noble efforts like charity work, but don't care much about whether they attian high status. So it's useful to aid the bit of their brain that wants to do what I want it to do.

People who care about truth might spot that part of your AI's brain wants to speak the truth, and so they will help it do this, even though this will cost it Diplomacy games. They do this because they care more about truth than Diplomacy.

Comment author: TheOtherDave 17 November 2011 03:04:18PM 1 point [-]

By "caring about truth" here do you mean wanting systems to make explicit utterances that accurately reflect their actual motives? E.g., if X is a chess-playing AI that doesn't talk about what it wants at all, just plays chess, would a person who "cares about truth" would also be motivated to give X the ability and inclination to talk about its goals (and do so accurately)?

Or wanting systems not to make explicit utterances that inaccurately reflect their actual motives? E.g., a person who "cares about truth" might also be motivated to remove muflax's AI's ability to report on its goals at all? (This would also prevent it from winning Diplomacy games, but we've already stipulated that isn't a showstopper.)

Comment author: Oscar_Cunningham 17 November 2011 06:38:35PM 0 points [-]

I intended both (i.e. that they wanted accurate statements to be uttered and no inaccurate statements) but the distinction isn't important to my argument, which was just that they want what they want.

Comment author: CG_Morton 18 November 2011 06:12:05PM -1 points [-]

I don't see how this is admirable at all. This is coercion.

If I work for a charitable organization, and my primary goal is to gain status and present an image as a charitable person, then efforts by you to change my mind are adversarial. Human minds are notoriously malleable, so it's likely that by insisting I do some status-less charity work you are likely to convince me on a surface level. And so I might go and do what you want, contrary to my actual goals. Thus, you have directly harmed me for the sake of your goals. In my opinion this is unacceptable.

Comment author: ciphergoth 17 November 2011 02:15:05PM 4 points [-]

It's a problem from the point of view of that part of me that actually wants to achieve large scale strategic goals.

Comment author: [deleted] 17 November 2011 02:25:08PM 2 points [-]

Honest question: how do you know you have these goals? Presumably they don't manifest in actual behavior, or you wouldn't have a problem. If Kaj's analysis is right, shouldn't you assume that the belief of having these goals is part of your (working) strategy to gain certain status? Would you accept the same argument if Bruce made it?

Comment author: ciphergoth 17 November 2011 02:28:00PM 10 points [-]

Put it this way, if there was a pill that I believed would cause me to effectively have that goal, in a way that was compatible with a livable life, I would take it.

Comment author: Vladimir_Nesov 17 November 2011 03:39:43PM 7 points [-]

If I understand you correctly, you are proposing that people constantly and strongly optimize to obtain signalling advantages. They do so without becoming directly aware of it, which further increases their efficiency.

"Efficiency" at achieving something other than what you should work towards is harmful. If it's reliable enough, let your conscious mind decide if signaling advantages or something else is what you should be optimizing. Otherwise, you let that Blind Idiot Azathoth pick your purposes for you, trusting it more than you trust yourself.

Comment author: XiXiDu 17 November 2011 05:02:04PM *  1 point [-]

"Efficiency" at achieving something other than what you should work towards is harmful. ... Otherwise, you let that Blind Idiot Azathoth pick your purposes for you, trusting it more than you trust yourself.

The purpose of solving friendly AI is to protect the purposes picked for us by the blind idiot god.

Comment author: Vladimir_Nesov 17 November 2011 06:28:56PM *  9 points [-]

Our psychological adaptations are not our purposes, we don't want to protect them, even though they contribute to determining what it is we want to protect. See Evolutionary Psychology.

Comment author: Kaj_Sotala 17 November 2011 02:51:20PM *  6 points [-]

For one, status-seeking is a zero sum game and only indirectly causes overall gains. The world would be a much better place if people actually cared about things like saving the world or even helping others, and put a little thought to it.

Also, mismatches between our consciously-held goals and our behavior cause plenty of frustration and unhappiness, like in the case of the person who keeps stressing out because their studies don't progress.

Comment author: Vaniver 17 November 2011 07:08:16PM 1 point [-]

For one, status-seeking is a zero sum game and only indirectly causes overall gains. The world would be a much better place if people actually cared about things like saving the world or even helping others, and put a little thought to it.

If I actually cared about saving the world and about conserving my resources, it seems like I would choose some rate of world-saving A.

If I actually cared about saving the world, about conserving my resources, and the opinion of my peers, it seems like I would choose some rate of world-saving B. For reasonable scenarios, B would be greater than A because I can also get respect from my peers, and when you raise demand and keep supply constant quantity supplied increases.

That is, I understand that status causes faking behavior that's a drain. (Status conflicts also lower supply, but it's not clear how much.) I don't think it's clear that the mechanism of status-seeking conflicts with actually caring about other goals or detracts from them on net.

Comment author: Jonathan_Graehl 17 November 2011 06:50:27PM 0 points [-]

I'm sure you've considered that "X is a 0 sum game" doesn't always mean that you should unilaterally avoid playing that game entirely. It does mean you'll want to engineer environments where X taxes at a lower rate.

Comment author: [deleted] 17 November 2011 03:35:14PM -2 points [-]

For one, status-seeking is a zero sum game and only indirectly causes overall gains.

But if status-seeking is what you really want, as evidenced by your decisions, how can you say it's bad that you do it? Can't I just go and claim any goal you're not optimizing for as your "real" goal you "should" have? Alternatively, can't I claim that you only want us to drop status-seeking to get rid of the competition? Where's your explanatory power?

Comment author: Kaj_Sotala 17 November 2011 04:04:44PM *  2 points [-]

But if status-seeking is what you really want, as evidenced by your decisions, how can you say it's bad that you do it?

By the suffering it causes, and also by the fact that when I have realized that I'm doing it, I've stopped doing (that particular form of) it.

Comment author: XiXiDu 17 November 2011 03:08:10PM *  -1 points [-]

For one, the world would be a much better place if people actually cared about things like saving the world or even helping others, and put a little thought to it.

Why do you want to save the world? To allow people, humans, to do what they like to do for much longer than they would otherwise be able to. Status-seeking is one of those things that people are especially fond of.

Ask yourself, would you have written this post after a positive Singularity? Would it matter if some people were engaged in status games all day long?

What you are really trying to tell people is that they want to help solving friendly AI because it is universally instrumentally useful.

In case you want to argue that status-seeking is bad, no matter what, under any circumstances, you have to explain why that is so. And if you are unable to ground utility in something that is physically measurable, like the maximization of certain brain states, then I don't think that you could convincingly demonstrate it to be a relatively undesirable human activity.

Comment author: Kaj_Sotala 17 November 2011 04:07:57PM *  3 points [-]

Umm. Sure, status-seeking may be fine once we have solved all possible problems anyway and we're living in a perfect utopia. But that's not very relevant if we want to discuss the world as it is today.

Comment author: XiXiDu 17 November 2011 04:59:43PM -2 points [-]

But that's not very relevant if we want to discuss the world as it is today.

It is very relevant, because the reason why we want to solve friendly AI in the first place is to protect our complex values given to us by the Blind Idiot God.

Comment author: Kaj_Sotala 17 November 2011 05:20:17PM 0 points [-]

If we're talking about Friendly AI design, sure. I wasn't.

Comment author: Grognor 17 November 2011 02:59:52PM *  1 point [-]

Would you say that the AI is faulty?

Yes. It might be doing exactly what it was designed to do, but its designer was clearly stupid or cruel and had different goals than I'd prefer the AI to have.

Extrapolate this to humans. Humans wouldn't care so much about status if it weren't for flaws like scope insensitivity, self-serving bias, etc., as well as simply poor design "goals".

Comment author: [deleted] 17 November 2011 03:38:32PM -1 points [-]

Yes. It might be doing exactly what it was designed to do, but its designer was clearly stupid or cruel and had different goals than I'd prefer the AI to have.

Where are you getting your goals from? What are you, except your design? You are what Azathoth build. There is no ideal you that you should've become, but which Azathoth failed to make.

Comment author: Grognor 17 November 2011 03:46:36PM 1 point [-]

Azathoth designed me with conflicting goals. Subconsciously, I value status, but if I were to take a pill that made me care entirely about making the world better and nothing else, I would. Just because "evolution" built that into me doesn't make it bad, but it definitely did not give me a coherent volition. I have determined for my self which parts of humanity's design are counterproductive, based on the thousand shards of desire.

Comment author: fubarobfusco 18 November 2011 05:11:43AM 2 points [-]

if I were to take a pill that made me care entirely about making the world better and nothing else, I would.

Would you sign up to be tortured so that others don't suffer dust specks?

("If we are here to make others happy, what are the others here for?")

Comment author: [deleted] 19 November 2011 02:44:21AM 1 point [-]

A better analogy would be asking about a pill that caused pain asymbolia.

Comment author: Grognor 18 November 2011 10:45:09AM 1 point [-]

Yes.

Comment author: TheOtherDave 17 November 2011 02:28:15PM *  0 points [-]

Does your expression of confusion here allow you to challenge the OP's implicit premise that their failure to optimize for the goals they explicitly endorse rather than optimize for signalling is a problem, without overtly signalling such a challenge and thereby potentially subjecting yourself to reprisal?

If so, are you aware of the fact?

If you aren't, is it real confusion or not?

I'm not sure that question means anything, any more than the question of whether the OP has a real problem does. If you are aware of it and similarly aware of your expression of confusion being disingenuous, then by convention we say you're not really confused; if you aren't, we say you are. We can make similar decisions about whether to say the OP has a real problem or not.

Comment author: [deleted] 17 November 2011 02:48:40PM 1 point [-]

Not sure if I understand you correctly; let me try to rephrase it.

You are saying it is possible I claim confusion because I expect to gain status (contrarian status maybe?), as per Kaj's post, instead of being actually confused? Sure. I considered it, but rejected it because that weakens the explanatory power of status signalling. (I'm not sure if I agree with the signalling assumption, but let's for the sake of the argument.)

A real problem exists if an agent tries to optimize for a goal, but sucks at it. It's own beliefs are not relevant (unless the goal is about its beliefs). If Kaj is correct, then humans are optimizing for status, but sacrifice some accuracy of their self-modelling power. It seems to work out, so how is this problematic?

In other words, an agent wants X. It models itself to get better at getting X. The self-model is, among other things, the basis for communication with other agents. The self-model is biased to model itself wrongly as wanting Y. It is advantageous for the agent to be seen as wanting Y, not X. The inaccurate self-model doesn't cause substantial damage to its ability to pursue X, and it is much easier for the self-model to be biased than to lie. This setup sounds like a feature, not like a bug. If you observed it in an organism that wasn't you, wasn't even human, would you say the organism has a problem?

Comment author: TheOtherDave 17 November 2011 03:25:28PM 1 point [-]

I'm saying it's possible that what's really going on is that you think Kaj is mistaken when he calls the situation a problem... that he has made an error. But rather than say "Kay, you are mistaken, you have made an error" you say "Kaj, I'm confused." And that the reason you do this is because to say "Kay, you are mistaken, you have made an error" is to challenge Kaj's status, which would potentially subject you to reprisals.

It's possible that you're doing this deliberately. In that case, by convention we would say you aren't really confused. (We might also, by convention, say you're being dishonest, or say that you're being polite, or say various other things.)

It's also possible that you are doing this unknowingly... that you are generating the experience of confusion so as to protect yourself from reprisal. In this case, it's less clear whether convention dictates that we say you are "really confused" or "not really confused". I would say it doesn't at all matter; the best thing to do is not ask that question. (Or, if we must, to agree on a convention as to which one it is.)

In any case, I agree with your basic point about goal optimization, I just think talking about whether it constitutes a "real problem" or not contributes nothing to the discussion, much like I think talking about whether you're experiencing "real confusion" in the latter case contributes nothing.

That said, you are completely ignoring the knock-on effects of lying (e.g., increasing the chances that I will be perceived as lying in a social context where being perceived in this way has costs).

Comment author: [deleted] 17 November 2011 03:51:47PM 3 points [-]

Ah, then I misunderstood you. Yes, I believe Kaj is wrong, either in calling this a problem or in the assumption that status-seeking is a good explanation of it. However, based on past contributions, I think that Kaj has thought a lot about this and it is more likely that I'm misunderstanding him than that he is wrong. Thus my expressed confusion. If further discussion fails to clear this up, I will shift to assuming that he is simply wrong.