I could have got that child sponsored. I could have kept my job, and Mary could have stopped crying that evening. She’d have thanked me for coming by, and after I left she would have cuddled on the couch with her new Sponsor Child, tears drying as she found hope in the world.
But I didn’t do it. Instead I apologized for interrupting her grief, and left.
Because I am not a Meat Fucker.
Translation: you fell prey to a different Dark meme, one that linked your behavior to a perception of status, which you then pursued at the expense of real utility for at least two real people besides yourself.
In other words, your vaunted "honesty" actually equals nothing more than selfish egoism: you prefer to pride yourself on it more than you prefer actually helping people.
That is, the rhetoric around honesty you're using is nothing more than the creed of the Confessor ("the ultimate sin is the exercise of command"), which is equally countered and mirrored by that of the Kiritsugu (who would see you as flawed for being "one who refuses to help").
It's not obvious that either of these is a truly correct stance, vs. just being different from each other. Neither post...
This is a revealing story about the double-binds in influence and persuasion. To me, your hypothetical form of influence probably would have been "Dark Arts," but not just for the reasons you describe.
Actually, I'd prefer to taboo the term "Dark Arts" for a while, because I feel it's poisoned the well on the subject of influence and persuasion (like the word "manipulation"). The problem with the term "Dark Arts" is that it conflates both ethical and unethical forms of influence, and creates an "Ugh Field" around the subject of influence in general. Let's talk about how influence and persuasion work, and then later we will decide what's ethical and what's unethical.
The irony of a post criticizing the Dark Arts is that this post is full of rhetoric and persuasion itself, whether intentionally or unintentionally. I'd like to examine some of the language you use in this post ("Dark Arts", "Jedi Mind Tricks", "hacking", "Murder Pill"). In general, I consider these forms of emotive language about influence and persuasion to inhibit understanding these topics.
...When you take advantage of the Dark A
There was a step somewhere in here I missed. You're clear that these tactics violate your morals, and you specifically do not allege that learning about them will help us avoid them (and nor will anything else). What's your purpose in teaching them?
Clarification: if you walk into the leather shop, with $300 burning a hole in your pocket, and see a nice jacket - and the guy behind the counter knows what he's doing (and gives enough of a shit) you will walk out of their, happy with your new jacket.
Saying there's no defense whatsoever was a bit of an exagerration; it's true that some people are more resistant than others (though not infinitely resistant), but even if you're weak you can always avoid the situations in the first place.
Having a blanket policy of "I don't do anything at the door" is highly effective.
It's sort of like a bullet; you can't resist it, but you can avoid it [assuming for the sake of analogy you ignore kevlar].
You know what? I will write a piece on defence, once I'm done with the other pieces in the series. Hopefully what I was trying to convey will be clearer at that point.
As for my purpose - 1) it's fun; this is stuff I've thought a lot about, and the community here is smart enough to understand it, 2) I get LW points and ego validation for writing it, and 3) the information's already out there, I figure it might as well be in the hands of the Good Guys.
This is a followup to the thread you linked, but I'll mention it here because it may be relevant to the current conversation.
I spent a little while with TheValliant (offsite) trying to pin down a definition of "manipulation"--specifically, to divide it from acceptable forms of nonverbal communication. The facets which came to mind immediately were "trying to get someone to do something they otherwise wouldn't have," which is clearly too broad, and "displaying exaggerated emotion," which is sometimes valid (when you're trying to communicate an emotion and your natural expression is too subtle). We got from there to "trying to make someone feel an emotion they otherwise wouldn't have," which still isn't right, because it covers gestures of affection.
What we eventually settled on was this: "Using emotion to bypass someone's normal decision-making process." That is, creating emotions in someone else for the purpose of getting them to do something. This phrasing also makes it pretty clear why we find it abhorrent: it's opening a back door into someone else's brain, and about as invasive as that makes it sound.
The reason I felt a need to pin it down is that TheValliant, like you, has a policy of not tolerating it under any circumstances, and it seemed to me that that required understanding what it was. So now I'm curious--does the above definition match the thing you hate?
To illustrate another example where the "avoid buyer's remorse" principle is overbroad (which may or may not be the principle you are advocating), let's talk about cookies.
You're having a dinner party, and everyone is stuffed. You bring out some freshly baked cookies for dessert. Guest: "Oh no, I'm stuffed." You: "They are warm and soft!" Guest: "Well, that does sound good, I'll have just one." Guest eats a cookie. Later, Guest looks a bit queasy and is obviously regretting eating so much.
This is another case that falls afoul of the "avoid creating buyer's remorse" principle, but doesn't deserve such a negative term as "manipulation."
The fact is that a lot of people enjoy taking actions that they may later regret. It's not a moral requirement to protect people from themselves. As long as they are making the decision with free will (or the closest thing to it that humans have), and informed consent exists, then it's valid for people to take responsibility for the risks of their behavior. We should assume that people can assess their best interests, unless we have reasons to believe otherwise (for instance, if you know that ...
Influencing someone to take an action that you know she will regret afterwards is manipulative.
What if you also "manipulate" so that it will also not be regretted afterwards?
In marketing at least, there is the concept of customer retention through anticipating and countering buyers' remorse... which mostly, AFAICT, consists of providing a customer with arguments to use to explain to co-workers, friends, relatives, spouses, or whomever why their purchase decision was a good one. This strongly implies that at least in the purchasing arena, the main reason people come to have buyers' remorse (besides crappy products) is that the purchase makes them look bad in the eyes of others.
Hence the marketing adage that the primary function of facts and logic in a sales pitch is to provide the customer with a rationale that lets them prove to themselves and others that they made the right decision... but only after they've been convinced to make that decision based on emotion.
I believe there's some discussion in the PUA field of similar "buyer's remorse" issues and providing the same sort of supporting rationale, except that such rationales are more to allow those women...
Yay, ethics of influence and seduction, one of my favorite subjects. I'm still figuring out my thoughts, so I would appreciate it if people tell me if I'm making incorrect assumptions.
First, I'll state that I prefer the terms "unethical influence" and "unethical influence" to "manipulation" and "non-manipulation," because people use "manipulation" to mean too many different things.
Second, in ethical discussions, we should distinguish between things that would be a good things to do that aren't morally required, and things that are morally required (Kant called these "imperfect duties" vs "perfect duties"). Also, we should distinguish between ethical courses of action, and empathetic courses of action. There are lots of cases where there's an action that's a good and empathetic thing to do, but actually requiring it as a perfect duty would screw everything up.
Physical coercion forces you to do something that you don't want to do, don't enjoy while you're doing it, and regret doing afterwards.
There is clearly a perfect duty against physical coercion, and that duty applies even if someone happens to enjoy what y...
In the case of sex, I propose a different demarcation criterion between ethical and unethical social influence than yours. The dimension I'm most concerned with is not remorse or lack thereof after the fact, but rather the reasons for consent at the moment of consent. In the past, I proposed the following definition for ethical seduction on a pickup blog:
Taking intentional action to create a context that raises someone’s chances of wanting to be sexual with you of their own free will out of anticipation of intrinsic enjoyment of the experience.
The "anticipation of intrinsic enjoyment of the experience" criterion is important, because it gets rid of cases where people consent to sex out of feelings of obligation, pity, merely because the other person wants it, or because they had trouble putting on the brakes. This notion is similar to the notion of "enthusiastic consent," but without the confusing connotations that "enthusiastic" may hold.
I'm not sure whether this is a perfect or merely an imperfect duty. The argument for this principle being a perfect duty is that if someone has sex with you for reasons other than anticipating inherent enjoyment o...
No matter how hard I try and kick it, there it is: Honesty.
Where in particular do you perceive the lie in the situation you described? Would her donation not have sponsored the child? Would the donation not have made her feel better? Or is that you do not believe she should feel better for sponsoring a child; that in your mind it would be dishonest for her to displace her current grief with this balm?
Of course if the donation would not have sponsored the child, then it would be dishonest to claim that it would.
I can also imagine the donation not actually making her feel better. It would be possible to simply overwhelm her (e.g., intimidate them with a whirlwind of emotional stimuli) into doing something she didn't want to do. This is in the direction of being mean/bullyish. People can push boundaries and this is being pushy...but we don't often describe it explicitly, perhaps because it requires such a high emotional intelligence to identify and name it.
Finally, the third case is that you don't believe sponsoring children is a feel-good thing. Which would be a strong indication that that wasn't the right job for you, but wouldn't mean that sponsoring a child wasn't the right thing for her.
The bit about Sheila from accounting prompts a fun question for those LWers who subscribe to specific theories about gender relationships (PUA, feminism or whatever): within a couple, do you predict the events "husband cheats" and "wife cheats" to be independent, correlated or anticorrelated? To me it sounds like it could just as easily go one way or the other, and each answer has many rationalizations that immediately spring to mind. (Won't spell them out to avoid spoiling your fun.)
Correlated, because time spent apart causes infidelity for both. Also, cultural norms make a difference, and both are exposed to the same culture.
Anti-correlated, because differences in libido causes a higher probability of infidelity on the part of whoever wants more sex, and a lower probability on the part of whoever wants less.
I'm not sure which of these I find more convincing.
Anticorrelated: Sheila's attraction to Hubby implies Hubby has qualities which make him appealing to women, which makes Wifey less likely to cheat.
Correlated because cheating by one partner implies less mate-guarding by the other. A partner who doesn't bother to mate guard is less concerned about losing that partner, and thus more likely to cheat.
Also both partners are more likely to cheat if the passion has died down in the relationship, or if they're in the midst of a conflict.
Interesting that given the choice between leaving the woman to cry and an obvious route to make her feel better, you felt that the correct, moral thing to do was to leave her miserable.
There would certainly seem to have been a third option: comfort her without selling her anything...
FWIW, I second Relsqui's curiosity about why you think this sequence is a good idea.
I really can't help but correct your math: 12! would be if you could choose how to order the 12 toppings, but that's not what you're doing - you're choosing a subset of them - that's 2^12 options. Don't worry, you were only off by five orders of magnitude.
Sorry about that.
Would we expect those who consciously learned the Dark Arts to be more or less responsible with them (that is, using them ethically) than those who are naturally skilled?
Out of interest, how many of the people who listened to your pitch subsequently gave donations?
In Ian M Banks' novel 'Excession', the term meatfucker is used as an derogatory term for AIs that violate a cultural taboo again reading human minds.
This is the only comment I've ever seen here where I've vacillating between voting up or down, rather than just leaving it alone. I can recognize that it's either really good or really bad, but I can't decide which one, so congratulations, I guess.
Dark Arts done right... people need to get used to being manipulated a little to at least get the feel of it happening, or to learn a bit of it if that doesn't work. Experience as a net troll does wonders to keep me from getting riled up. But honesty does not have the least collateral damage trolling does.
1) Any group of mostly men dedicated to any sort of hardware--guns, cars, for all I know chainsaws. For some reason men don't just form attachments to the specific tool they actually have (Cars, Guns, Hammers, scapels are all tools), they have to defend the BRAND and/or model (Mopar, 1911, Macintosh are the most vociferous defenders).
2) Any group that thinks unreasonably highly of itself. Atheists, Mensa, Vegetarians/Vegans, Hippies, Mac Users.
3) Any group that feels under threat, either from the march of technology, a conspiracy, or just reality damaging their world view like Socialists, Vegetarians/Vegans, 1911 owners and Mac Users.
I salute your ability to troll all of these groups in a post about what kind of groups are easy to troll. I almost started to argue on some of these points before I saw your game.
When you manipulate a person's distressed emotional state to convince them into an action, even when that action is a positive one (sponsoring a child) there is always the potential that the person who was manipulated will later realize the manipulation.
Essentially - If he had convinced the crying woman to sponsor the child she very well have felt better on an ongoing and permanent basis (a positive result for everyone involved), however there is also the potential that at some later date she would have realized the manipulation.
My understanding of being ...
I'm pretty sure I don't like being Sold. I generally have a pretty good idea of what I do or don't want, and (to the extent that I recognize it), I dislike being manipulated.
You might be right about people in general, but it seems like a hard thing to check on whether people like being Sold as distinct from whether it's possible to Sell them.
So, is the story real, and why did you include the spider (I reckon that is not real, too perfect)?
I notice that I am confused. Manipulation may have occurred if someone's free will has been overridden. There was a nice post about the compatibility between free will and determinism, which I agree with, but I find the issue of manipulation shows I still have a fuzzy idea about what free will means.
The work on Friendly AI gives me a fairly good idea about how it's possible to avoid being manipulative to a person, but the answer - following their extrapolated volition - is much too high a standard. What if you want to be neutral rather than selflessly and altrusticly benevolent towards them?
Still, their extrapolated volition seems to be relevant.
Fascinating post!
While the Dark Arts are a Power, it’s how you use them that matters, like any other tool. I’m running mind-games on people, but I usually won’t; I’m also good at fighting, but I don’t assault people for no reason. I find both concepts repulsive.
I agree with your reasoning here. Along these lines, I'm curious about something:
Do you think learning this stuff made you better or worse as a rationalist?
What I was trying to get at was that it isn't nice to tempt people to do things that are bad for them.
I think we're saying something similar here. It's a failure of empathy to tempt people to do things that might be bad for them, but I'm not convinced it's always a failure of ethics in the sense of perfect duty. There are plenty of forms of influence where I would be comfortable saying that the influencer is being a jerk, but where I wouldn't be comfortable saying that they are doing something unethical.
I don't think it's nice to influence people by selling a product you know is defective,
This is unethical under my analysis, also. It falls into the category of where you know something big that the other person doesn't which destroys informed consent.
or trying to get them to buy something you know they can't afford.
If you know this for sure, that would also be unethical under my analysis. My analysis only applies to situations where people are capable of watching out for their own interests. This is the default situation, but if someone gives you strong evidence that they can't watch out for themselves (e.g. they are trying to buy things you know they can't afford), then you should scale down the influence. Unfortunately, most real-life situations of influence are less clear-cut.
I don't think it's nice to talk your friend into going out to a party the night before his big test.
This one seems a bit less clear-cut. If you get your friend to miss out on sufficient sleep, or get hung over, then yeah, that's unethical. But if the problem is that he might skimp on his studying? That's less clear. Your friend knows better than you do whether they've done enough studying for their test. If you invite your friend, you know that he can refuse if he needs to study. On the other hand, maybe he's done enough studying, and would benefit from relaxing. Unless you have some evidence either way, you're not doing anything wrong by inviting him and letting him decide.
Yes, in some of these cases, the "victim" should have known better, but strong influence can bypass willpower.
Could you give an example of such a scenario? I'm not seeing it in either of the three in your post. Can you give an example of influence that is morally suspect, and (a) involves adequate informed consent, unlike the defective product scenario, (b) involves someone capable of watching out for themselves, unlike the scenario of buyers trying to buy things you know they can't afford, and (c) is actually strong, unlike talking your friend into going to a party?
I tried to come up with an example, and the first thing I thought of was running an unhealthy fast food business and influencing people to eat your food. Then I realized that this example fails the test I proposed, because it would be known that many people don't watch out for their health. While eating at your restaurant every once in a while might not be a health risk, your advertising will pull people in on a regular basis (in contrast, if you are running an ice cream shop, you aren't trying to get people to eat there every day, and they won't be eating ice cream instead of healthier food). There is something inherently wrong with supplying tasting food that is unhealthy, but if you do it in a way that influences people to substitute healthier food for what you are providing, and eat your food regularly enough that it's a health risk, that's unethical.
The better you are at tempting people, the more responsible you are for the results of the temptation.
The best way to tempt people is to offer them what they want (in a way that they identify it as satisfying one of their wants). I think your claim implies that people who can offer things that other people want more should have higher responsibility for the results of influencing people. That principle may be a little to broad, and lead the people who actually have good things to offer (which is influential) to have a moral panic and become too shy.
In terms of sales and products, there are four simplified categories:
The principle you suggest would make good salesmen with bad products more inhibited... but it will also make good salesmen with good products more inhibited! Actually, many of the most influential products may be good products with good salesmen... should these salesmen with the good products really be fretting the most about responsibility just because they are good at sales?
I think we need to articulate a principle that makes good salesmen with bad products scrutinize themselves more than good salesmen with good products.
The degree that people want what you offer isn't quite the right index for the degree of responsibility you should shoulder for the result. People who want something really badly might be more likely to throw caution to the winds and make mistakes, so degree of want is weakly related to the responsibility you should take to protect them. But there are other variables that are much better indexes for the level of responsibility you should take:
How do these criteria sound to you?
On the contrary, I don't think I'd mind being influenced to do something that turned out to be good for me
Unfortunately, people who influence you can't predict the future and know whether the results will turn out good or bad for you. Of course, they should make guesses, and refrain from influencing you if they think the expected value of the result will be negative for you, and you haven't given informed consent to that risk. But when there is uncertainty about the result, the ability of people to make those sort of guesses varies depending on the domain.
In financial services, your broker really does know better than you about whether their product will be good for you, and the criteria for satisfying your preferences is objective and easily understood by them. As a result, a lot of responsibility falls on their shoulders.
In social interaction and dating, it's a lot harder to know whether you are a good match with someone better than they do. You don't know exactly what their preferences are and how you fit into them. Responsibility for protecting them from buyer's remorse falls primarily (but of course, not completely) on them. Since they have the higher quality information about how you fit into their preferences, it's their responsibility to communicate their preferences to you to help protect them from being unhappy later.
I’d like to tell you all a story.
Once upon a time I was working for a charity – a major charity – going door-to-door to raise money while pretending it wasn’t sales.
This story happened on my last day working there. I didn’t know that at the time; I wouldn’t find out until the following morning when my boss called me up to fire me, but I knew it was coming. For weeks I’d been fed up with the job, milking it for the last few dollars I could pull out, hating every minute of it but needing the money. The Sudden Career Readjustment would come as a relief.
So on that day, my last day, I was moving slowly. I knocked on one particular door and there was no response. I had little desire to walk to the next one, however, and there was an interesting spider who’d built its web below the doorbell. I tapped its belly with the tip of my pen, and it reacted with aggression – trying to envenom and ensnare the tip of my ballpoint. I must have been playing with it for a good minute or so when the door suddenly opened.
A distraught woman stood before me. After a brief period of Relating I launched into my pitch.“So you’re probably wondering why there’s a bald weirdo at your door? Actually I’m just coming around with Major Charity1 on an emergency campaign. You’ve heard of us, right? Brilliant! So obviously you’ve thought of getting involved, right? That’s awesome! You see, the reason I’m coming around is for these guys – some of our emergency cases...”2
I handed her the pictures of the Developing World Children (yeah, it was one of those charities). She took them, a wistful look on her face.
“Oh God, don’t show me these. I’m such a Rescuer.”
“Rescuer? Do you have a Rescue Dog?” [Where I’m from, abused animals brought into a new home are called ‘Rescue Dogs’.]
“No, I...”
“You mean your personality? You care about people, don’t you?”
She nodded slowly. Her face began to crumble.
“I’m sorry – I can’t look at these children,” she handed back the photographs, “Not right now. I’ve been crying all day and I just can’t deal with those emotions...”
I took back the children, a look of honest sympathy on my face. The Demon Wheel began spinning. I could see that she was on the verge of crying again. My gut told me that her father had recently died, but the actual cause didn’t matter. I could discover that information. The upcoming dialogue played itself out in my mind...
"Oh jeez, what happened? Oh my god, seriously..?” Head tilted as an Alpha confidant enough for Beta behaviour, looking down and shaking, “I’m lucky enough to have never been through that. Were the two of you close?” As she talks I nod, prompting her until she breaks out in tears. I put down my binder and step into her house, embracing her as she cries on my shoulder.
She sniffles.
“I’m sorry... sorry to do this to you.”
“No, don’t be. Listen... Mary, is it? What you’re going through is normal. It’s nothing to be ashamed of…” Cue personal anecdote, then pause for a beat. “Listen, about the Major Charity thing; this is something you’ve always wanted to do, isn’t it? Yeah, I can tell. You’re a caring person, after all. I tell you what: we’ll get you set up with this little boy – he’s from Ecuador, and we’re trying to get him eating a healthy diet. We’re going to make you his super hero today. And then you’ll know – Mary, you’ll know that even at your darkest moment, you still have the strength in you to save a life.
“And you know what else?” I reach out to touch her arm, “Tonight you’re going to sleep like a baby knowing that you did this. So you go and get your Credit Card and I’ll start filling out the form.”
* * *
I could have done it. I could have got that child sponsored. I could have kept my job, and Mary could have stopped crying that evening. She’d have thanked me for coming by, and after I left she would have cuddled on the couch with her new Sponsor Child, tears drying as she found hope in the world.
But I didn’t do it. Instead I apologized for interrupting her grief, and left.
Because I am not a Meat Fucker.
* * *
All my life, I’ve had this bad habit. No matter how hard I try and kick it, there it is: Honesty. I can’t tell you how many times it’s dug me into a hole. As far as concepts go, it’s about as foolish and utopian as Truth and Justice, and I know that, but I just can’t seem to let it go. That’s a large part of the reason I left Mary alone to her tears – backed off, rather than digging into her psyche to recalibrate a few clusters of neuron.
The other half is my status as a card-carrying (union-dues-paid-in-full) Anarchist. The way I look at things, the only time you can justify using the Jedi Mind Trick on somebody is when your ethics would stand clean with murdering them as well.
Sending Storm Troopers on a Wild Droid Chase is one thing; scamming Waddo out of a distributor cap for your CGI Space Plane is another.
When you take advantage of the Dark Arts, you’re not simply tricking people into giving you what you want; you’re making them want to give it to you. You’re hacking into their brain and inserting a Murder Pill; afterwards they will literally thank you for doing so (the only sponsor I ever met who wasn’t glad that I’d come by was the lady whose 6 year old daughter I primed into wanting it). In ninety percent of the situations where the Dark Arts are useful or possible, you can’t do it out of spite; when you realign someone’s desires to match your own they want to do what you want them to do.
And yet there’s no clear distinction between using these skills and regular social interaction. Manipulation works best when you’re sincere about it. Ethically speaking it’s a grey, wavy line.
The thing is, we all like to be Sold, Led, Dominated; if I walk into Subway, and I ask the kid at the counter to give me his Best Submarine Sandwich, I want him to tell me what I want, and make me love it after it’s paid for. The last thing he should do is say that “They’re all good!” and make me regret the [(5 breads)x(16 meats)x(212 Toppings)-1] subs that I didn’t get.3 Retail is the Dark Arts Done Right (usually). The Sales Lady figures out what I want, uses her expertise to find the best fit, and then kills the cognitive dissonance that could ruin my enjoyment of the product; “You really pull off that colour. Seriously, that jacket looks great on you – you see how these lines naturally compliment your shoulders? Of course you can!”
Sexual dynamics are similar; if somebody’s drinking in public at 2 in the morning it’s because they’re on the market. Let’s say a ‘faithful wife’ goes to the club one weekend while her husband is out of town, and she has a few drinks with a bunch of college boys she just met. One of them happens to be a PUA. When it comes to things like date rape drugs, or taking advantage of a person who’s sloppy-drunk there is a clear line in the sand. But in this hypothetical the woman’s relatively sober. It’s just that the young rake is so damned charming!
Meanwhile her husband’s having a few pints at the hotel bar with Sheila from accounting, and she just keeps making eyes at him…
Neither Sheila nor the PUA is responsible for the ensuing infidelity. If the husband and wife didn’t want it in the first place, they would have never availed themselves to the temptation. If, on the other hand, you meet somebody at a Neighbourhood Watch meeting, and spend the next three months seducing them… that’s when you’ve got to start questioning your ethics. Anybody is going to be vulnerable at some time or another.
While the Dark Arts are a Power, it’s how you use them that matters, like any other tool. I’m running mind-games on people, but I usually won’t; I’m also good at fighting, but I don’t assault people for no reason. I find both concepts repulsive.
That’s the end of my moralizing on the matter. The upcoming series is going to be purely descriptive in nature, exploring different strategies for manipulating others. I’ll provide tactical examples showing how these strategies can be put into play, but for the most part each battlefield is unique; these are broader methods that apply across the board. What you do with these techniques is up to you.
As for defence… I don’t think I’ll have much to say about that. When done properly, the victim doesn’t realize it until it’s already over, and by then it doesn’t matter. You’re aware that the AI manipulated you into opening the box, but you’re going to open it anyways because that’s your new utility function. It’s like a game of Roshambo, or when you’re thinking about joining Facebook: the only way to win is not to play.
Endnotes
1. Major Charity’s methods of acquiring funding don’t have any bearing on whether or not it’s an effective charity. Whether or not the money going overseas actually makes a difference is a question I cannot answer.
2. The repetition here is intentional. I was trying to prime key concepts.
3. My theory as to what is going on with these sub places and their myriad of options: the target is not a new customers, those people are going to be intimidated by all the choices, and the restaurants know that. Rather, it is to provide ‘fresh’ options so that their current customers don’t get bored and go elsewhere.