Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

The Power of Reinforcement

92 Post author: lukeprog 21 June 2012 01:42PM

Part of the sequence: The Science of Winning at Life

Also see: Basics of Animal Reinforcement, Basics of Human Reinforcement, Physical and Mental Behavior, Wanting vs. Liking Revisited, Approving reinforces low-effort behaviors, Applying Behavioral Psychology on Myself.

 

Story 1:

On Skype with Eliezer, I said: "Eliezer, you've been unusually pleasant these past three weeks. I'm really happy to see that, and moreover, it increases my probability than an Eliezer-led FAI research team will work. What caused this change, do you think?"

Eliezer replied: "Well, three weeks ago I was working with Anna and Alicorn, and every time I said something nice they fed me an M&M."

 

Story 2:

I once witnessed a worker who hated keeping a work log because it was only used "against" him. His supervisor would call to say "Why did you spend so much time on that?" or "Why isn't this done yet?" but never "I saw you handled X, great job!" Not surprisingly, he often "forgot" to fill out his worklog.

Ever since I got everyone at the Singularity Institute to keep work logs, I've tried to avoid connections between "concerned" feedback and staff work logs, and instead take time to comment positively on things I see in those work logs.

 

Story 3:

Chatting with Eliezer, I said, "Eliezer, I get the sense that I've inadvertently caused you to be slightly averse to talking to me. Maybe because we disagree on so many things, or something?"

Eliezer's reply was: "No, it's much simpler. Our conversations usually run longer than our previously set deadline, so whenever I finish talking with you I feel drained and slightly cranky."

Now I finish our conversations on time.

 

Story 4:

A major Singularity Institute donor recently said to me: "By the way, I decided that every time I donate to the Singularity Institute, I'll set aside an additional 5% for myself to do fun things with, as a motivation to donate."


The power of reinforcement

It's amazing to me how consistently we fail to take advantage of the power of reinforcement.

Maybe it's because behaviorist techniques like reinforcement feel like they don't respect human agency enough. But if you aren't treating humans more like animals than most people are, then you're modeling humans poorly.

You are not an agenty homunculus "corrupted" by heuristics and biases. You just are heuristics and biases. And you respond to reinforcement, because most of your motivation systems still work like the motivation systems of other animals.

 

A quick reminder of what you learned in high school

  • A reinforcer is anything that, when it occurs in conjunction with an act, increases the probability that the act will occur again.
  • A positive reinforcer is something the subject wants, such as food, petting, or praise. Positive reinforcement occurs when a target behavior is followed by something the subject wants, and this increases the probability that the behavior will occur again.
  • A negative reinforcer is something the subject wants to avoid, such as a blow, a frown, or an unpleasant sound. Negative reinforcement occurs when a target behavior is followed by some relief from something the subject doesn't want, and this increases the probability that the behavior will happen again.

 

What works

  1. Small reinforcers are fine, as long as there is a strong correlation between the behavior and the reinforcer (Schneider 1973; Todorov et al. 1984). All else equal, a large reinforcer is more effective than a small one (Christopher 1988; Ludvig et al. 2007; Wolfe 1936), but the more you increase the reinforcer magnitude, the less benefit you get from the increase (Frisch & Dickinson 1990).
  2. The reinforcer should immediately follow the target behavior (Escobar & Bruner 2007; Schlinger & Blakely 1994; Schneider 1990). Pryor (2007) notes that when the reward is food, small bits (like M&Ms) are best because they can be consumed instantly instead of being consumed over an extended period of time.
  3. Any feature of a behavior can be strengthened (e.g., its intensity, frequency, rate, duration, persistence, its shape or form), so long as a reinforcer can be made contingent on that particular feature (Neuringer 2002).

 

Example applications

  • If you want someone to call you, then when they do call, don't nag them about how they never call you. Instead, be engaging and positive.
  • When trying to maintain order in a class, ignore unruly behavior and praise good behavior (Madsen et al. 1968; McNamara 1987).
  • Reward originality to encourage creativity (Pryor et al. 1969; Chambers et al. 1977Eisenberger & Armeli 1997; Eisenberger & Rhoades 2001).
  • If you want students to understand the material, don't get excited when they guess the teacher's password but instead when they demonstrate a technical understanding.
  • To help someone improve at dance or sport, ignore poor performance but reward good performance immediately, for example by shouting "Good!" (Buzas & Allyon 1981) The reason you should ignore poor performance if you say "No, you're doing it wrong!" you are inadvertently punishing the effort. A better response to a mistake would be to reinforce the effort: "Good effort! You're almost there! Try once more." 
  • Reward honesty to help people be more honest with you (Lanza et al 1982).
  • Reward opinion-expressing to get people to express their opinions more often (Verplanck 1955).
  • You may even be able to reinforce-away annoying involuntary behaviors, such as twitches (Laurenti-Lions et al. 1985) or vomiting (Wolf et al. 1965).
  • Want a young infant to learn to speak more quickly? Reinforce their attempts at vocalization (Ramely & Finkelstein 1978).
  • More training should occur via video games like DragonBox, because computer programs can easily provide instant reinforcement many times a minute for very specific behaviors (Fletcher-Flinn & Gravatt 1995).

For additional examples and studies, see The Power of Reinforcement (2004), Don't Shoot the Dog (2006), and Learning and Behavior (2008).

 

I close with Story 5, from Amy Sutherland:

For a book I was writing about a school for exotic animal trainers, I started commuting from Maine to California, where I spent my days watching students do the seemingly impossible: teaching hyenas to pirouette on command, cougars to offer their paws for a nail clipping, and baboons to skateboard.

I listened, rapt, as professional trainers explained how they taught dolphins to flip and elephants to paint. Eventually it hit me that the same techniques might work on that stubborn but lovable species, the American husband.

The central lesson I learned from exotic animal trainers is that I should reward behavior I like and ignore behavior I don't. After all, you don't get a sea lion to balance a ball on the end of its nose by nagging. The same goes for the American husband.

Back in Maine, I began thanking Scott if he threw one dirty shirt into the hamper. If he threw in two, I'd kiss him. Meanwhile, I would step over any soiled clothes on the floor without one sharp word, though I did sometimes kick them under the bed. But as he basked in my appreciation, the piles became smaller.

I was using what trainers call "approximations," rewarding the small steps toward learning a whole new behavior...

Once I started thinking this way, I couldn't stop. At the school in California, I'd be scribbling notes on how to walk an emu or have a wolf accept you as a pack member, but I'd be thinking, "I can't wait to try this on Scott."

...After two years of exotic animal training, my marriage is far smoother, my husband much easier to love.

 

Next post: Rational Romantic Relationships Part 1

Previous post: The Good News of Situationist Psychology

 

 

My thanks to Erica Edelman for doing much of the research for this post.

Comments (466)

Comment author: AdeleneDawner 21 June 2012 05:03:47AM 35 points [-]

Bit of a tangent, but if you ever run across someone for whom this doesn't seem to work, check the hypothesis that they don't parse praise as a positive reinforcer. I don't know how common this is, but I actually have to make a conscious effort to keep it from acting as a mild punishment in most cases when it's applied to me. (Ditto M&Ms in the given context, I expect. Attention Bad.)

Comment author: [deleted] 21 June 2012 06:37:00AM *  10 points [-]

You are correct that there are many kinds of reinforcers, and it's important to make sure that the one you choose to use is something the receiver will desire.

"In other studies, animals and people given a choice between performing a task for either of two reinforcers often show strong preferences (Parsons & Reid, 1990; Simmons, 1924). Identifying preferred reinforcers can improve the effectiveness of a reinforcement procedure in applied settings (Mace et al., 1997).”

-Learning and Behavior, p149

Comment author: Viliam_Bur 21 June 2012 09:56:42AM 3 points [-]

Yes, the situation is usually not so easy that behavior is just a result of inputs, like this:

output := f(input)

People have minds, and a mind is an environment, different for different people. The real equation would be more like this:

[mind1, output] := f([mind0, input])

For example many people like attention of others, but some people may be trained (for example by a previous abuse) that attention of others is usually followed by pain. For them, a positive reinforcement by giving them attention wouldn't work, because the important things is not the attention per se, but what it means for them.

On a meta level, for someone even the idea of "learning" or "improving" or "changing" may be already associated with pain, so they will resist any such process if they notice it. A human mind can be messed up rather easily.

Comment author: DaFranker 07 July 2012 04:16:39AM *  1 point [-]

[...] On a meta level, for someone even the idea of "learning" or "improving" or "changing" may be already associated with pain, so they will resist any such process if they notice it. A human mind can be messed up rather easily.

This becomes painfully common (and obvious to any observant third-party that knows these concepts) for subjects that students "are just not made for", such as large amounts of students that "just don't get" maths. They've been trained in so many ways to associate actual learning (especially the actions taken when attempting to learn a concept) with negativity that it becomes obviously so much more rewarding to just guess the teacher's password, and so they are positively reinforced into doing everything they can to avoid mental modeling and seek password-guessing through aggregation and correlation of symbol-data. In most cases I've observed, they become experts at the skill of subconsciously forming "truth-tables" of teachers' passwords through brute-force trial-and-error tactics. What's more, this tactic, which they've been trained to do and learned so well and associate so much with positive feedback, often feeds itself into a vicious circle through several possible methods, which makes getting out (or, for that matter, even realizing that it's there and you need to get out of it) so much more difficult than if that behavior had been blocked immediately when it first appeared.

When I realized that, I've started to feel sad for every student I see showing signs of spending hours upon hours studying and memorizing and headsmashing against the same math problems "until they finally understand them", when in truth they haven't really gained anything worthwhile (IMO) from the experience.

Comment author: Will_Newsome 21 June 2012 10:17:15PM 8 points [-]

Furthermore at least one person I know (er, myself) picks up on any sort of test-like or game-like or we're-judging-you-so-you-better-not-screw-up-like context and starts acting in extremely confusing/uninformative/atypical/misleading ways so as not to be seen as the kind of person who is easily manipulable (there are probably other motivations involved too). Any incentive structure I'm put under thus has to somehow take this into account, even e.g. the LessWrong karma system. Explicitly manipulative socially mediated praise/M&Ms would strike my brain as outright evil and would stand some chance of being inverted entirely. That said I don't get the impression this sort of defense mechanism is very common.

Comment author: Oligopsony 27 June 2012 08:37:23PM 25 points [-]

Excellent insight. Downvoted.

Comment author: Lethalmud 26 June 2013 02:45:51PM *  4 points [-]

So you are saying that, to change your mode of behavior, all one has to do is create a judging context? That would actually make you very easy to manipulate..

Comment author: TheOtherDave 21 June 2012 10:21:21PM 2 points [-]

Explicitly manipulative socially mediated praise/M&Ms would strike my brain as outright evil and would stand some chance of being inverted entirely. That said I don't get the impression this sort of defense mechanism is very common.

I experience this as common, but I suspect it's because of a small number of exceptionally vocal "manipulation is evil!" types in my life, rather than a larger number of typically vocal ones.

Comment author: [deleted] 21 June 2012 05:20:21AM 2 points [-]

I'd have to say that it shouldn't be that common. Most people want to be praised.

Comment author: erratio 21 June 2012 01:21:35PM 19 points [-]

Most people want to be sincerely praised. Someone who reads this post and applies it poorly is going to be saying praise while their body language says something else entirely. Or acting out of character for themselves, leading the reinforcee to suspect that the praise is insincere. Or they may go around praising seemingly everything, causing the reinforcee to interpret the praise as meaningless noise.

There are lots of ways for using praise as reinforcement to go wrong, and if someone is in one of those environments for long enough they will end up being conditioned to interpret praise as neutral or negative.

Comment author: JGWeissman 21 June 2012 05:35:09AM 3 points [-]

I suspect it is common enough that when you observe that praising someone doesn't reinforce their behavior or makes them uncomfortable, you should consider that they might have an unusual aversion to praise.

Comment author: pjeby 21 June 2012 04:41:34PM 9 points [-]

I suspect it is common enough that when you observe that praising someone doesn't reinforce their behavior or makes them uncomfortable, you should consider that they might have an unusual aversion to praise.

And also, that you might just be really bad at it. ;-)

This was my problem for quite a while: believing that I ought to praise people, while alieving that there wasn't anything to praise and that they didn't deserve it, due to all their obvious imperfections.

This, as you can imagine, produced sub-optimal results. ;-)

Comment author: mstevens 21 June 2012 03:47:01PM 19 points [-]

To help someone improve at dance or sport, ignore poor performance but reward good performance immediately, for example by shouting "Good!" (Buzas & Allyon 1981) The reason you should ignore poor performance if you say "No, you're > doing it wrong!" you are inadvertently punishing the effort. A better response to a mistake would be to reinforce the effort: "Good effort! You're almost there! Try once more."

I've noticed in pilates classes with one specific teacher you get positive feedback in one specific situation - when you're having trouble, and have just barely managed something basic. This leads to the association that whenever you get positive comments you know you're doing badly.

Comment author: [deleted] 21 June 2012 06:20:33PM 6 points [-]

Yeah, there's kind of a perceptual/patternmatching arms race going on there -- if you're too blatant about it, or the intended recipient of the reinforcement is just that perceptive, then they're reading the script too and it won't have the intended result. It could backfire (as in your example; semantically-positive reinforcement becomes pragmatically-negative), or send undesirable information ("you wouldn't have put it that way unless something were up, and that gives me a clue"), or open you to counter social-engineering scripts if the part knows what they're doing.

Comment author: mstevens 21 June 2012 08:18:31PM 2 points [-]

In my case I'm not terribly perceptive, but there's a lot of repetition of the same situation to give you a clue.

Comment author: pnrjulius 05 July 2012 01:27:58AM 1 point [-]

If that's the case (and it seems like it is), then reinforcing yourself is going to be almost impossible, because you will by definition know the reinforcement script.

Comment author: [deleted] 26 June 2012 07:10:24PM *  12 points [-]

The lead article conflates two process: habits and incentives. The very term "reinforcement" dates back to before the distinction was well-understood. Only in the last decade has it been known that habit operates from a neurology distinct from incentives. (The habit mechanism is in a much older part of the brain.) Only the first story, Yudkowsky and the jellybeans, deals clearly with reinforcement of habit. The others are probably primarily adjustment of incentives.

In using habit and incentive, different rules apply. Incentives require that the subject discern the contingency. The processes Skinner studied as "reinforcement" are mostly about incentives. You adjust schedules of reinforcement to alter the organism's expectancies. For incentive effects, consistent reinforcement is not usually best, as the results are subject to extinction soon after the organism stops getting the reward.

Habits, on the other hand, are blind. The organism doesn't need to see any contingency. Yudkowsky continued to be nice even after he no longer received the jellybeans. To form habits, as opposed to incentive structures, consistency is key.

In short, as a general rule, you want consistency to reward habits and considerable randomness to create lasting incentives.

But the difference extends also to the ethical questions raised. Altering others' incentives for our own benefit is part of ordinary human interaction. If his colleagues surreptitiously timed the offer of jellybeans to Yudkowsky when he acted nice, this is something else; the ethical reason is that Yudkowsky need not recognize what he's being rewarded for to be affected by the jellybeans.

Both habit and incentive are "powerful." But they're powerful for different reasons, in different ways; and to apply them effectively and ethically requires different procedures.

Comment author: mwengler 03 July 2012 09:52:14PM 4 points [-]

How do you tell which things you want to reinforce are habits (and should therefore be reinforced consistently) and which things are incentives?

Comment author: bbleeker 19 February 2013 11:22:12AM 1 point [-]

I'd think a habit is something that just goes on as long as nothing happens to disrupt it. You no longer need to reinforce it.

Comment author: Pablo_Stafforini 19 October 2012 12:21:03AM 1 point [-]

Can anyone here point me to the relevant scholarly literature discussing the differences between habits and incentives? I tried Google and Google Scholar but failed to find any paper or survey article that explicitly contrasts these two processes.

Comment author: shminux 21 June 2012 03:21:11PM *  10 points [-]

But if you aren't treating humans more like animals than most people are, then you're modeling humans poorly.

Thanks for pointing out this particular low-hanging fruit.

Eliezer replied: "Well, three weeks ago I was working with Anna and Alicorn, and every time I said something nice they fed me an M&M."

I wonder if they had just (re-)watched this Big Bang Theory episode.

you don't get a sea lion to balance a ball on the end of its nose by nagging

Hmm, I better keep this in mind at all times when dealing with my family.

Comment author: lukeprog 21 June 2012 03:35:23AM 22 points [-]

Reason #228 I'm crazy and irrational: Without conscious attention to the reinforcement process, my behaviors are selected for reinforcement almost at random. The process selecting behaviors for reinforcement has tons of steps in it like "Did I happen to glance in the direction of the bag of M&Ms right now?" instead of "Is the thing I'm doing now something I want to reinforce?"

Comment author: TheOtherDave 21 June 2012 03:39:47AM 8 points [-]

(nods) For my own part, it's frequently worse than random... when I don't attend to what I'm doing, I frequently berate or otherwise punish myself for attempts to achieve a target that fall short of that target, and I'm more likely to do that the more I value achieving the target. Which is a great way to extinguish the behaviors I value.

Comment author: Viliam_Bur 21 June 2012 09:45:25AM *  4 points [-]

I suspect it's very difficult to design the right reinforcement strategy. It's easy to reward something that seems related to the goal, but can gradually become a replacement for the goal.

For example rewarding success and punishing failure reinforces choosing only trivial tasks, which prevents learning new things. Rewarding starting new things reinforces starting new tasks without finishing them, also choosing tasks for being new, not being useful. Etc.

Rational thinking about consequences, and changing the strategy when necessary, cannot be avoided. So perhaps this should be reinforced. But how do we distinguish between genuine rationality and signalling? Yeah, rationalists should win, but by rewarding success and punishing failure... see the previous paragraph.

Anyway, many people do worse than random, so some reinforcement can be used to improve the situation.

EDIT: Another problem: I suspect that any reinforcement inevitably goes meta. When I get a reward for doing X, I will do X more, but I will also like the reinforcement mechanism more. When I get punished for doing Y, I will do Y less, but I will also hate the reinforcement mechanism and rationalize why I must get rid of it.

I suspect that people prefer wireheading, except in cases when it becomes too obvious that it is wireheading. If I am allowed to choose my reinforcement mechanisms, I will probably unknowingly slowly optimize them towards wireheading. If someone else chooses my reinforcement mechanisms, I suspect they will choose it to optimize their utility function instead of mine.

Comment author: mwengler 03 July 2012 09:54:07PM 2 points [-]

Yoiks! This may well be why my procrastination at work has increased and increased over the decades. I almost always (habitually?) feel like my efforts are not good enough, will be criticized negatively.

Comment author: TheOtherDave 03 July 2012 10:02:44PM 1 point [-]

(nods) That's a pretty common result of relying on punishment to shape behavior.

Comment author: johnlawrenceaspden 21 June 2012 03:45:25PM 20 points [-]

Thank you Luke for this beautifully written post.

A while ago I saw a kindly waitress give my friend's two year old daughter a small cookie in a restaurant. Various emotions flickered across her tiny face, and then she made a decision, accompanied by a small smile.

She broke the cookie into three pieces and gave them to her brothers. Completely unprompted.

I couldn't believe my eyes. I asked my friend, who is a lecturer in experimental psychology, whether altruism was normal amongst very young siblings.

He looked a bit smug and said "Well we put a lot of reinforcement into that."

I hadn't really thought about what that meant until now. Your clear writing has made it obvious.

As a result of your post, I think I'm going to try deliberately modifying some of my own behaviours this way, and maybe try the techniques on some friends. (The first time, by the way, that I've changed my behaviour as a result of reading less wrong, rather than just treating it as philosophical crack.)

For friends it seems that sincere praise / avoiding criticism would be good, but what would you recommend as rewards to self? I'm pretty sure that nicotine and pizza slices would work for me, but I'm also sure that those aren't things I want to do more of.

Comment author: TheOtherDave 22 June 2012 12:33:49PM 8 points [-]

Don't underestimate the power of praise as a self-reward. It feels really goofy to explicitly praise myself -- especially to do it out loud -- but that doesn't mean it doesn't work.

IME, the biggest problem with self-reward, whatever the mechanism, is that it requires quite a lot of discipline to differentially reward the thing I want to reinforce at all consistently.

The only time I ever really maintained that discipline for any length of time was when I was recovering from brain damage, when continued focus on self-improvement was the single most important thing in my life for about 18 months. In my real life, I just don't care that much. YMMV.

Recruiting allies to reward me works better for me.

Comment author: Viliam_Bur 22 June 2012 10:16:30AM *  8 points [-]

For friends it seems that sincere praise / avoiding criticism would be good, but what would you recommend as rewards to self? I'm pretty sure that nicotine and pizza slices would work for me, but I'm also sure that those aren't things I want to do more of.

M&Ms, one piece at a time -- they are small enough. (It would probably be good if you stop eating them in all other circumstances, but that is not big sacrifice.)

Or try a symbolic reward. For example put on your table two glass boxes, put 100 stones in first one, and every time you want to reward yourself, move one stone from the first box to the second one, and congratulate yourself on progress. When all stones are in the second box, give yourself a big reward (pizza or whatever), change the boxes, and start again. (This way the reward is still linked to pizza, but it is less pizza. And you see your progress all the time.)

Comment author: Rain 21 June 2012 03:13:15AM *  12 points [-]

That's why I tried to stay positive when talking about the new SI website. Especially with technical changes like that, the (vocal) negative response can be overwhelming.

Comment author: lukeprog 21 June 2012 03:37:15AM 22 points [-]

Yup. When reading through the comments about the new website, I could feel my effort being punished.

Comment author: pjeby 21 June 2012 04:52:37PM 15 points [-]

Yup. When reading through the comments about the new website, I could feel my effort being punished.

Perhaps you could have somebody read them for you and summarize them in a non-critical way, thus creating a reinforcement shield.

Alternately, you could adapt what internet marketing "personalities" do, and promote doing: practice celebrating criticism. One marketer (I forget which one) described making a practice of throwing his hands in the air and shouting "Woo!" when he received a criticism via email.

(Background: "personality" marketers promote by writing emotionally charged material that's intended to divide their audience into people who either love or hate them. Thus, the presence of hate mail is evidence that their strategy is working. They will then often publicize the hate mail, in order to stir up the emotions of the people on the opposite side of the debate. Talk radio hosts, bloggers, political commentators, etc. also use these strategies, even if they're not always considered "marketers" in a traditional sense. Whether you consider this "dark arts" is largely a political question, since the LW sequences use these tactics also. Whether he knows it or not, Eliezer is a personality marketer in this sense, it's just that he's not as efficiently monetizing the results. ;-) )

Comment author: John_Maxwell_IV 21 June 2012 05:45:32AM 5 points [-]

Sorry about that.

It seems to me that if humans were emotionless utility maximizers, we would prefer hearing criticism over praise, the same way programmers purchase more utility by fixing bugs in their programs than polishing features that already work. I suspect criticism is generally more valuable from a pure decision theoretic perspective.

I wonder if there is an effective way to buy encouragement and criticism separately. Also, it's hard to know exactly how best to encourage folks. In theory it's possible that making a new website is not the best use of SI's resources, which suggests reinforcement would not be optimal. But we still may want to reinforce you towards the more general behavior of taking steps to achieve your organizational goals. So what's the best response?

Maybe someone can develop some general guidelines for reinforcing/criticizing people, similar to what the nonviolent communication people came up with. (When {observable event} happened, I felt {feeling} because I need/value {underlying need that felt unmet or value that felt jeopardized}. Would you be willing to {specific request that person could do} in the future?) E.g. check to see if the person was acting with good intentions and reinforce them for those if they existed, check for super goals you endorse and reinforce them for working to accomplish those, check to see if the person could just have easily have sat around doing nothing and reinforce them for expending effort if this was the case, etc.

I think optimally criticism would have lots more reinforcers associated with it: people should be reinforced for requesting, giving, and receiving criticism because these are all activities that are naturally aversive but actually have high expected value.

So, I wholeheartedly endorse the following actions of yours: attempting to maximize humanity's collective utility function, working on the super goal of AGI safety, actually doing stuff, and deliberately gathering critical feedback. Go Luke!

Comment author: wedrifid 21 June 2012 05:16:20AM 2 points [-]

Yup. When reading through the comments about the new website, I could feel my effort being punished.

I am slightly surprised to hear this. I perhaps expected slightly less emotional involvement with the effort and more of a "<Website minion/>, Go! Fix!" feeling.

Comment author: lukeprog 21 June 2012 05:20:05AM 2 points [-]

What happened is that (1) I felt my effort being punished, and then (2) I sent an email to Nickolai or Kamil asking them to fix X.

Comment author: wedrifid 21 June 2012 05:48:36AM 4 points [-]

(2) I sent an email to Nickolai or Kamil asking them to fix X.

I've noticed (while being such a minion) that when making such change requests yourself manage to do so with a frame that minimises a criticism vibe or 'effort punishment' feelings. I would pay many, many M&Ms for that effort in careful phrasing.

Comment author: wedrifid 21 June 2012 05:26:36AM 10 points [-]

I sent an email to Nickolai or Kamil asking them to fix X.

Great work Nickolai or Kamil, if either of you read lesswrong at all. The website is a much needed improvement! ;)

Comment author: dbaupp 21 June 2012 05:49:48AM *  2 points [-]

I tried to do the same. Although, I was probably significantly less successful than I'd liked to have been (sorry Luke, Nickolai, Kamil and anyone else who'd made an effort!).

Also, given lukeprog's comment, this unfortunately appears to be a case of history repeating itself: matt had a similarly negative experience when LW was redesigned a little while ago.

Comment author: wedrifid 21 June 2012 06:28:39AM 7 points [-]

matt had a similarly negative experience when LW was redesigned a little while ago.

That circumstance is somewhat different in nature. While as far as I know nobody wanted matt to experience negative affect the discouragement of 'effort' was actually a perceived instrumental good, given an expectation that more effort would produce undesired outcomes.

I note that this relies on beliefs at the time. In that context users had to make the prediction "If a website administrator implements detrimental changes when previous discussion had already explained why such a thing was not desired and a prediction had been made that a change to the website would probably be bad, what is the probability that future 'effort' will be beneficial?" The answer is very, very low. The emotional distress matt experience was his social instincts warning him that interfering with the tribe when they do not want you to is a dangerous act - especially when that interference is to (in effect) institute a prohibition against something they could previously do.

It turns out, however, that matt is a superior human being to the typical person in his role. While his ego did cause him to act more defensive than optimal and seemed to cause him to experience emotional distress it did not cripple his ability to respond to user feedback or cause him to lash out with actions against the users as many would. The undesired change was eventually fixed, as were the few bugs that were introduced.

I expect users to drastically update how they would respond to matt if he made future website upgrades due to having more information about matt. He definitely deserves a lot of rewarding for going ahead and doing the bugfixes and implementing 'retraction then deletion' despite having received discouragement. He lives (here) in Melbourne. Perhaps I should give him a packet of M&Ms if I run in to him at one of our meetups!

Comment author: coffeespoons 22 June 2012 10:32:33AM 11 points [-]

I read this post last night. I was in the office late, not because I had a great deal to do, but because I was procrastinating. After reading it, I asked my friend to give me a quick call to say congratulations in a half an hour if I'd finished all the work. It took me 10 minutes to finish! :)

Comment author: Jonathan_Graehl 22 June 2012 11:18:15PM 8 points [-]

But that's probably more of a public commitment effect.

Comment author: Normal_Anomaly 23 June 2012 12:36:57AM 6 points [-]

True. But I bet if coffeespoons makes this a routine thing, they'll eventually find themselves enjoying work more.

Comment author: Vladimir_Golovin 21 June 2012 09:44:46AM 11 points [-]
  1. Nice post SIAI! Have an $5 donation!

  2. I tried a similar reinforcement technique on myself but it didn't stick because I couldn't find a reliable trigger condition for dispensing the reward.

  3. Does this mean that we should stop punishing ourselves for procrastination?

Comment author: Kaj_Sotala 21 June 2012 11:14:47AM *  14 points [-]

Does this mean that we should stop punishing ourselves for procrastination?

My personal experience strongly suggests that "stop punishing yourself for X" helps avoid X, for most if not all X. For instance, becoming a vegetarian was much easier when I didn't try to go cold turkey, but rather was fine with the fact that I would succumb to the lure of eating meat every now and then. When I did, I felt a little guilty, but then shrugged and thought that I'd try better the next time. I still fall victim to that temptation occasionally, but it's much more rare now than it used to be.

This might have something to do with the fact that if you punish yourself for trying and failing, you stop wanting to try in the first place, as it becomes associated with the negative emotions. Also, accepting and being okay with the occasional failure makes you treat it as a genuine choice where you have agency, not something that you're forced to do against your will.

See also It's okay to be (at least a little) irrational.

Comment author: Vladimir_Golovin 21 June 2012 12:05:24PM 5 points [-]

Perhaps this is why I like Autofocus better than GTD. "It is fine to have incomplete tasks in your task list".

Also, non-punishment for failures may be one of the distinctions between play-like work and work-like work.

Comment author: Caspian 18 July 2013 01:35:29PM 1 point [-]

I think I even have work-like play where a game stops being fun. And yes, play-like work is what I want to achieve.

Comment author: Vladimir_Golovin 19 July 2013 06:29:36AM 1 point [-]

In case of work-like play, I have a resolution: stop playing immediately. It doesn't mean quitting the game for good, but rather "end the session now, if a game permits that". Also, this is why I generally don't play games that punish me for leaving early (e.g. WoW raids, DOTA2).

Comment author: CAE_Jones 18 July 2013 02:06:39PM 1 point [-]

Anecdote: I've made suggestions to someone on how he might optimize the time he spends writing for his various projects, and more than once he's responded that a given strategy would make it feel too much like work (I don't remember off hand if he said explicitly that this would be an instrumental problem, or if that was only implied). I'm not really sure how I feel on how I might go about applying this concept, mostly because of my extremely vague definitions of work / play, but I do find that having certain restrictions--something as simple as paper size, for example--tends to make it much easier to work on something. (I wrote a shortstory by specifying what it would need to fit in, and measuring books I'd made in the same format years earlier; I made a large number of maps for a game by using a format restricted to 32 tiles across, etc. I haven't found good ways to apply this strategy to most of what I try to do, though.).

Comment author: JGWeissman 21 June 2012 01:58:56AM 11 points [-]

On Skype with Eliezer, I said: "Eliezer, you've been unusually pleasant these past three weeks. I'm really happy to see that, and moreover, it increases my probability than an Eliezer-led FAI research team will work. What caused this change, do you think?"

Eliezer replied: "Well, three weeks ago I was working with Anna and Alicorn, and every time I said something nice they fed me an M&M."

If I recall my high school psychology class correctly, you can get a stronger and more persistent effect by secretly rolling a dice and note the number, and when Eliezer says that many nice things, give him an M&M, roll the dice again for a new target number of nice things.

Comment author: TheOtherDave 21 June 2012 02:12:48AM 19 points [-]

That's true and false. Intermittent reinforcement gets a more robust effect than continual reinforcement, yes, but randomly intermittent reinforcement isn't as effective as setting the reward threshold higher as the behavior becomes more common... e.g., rewarding only the 10% nicest things.

Comment author: matt 21 June 2012 07:09:46PM 5 points [-]

I want to design a reinforcement schedule in one of our apps. Can anyone link me to some specific guidelines on how to optimise this?

(Reinforce exactly what % of successes (30%? 26%? 8%?)? Reinforce performances in the top 10% of past performances (or the top 12%, or the top 8%?)? How does time factor (if the user hasn't used the app for a week, should I push a reinforcer forward?)?)

Comment author: dbaupp 21 June 2012 05:41:52AM 4 points [-]

Some previous discussion about this form of conditioning.

Comment author: ciphergoth 21 June 2012 06:18:23AM 5 points [-]

When the threshold is "something nice", there's going to be randomness in the reinforcement anyway.

Comment author: Arkanj3l 05 July 2012 05:47:22PM 4 points [-]

So, reinforcement with M&Ms doesn't translate into an addiction for extrinsic rewards and the reduction of intrinsic motivation?

I'm missing something here, I know.

Comment author: EphemeralNight 21 June 2012 09:43:29PM 4 points [-]

The reason you should ignore poor performance if you say "No, you're doing it wrong!" you are inadvertently punishing the effort. A better response to a mistake would be to reinforce the effort: "Good effort! You're almost there! Try once more.

I am probably unusual in this regard, but I think I would find both approaches equally aggravating. If someone points out that I've made a mistake, anything other than a concise detailing of exactly how what I did differs from what I was supposed to do, is just going to irritate me. Also, my brain tends to interpret being ignored as a signal that I'm doing correctly.

Comment author: Swimmer963 21 June 2012 09:56:04PM 2 points [-]

If someone points out that I've made a mistake, anything other than a concise detailing of exactly how what I did differs from what I was supposed to do, is just going to irritate me.

Is this because of the "damn it, I know I made a mistake, you telling me I did doesn't help!" effect? I get that too... A good thought experiment is that if I was making a type of mistake that I couldn't automatically tell I was making on my own, I would prefer it to be pointed out, even if not in a concise detailed fashion–the idea of not knowing that I'm making a mistake is kind of scary. What would your reaction be in that situation?

Comment author: EphemeralNight 21 June 2012 10:23:58PM *  2 points [-]

Is this because of the "damn it, I know I made a mistake, you telling me I did doesn't help!" effect?

No, I react the same way whether I was previously aware of my mistake or not. I only experience that effect when I'm told to do something I am already doing.

A good thought experiment is that if I was making a type of mistake that I couldn't automatically tell I was making on my own, I would prefer it to be pointed out, even if not in a concise detailed fashion–the idea of not knowing that I'm making a mistake is kind of scary. What would your reaction be in that situation?

Pragmatically, we as humans, just barely over the threshold into sapient intelligence, make mistakes we're not aware of constantly. If we didn't, we wouldn't need a superintelligence to fix the world; we'd have already done it ourselves. So finding the concept scary seems kind of pointless.(Sort of like being hydrophobic about the water in one's own body.) However, I would, of course, rather be aware of my mistakes than not.

But none of this is really on the topic, which was that the listed reinforcements don't seem even remotely applicable to humans in a universal way.

Comment author: Swimmer963 22 June 2012 02:16:26AM 2 points [-]

So finding the concept scary seems kind of pointless. However, I would, of course, rather be aware of my mistakes than not.

My actions have impacts on others. In general, I prefer to help other people or at least not harm them–however, I may harm someone by mistake, and I really don't want this to happen. If I make a mistake once and I realize it–fine, hopefully no harm done, I won't do it again. If I make a mistake and I don't know about it, well, maybe no harm done that time in particular, but I'm likely to keep making this mistake over and over, and possibly the first time I'll find out is when there is harm done. I think that justifies finding it scary.

Comment author: Konkvistador 21 June 2012 03:11:37PM 37 points [-]

"Eventually it hit me that the same techniques might work on that stubborn but lovable species, the American wife." "Back in Maine, I began thanking Amy if she threw one dirty shirt into the hamper. If she threw in two, I'd kiss her." "...After two years of exotic animal training, my marriage is far smoother, my wife much easier to love."

Comment author: Kaj_Sotala 26 June 2012 11:53:24AM 16 points [-]

It's probably worth noting that the original article, which lukeprog quoted, ended with this:

PROFESSIONALS talk of animals that understand training so well they eventually use it back on the trainer. My animal did the same. When the training techniques worked so beautifully, I couldn't resist telling my husband what I was up to. He wasn't offended, just amused. As I explained the techniques and terminology, he soaked it up. Far more than I realized.

Last fall, firmly in middle age, I learned that I needed braces. They were not only humiliating, but also excruciating. For weeks my gums, teeth, jaw and sinuses throbbed. I complained frequently and loudly. Scott assured me that I would become used to all the metal in my mouth. I did not.

One morning, as I launched into yet another tirade about how uncomfortable I was, Scott just looked at me blankly. He didn't say a word or acknowledge my rant in any way, not even with a nod.

I quickly ran out of steam and started to walk away. Then I realized what was happening, and I turned and asked, "Are you giving me an L. R. S.?" Silence. "You are, aren't you?"

He finally smiled, but his L. R. S. has already done the trick. He'd begun to train me, the American wife.

Comment author: handoflixue 22 June 2012 06:57:41PM 12 points [-]

This actually bothers me less than the original, simply because the stereotype of "properly raised wife having to train her lower-status husband to act appropriately" is a VERY common social meme, whereas "husband training wife" is something I generally only see in the context of physical abuse (which, given the lack of violence, this obviously isn't).

Is there a cultural meme I'm missing here that makes THIS version the more offensive one? o.o

Comment author: Raemon 22 June 2012 07:14:16PM *  22 points [-]

"Woman Training Man" is generally presented as funny with no negative ramifications. "Husband training wife" is presented in the context of either physical abuse, emotional abuse, or as part of a widespread societal trend of women being "domesticated" which is now generally considered distasteful. If this had been phrased "husband training wife", it wouldn't pattern match to "funny, harmless joke", it'd pattern-match to either abuse or societal oppression. (The abuse angle wouldn't necessarily be accurate, but for many people it would come to mind before the "mirror-image-of-the-woman-training-man" concept did).

So whether it actually makes sense, the example would produce negative affect in many people.

Comment author: TheOtherDave 22 June 2012 07:13:50PM 2 points [-]

No, it sounds like you're aware of the relevant cultural meme.

Comment author: Viliam_Bur 22 June 2012 08:11:03PM *  5 points [-]

"wife training lower-status husband" is a cultural meme

"man abusing woman" is a very strong meme, and "man <something> woman" pattern-matches it

Comment author: private_messaging 23 June 2012 12:30:38PM *  0 points [-]

man abusing woman is not only a very strong "meme", but also a common occurrence due to biological detail of males in mammals generally a: being larger b: being more aggressive and c: likely being naturally more selfish (due to different reproductive role). edit: all I am saying is that there is a biologically justified prior here, that most people use, a body of utterly indisputable evidence across many species of mammals. Except subpar evidence-evaluators, of course, whom do not process the prior and are also subject to Dunning-Kruger effect about it.

Comment author: army1987 25 June 2012 06:58:49PM 1 point [-]

Why the hell was that downvoted? I guess it was supposed to be a descriptive statement but people misunderstood it as a normative one.

Comment author: private_messaging 26 June 2012 08:49:54AM *  -1 points [-]

At least 2 people seem to think you guess wrong.

edit: as of how i interpret reactions to such statements, i have already an explanation for e.g. gaming forums where we have very similar white privileged male nerd demographics. We don't do downvoting there because enabling downvotes lets the white privileged male nerd majority enforce their worldviews and discourage any dissent, which we can not afford because we make games for everyone not just the white privileged male nerd majority. Tho its up to -1 here.

Comment author: lukeprog 22 June 2012 01:31:22AM *  8 points [-]

Have some tact, man. My post was fine, but you... you are a god damned sexist.

Comment author: mapnoterritory 21 June 2012 07:30:51PM *  9 points [-]

Daniel Kahneman in Thinking, Fast and Slow:

I had stumbled onto a significant fact of the human condition: the feedback to which life exposes us is perverse. Because we tend to be nice to other people when they please us and nasty when they do not, we are statistically punished for being nice and rewarded for being nasty.

There reason for that lies in regression to the mean when training (example of flight instructors in the israel airforce):

I pointed out to the instructors that what they saw on the board coincided with what we had heard about the performance of aerobatic maneuvers on successive attempts: poor performance was typically followed by improvement and good performance by deterioration, without any help from either praise or punishment.

Since positive reinforcement is so counterintuitive: don't forget to reward yourself for rewarding somebody for good behaviour! :)

Comment author: faul_sname 22 June 2012 01:47:15AM 9 points [-]

Speaking of regression to the mean, that seems to be one topic that wasn't really covered in the sequences that really should have been.

Comment author: tgb 21 June 2012 02:42:27AM 20 points [-]

I like this article because it is reasonably short, but very clear and highly actionable.

Comment author: sketerpot 21 June 2012 11:21:58PM 12 points [-]

This compliment is particularly effective because it's specific, verifiable, and true. I've never been very good at accepting vague compliments -- I tend to get embarrassed and self-conscious -- but more specific compliments are really nice.

Comment author: XFrequentist 25 June 2012 05:42:34PM 1 point [-]

This explanation of why the complementary comment on the article was effective is itself effective, because it gives specific reasons why the complement is unlikely to evoke the embarrassment sometimes associated with more vague complements.

Comment author: Will_Newsome 21 June 2012 04:51:33AM 15 points [-]

Eagerly awaiting "The Power of Punishment".

Comment author: JulianMorrison 21 June 2012 09:25:43AM 12 points [-]

Anecdotally, punishment seems to be a good guilt-releaser, while guilt is dysthymic. Punishment may be effective at snapping someone out of a blue funk and getting them to be responsive to rewards. Guilty people reject rewards. (The above may work better if you are kinked that way.)

Comment author: wedrifid 21 June 2012 05:10:45AM 8 points [-]

Eagerly awaiting "The Power of Punishment".

Particularly good for demonstrating to observers that you have more status and power than the person you are punishing.

Comment author: Viliam_Bur 21 June 2012 10:04:17AM *  6 points [-]

meh. downvoted.

(just joking)

Comment author: Will_Newsome 21 June 2012 05:14:54AM 1 point [-]

(demonstrating to observers / demonstrating to self / demonstrating to punished; status / power / resources / justification / need / etc; person / cognitive subsystem / institution / problem representation / etc)

Comment author: MBlume 21 June 2012 03:00:14AM 35 points [-]

Good post! Thank you for writing it Luke =)

Comment author: Eliezer_Yudkowsky 21 June 2012 03:57:36AM 28 points [-]

Thanks for reinforcing Luke! And it's great that you applied the theory so quickly!

Comment author: JGWeissman 21 June 2012 04:06:19AM 19 points [-]

Yay recursive reinforcement!

Comment author: Eliezer_Yudkowsky 21 June 2012 04:19:53AM 7 points [-]

Why, thanks! It's helpful to hear you say that!

Comment author: CharlieSheen 21 June 2012 02:35:41PM 23 points [-]

I think I'm going to be ill if this continues.

Comment author: Dorikka 21 June 2012 04:51:16AM -1 points [-]

Moar recursion! Keep it up! :D

Comment author: Will_Newsome 21 June 2012 04:56:58AM 28 points [-]

No. Unreflective happy death spirals get people killed. Shame on all of you for being bad people.

Comment author: RomeoStevens 21 June 2012 05:49:25AM 11 points [-]

I'm glad you mentioned this.

Comment author: Will_Newsome 21 June 2012 05:59:39AM *  9 points [-]

Don't be glad. If you need reinforcement, be relieved. Gladness tends to cause unreflective happy death spirals. Shame on you for being glad.

Presumably the emotion you actually felt was relief, and "glad" was merely used as an inaccurate/misleading synonym? In which case, shame on you for using inaccurate/misleading synonyms.

(I'm totally at least a quarter serious, maybe half.)

Comment author: JGWeissman 21 June 2012 06:06:54AM 12 points [-]

Thank you for wanting us to not have unreflective happy death spirals. I will have to repeat the behavior that caused you to express such caring.

Comment author: Viliam_Bur 21 June 2012 09:31:47AM 20 points [-]

I guess now it's the right time to say big thanks to everyone who didn't contribute to this thread!

Comment author: Will_Newsome 21 June 2012 06:17:24AM *  3 points [-]

I don't want you to not have unreflective happy death spirals, I'm just horrified at the potential consequences of not going out of my way to prevent you from having unreflective happy death spirals. Shame on you for imprecision and/or implicitly accusing me of hypocrisy.

Comment author: arundelo 21 June 2012 03:11:04AM 6 points [-]

I see what you did there!

Comment author: army1987 21 June 2012 08:33:42AM 0 points [-]

(I didn't until EY pointed that out.)

Comment author: CommanderShepard 21 June 2012 03:25:13PM *  5 points [-]

"god this is even more phygish than just that quote about eliezer getting fed mnms"

Comment author: Will_Sawin 21 June 2012 05:33:51PM 4 points [-]

what's "phygish"?

Comment author: shminux 21 June 2012 05:37:50PM 2 points [-]
Comment author: John_Maxwell_IV 22 June 2012 01:34:58AM 2 points [-]

That strikes me as goofy, not phygish.

Comment author: Swimmer963 21 June 2012 01:19:27AM *  11 points [-]

To help someone improve at dance or sport, ignore poor performance but reward good performance immediately, for example by shouting "Good!" (Buzas & Allyon 1981) The reason you should ignore poor performance if you say "No, you're doing it wrong!" you are inadvertently punishing the effort. A better response to a mistake would be to reinforce the effort: "Good effort! You're almost there! Try once more."

I got a demonstration of how true this is yesterday when, during my taekwondo class, I was paired up with one of the senior black belt students, who has some but not a lot of experience teaching. He was supposed to be fixing up my poomsae (same thing as a kata in karate) and each time he watched me do it, I would finish and he would immediately launch into a description of what I was doing wrong. His feedback was pretty useful–specific, with demonstrations of exactly what to change in order to do it right–but without any prelude of "yay, good job!" or even "okay, the punches were way better that time...now let's work on the stances", I found myself getting really discouraged. Reminding myself that I wasn't actually doing worse than usual, that he just had a different teaching style, helped a little... But my subconscious brain still decided to feel resentful and unenthusiastic, no matter how counterproductive that might be towards my actual goal of improving my poomsae.

As a swimming instructor, I do make sure to dole out a LOT of praise, but I'm wondering if I should push it even further...

Comment author: [deleted] 21 June 2012 01:32:15AM *  8 points [-]

I'm not sure a lot of praise is a good idea since that would lower its effectiveness as a reinforcer.

Comment author: Swimmer963 21 June 2012 02:01:00AM 12 points [-]

Well, a lot of non-specific praise would water down the value of non-specific praise as a reinforcer, but taking the time to pick out more specific elements that are good/improving would probably reduce discouragement.

I think one of the things I forget most as an instructor is how easy it is to get discouraged, especially when you're being taught by someone who seems to be able to do all of it effortlessly. There's also the element of "I already know I'm doing it wrong! I just can't get my body to listen to my brain!" Instructors who don't acknowledge this and give praise for trying or noticing that I'm doing it wrong are a major source of discouragement for any new physical skill I try to learn.

Comment author: NancyLebovitz 22 June 2012 03:12:39AM 2 points [-]

"I already know I'm doing it wrong! I just can't get my body to listen to my brain!

Any advice on getting one's body or one's student's body to be more cooperative?

Comment author: Swimmer963 22 June 2012 12:31:34PM 2 points [-]

Break complex movements down into lots of simple movements ("drills") and practice them individually, a lot...then string together the first two simple movements and practice that sequence a lot...then the first three in sequence...etc. Also, don't start by teaching/trying to learn the full complex movement in the first place–always start with the simplest possible subset, master that, and then worry about the next step.

Comment author: TheOtherDave 21 June 2012 01:36:18AM 6 points [-]

"Don't Shoot the Dog" remains my favorite book for these sorts of anecdotes, as well as some of the theory and a lot of the practice. I recommend it.

Comment author: cicatriz 21 June 2012 05:19:54PM 2 points [-]

This seems to contradict the very powerful effect of learning from failure and corrective feedback. See http://www.wired.com/wiredscience/2011/10/why-do-some-people-learn-faster-2/ for an accessible overview.

I'd conjecture this works better when someone can already perform the desired behavior and wants to form a habit, whereas learning from failure comes in when new information needs to be stored and reorganized.

Comment author: Will_Sawin 21 June 2012 05:51:44PM 3 points [-]

That article especially seems to demonstrate the critical importance of choosing what you reinforce, and how your a teacher's model of what they are reinforcing may differ from the students.

Comment author: wedrifid 21 June 2012 04:50:22PM *  7 points [-]

The central lesson I learned from exotic animal trainers is that I should reward behavior I like and ignore behavior I don't. After all, you don't get a sea lion to balance a ball on the end of its nose by nagging. The same goes for the American husband.

Back in Maine, I began thanking Scott if he threw one dirty shirt into the hamper. If he threw in two, I'd kiss him. Meanwhile, I would step over any soiled clothes on the floor without one sharp word, though I did sometimes kick them under the bed. But as he basked in my appreciation, the piles became smaller.

My wife, if pulling that kind of stunt, would quickly find that her affections were shunned and her thanks were met with clear contempt (after she was asked politely not to do that the first time). It is almost certainly not in her interests to produce a pavlovian association between her affections and attempts to control me against my wishes. My aversion to hostile takeover of internal motivations is much stronger than my desire for the affections of any particular individual.

This would be entirely different if I had made a prior agreement regarding shirts and hampers. Making it motivationally easier and more enjoyable to do things I am willing to do is to be encouraged.

Comment author: Viliam_Bur 22 June 2012 11:06:49AM *  5 points [-]

I do accept this kind of reinforcement from my significant other, assuming that:

  • it is for a goal I agree with (extrapolated volition)
  • I am free to say "stop doing this" if I don't feel like to be reinforced; and my wish is respected
  • I do get the same signs of affection in other situations too.

Actually I consider it very useful, and for me it would be a waste not to use this kind of cheap "external willpower". YMMV.

Comment author: pjeby 21 June 2012 05:27:30PM 6 points [-]

My wife, if pulling that kind of stunt, would quickly find that her affections were shunned and her thanks were met with clear contempt

Seriously? You'd shun your wife because she said thank you? i.e.

I began thanking Scott if he threw one dirty shirt into the hamper

Comment author: [deleted] 21 June 2012 06:15:21PM 12 points [-]

Some people react quite viscerally to the awareness that another party is trying intentionally to steer their behavior in any way. It seems to just be a massive squick button for some (indeed, I notice that most randomly-selected people who are made aware of explicit attempts to condition behavior react with discomfort at minimum); for others, there seems to be a correlation with triggers gained from abusive interactions earlier in life; a few I knew who reacted strongly showed strong indications of sociopathy and seemed to instinctively feel violated if someone else successfully, or even just obviously, tried to affect their behavior in a deliberate manner toward some end (a normal part of cognition and social interaction for them directed at others).

Comment author: wedrifid 21 June 2012 06:21:11PM 7 points [-]

Seriously? You'd shun your wife because she said thank you?

(No, I said I would shun kisses delivered under those circumstances. No cutting and pasting of my keywords for the sake of hyperbole thanks.)

If people use their affection in a way that is obviously intended to systematically manipulate me to do things that I do not, in fact, wish to do then yes, of course those instances of affection I will shun. While I know some people are more tolerant to that kind of blatant disrespect I would expect you to at least be able to comprehend the subset of people that will not.

I'm afraid that all women who want kisses to serve the role of doggy treats within our relationship are out of luck. I have yet to experience a problem with having that policy. My model of myself predicts that rewarding hostile-to-my-interests-reward-training with increased compliance or acceptance would leave me with relationships that were far less satisfying and in particular far less enjoyment of displays of affection.

Comment author: TimS 21 June 2012 06:34:20PM 3 points [-]

The question is not whether positive reinforcement is effective in changing your behavior. The question is whether kisses are positive reinforcement in particular contexts.

Suppose your spouse says, "Please pick up my prescription from the store" and you don't want to, but you do it anyway. When you get back, spouse says "Thanks for dealing with that." Do you really think continued experiences like that won't increase the frequency of the behavior "Run an errand even when I don't want to"?

Comment author: [deleted] 21 June 2012 06:40:03PM 1 point [-]

Do you really think continued experiences like that won't increase the frequency of the behavior "Run an errand even when I don't want to"?

I think it depends a lot on her intention. If she says 'thank you' for the purposes of positive reinforcement, I mean if she thinks about her 'thank you's' that way, then I think she's being manipulative.

If she says 'thank you' to say what those words mean, namely, that she's grateful, then even if this does have the effective positive reinforcement there's nothing wrong about her behavior.

Comment author: TheOtherDave 21 June 2012 06:57:05PM 14 points [-]

I find the idea of endorsing manipulative behavior if and only if I remain unaware of the fact that it's manipulative behavior deeply troubling.

It strikes me as similar to saying that hurting people is OK as long as I don't know I'm hurting them. No, it isn't. If hurting people is not OK, then it follows that I ought not hurt people, and learning to recognize when I'm hurting people is part of that, and I ought to learn to recognize it. The behavior doesn't suddenly become "not OK" the moment I learn to recognize it... it never was OK, and now I know it and can improve.

Conversely, if hurting people is OK, then it's OK whether I know I'm doing it or not.

The same goes for manipulating people. Whether I know I'm doing it or not isn't the determiner of whether I'm doing good or ill.

To my mind, the determiner of whether I'm doing good or ill is whether, when I'm done doing it, we're all better off or worse off.

Comment author: Gabriel 22 June 2012 03:41:16PM 2 points [-]

I find the idea of endorsing manipulative behavior if and only if I remain unaware of the fact that it's manipulative behavior deeply troubling.

Awareness of side effects isn't equivalent to intentionality. You can thank someone to express genuine feelings of gratitude. If you wouldn't do that in a counterfactual world in which the gratitude was absent, then I wouldn't call that behavior intentionally manipulative regardless of whether you know about positive reinforcement.

Comment author: TheOtherDave 22 June 2012 04:05:08PM *  6 points [-]

If you wouldn't do that in a counterfactual world in which the gratitude was absent, then I wouldn't call that behavior intentionally manipulative regardless of whether you know about positive reinforcement.

Suppose I am not in the habit of expressing gratitude when people do nice things for me. Never mind why... maybe I was raised wrong. For whatever reason, I'm not in that habit. I feel gratitude, certainly, I just don't express it.

Then one Monday, I learn that expressing gratitude to people for doing nice things for me will increase the odds that they will do it again. Suppose I want people to do nice things for me, and I therefore conclude that I ought to expressing gratitude when people do nice things for me, in order to get them to do it more, and I therefore start expressing gratitude when people do nice things for me, whether I feel gratitude or not.

Then on Wednesday, I learn that this only works when I genuinely do feel gratitude... when I express gratitude I don't actually feel, I get bad results. (Again, it doesn't matter why. Maybe I'm a lousy liar.) So I stop expressing gratitude when people do nice things for me when I don't feel gratitude, but I continue doing so when I do, since that still gets me stuff I want.

If I've understood you correctly, you would call me intentionally manipulative on Tuesday, but not on Thursday. I'm happy to restrict the term "intentionally manipulative" to Tuesday behavior and not Thursday behavior, if that makes communication easier, though I don't use those words that way myself.

Regardless of what words we use, presumably we agree that on both Tuesday and Thursday, I am doing something with the intention of causing changes in other people's behavior, and am doing so without their awareness or consent. Yes?

Do you endorse this on Tuesday?
Do you endorse this on Thursday?

For my own part, I find the idea of endorsing that behavior on Thursday but not on Tuesday deeply troubling, for many of the reasons I listed before.

Comment author: TimS 21 June 2012 07:12:06PM 3 points [-]

I agree with your point, but I think that "manipulate" needs to be tabooed. If we define manipulate as "acts that tend to change the behavior of others" then I agree with your implicit point that it is impossible to interact with others without changing their behaviors, and there is nothing wrong with thinking about how I would like someone else to behave when considering how I interact with them.

That said, there are connotations of manipulate as the word is ordinarily used that are not captured by the way you (and I) are using the word.

Comment author: TheOtherDave 21 June 2012 07:19:32PM 2 points [-]

Sure. I'm perfectly happy to drop the word altogether and instead talk about changing the behavior of others.

Comment author: [deleted] 21 June 2012 06:58:51PM *  2 points [-]

find the idea of endorsing manipulative behavior if and only if I remain unaware of the fact that it's manipulative behavior deeply troubling.

If you don't know you're manipulating someone, you're not manipulating someone. Manipulation is an intentional behavior, like lying, or congratulating, or taking a vow. Knowing what you're doing is part of doing it.

Comment author: TheOtherDave 21 June 2012 07:06:19PM 9 points [-]

Yeah, I pretty much disagree with this statement completely.

Comment author: Vaniver 21 June 2012 06:26:39PM 10 points [-]

So, I have to ask: do you in fact have a wife?

Comment author: handoflixue 22 June 2012 07:41:55PM 6 points [-]

The phrases "of course" and "blatant disrespect" imply a shared frame of reference that doesn't seem to be in evidence. While it might be considered rude to you, it's pretty much human nature. The phrase "thank you" is, as near as I can tell, pretty much entirely meant as a positive reinforcer.

So, having established that we have different frames of reference, can you go in to WHAT behaviors bother you? Is it the use of specific actions as reinforcers ("thank you" is okay but kissing is not?) or is it just the deliberate (as opposed to socialized and subconscious) application of these techniques? Or something else that I'm missing?

Comment author: pjeby 21 June 2012 09:04:15PM 5 points [-]

If people use their affection in a way that is obviously intended to systematically manipulate me to do things that I do not, in fact, wish to do then yes, of course those instances of affection I will shun.

Since positive reinforcement can only be applied after you already do a thing, then presumably, you at least wished to do it once. So, how is providing you with a bonus to something you've already done, manipulating you to do something you don't "wish to do"?

Comment author: wedrifid 22 June 2012 03:04:27AM 2 points [-]

Caveat: I don't know why the husband in question doesn't just put his damn clothes in the hamper. Doesn't the idea of having soiled clothes lying around repulse him anyway? Especially when sharing the space with another. I mean... ewww. But now back to assuming the target behavioral territory is not already granted by the obvious shelling point or prior arrangement.

So, how is providing you with a bonus to something you've already done, manipulating you to do something you don't "wish to do"?

It seems you wish to unilaterally accept rewarding behavior as positive. I don't. I have no trouble detecting when rewards are being used as "approximations" towards a behavioral landscape that I clearly don't want or, especially, have previously declared that I would not accept. I am also able to predict - by reference to past experience and knowledge of my own preferences - that encouraging that reward pattern gives undesired outcomes. As Vaniver mentioned, an important skill to develop is the ability to detect the difference between desired and undesired manipulations.

As a somewhat separate issue, excessive use of physical affection (kisses, hugs, sex) as a "reward" for good behavior changes the experience of those activities - and not in a good way.

Comment author: handoflixue 22 June 2012 07:45:02PM 4 points [-]

excessive use of physical affection (kisses, hugs, sex) as a "reward" for good behavior changes the experience of those activities - and not in a good way.

Could you elaborate on that? I'm entirely okay with physical affection being used as a "reward", as long as it's also clear that the person genuinely wants affection with me, and initiates it "just because" too (actually I'd probably be entirely okay with a strictly reward-based system of affection, as long as it was explicit...)

I have no trouble detecting when rewards are being used as "approximations" towards a behavioral landscape that I clearly don't want

You seem to be assuming, in the example, that the husband doesn't WANT to be modified to put away his laundry. Is that correct?

If so, is it correct that your objection is "you're manipulating me in to a state I don't desire" rather than simply "you're manipulating me"? Given that you PERSONALLY find soiled clothes disgusting, would you PERSONALLY appreciate reinforcement that helped you overcome such a habit?

Comment author: wedrifid 23 June 2012 03:39:59AM 2 points [-]

You seem to be assuming, in the example, that the husband doesn't WANT to be modified to put away his laundry. Is that correct?

Yes.

If so, is it correct that your objection is "you're manipulating me in to a state I don't desire" rather than simply "you're manipulating me"? Given that you PERSONALLY find soiled clothes disgusting, would you PERSONALLY appreciate reinforcement that helped you overcome such a habit?

Yes.

Comment author: pjeby 22 June 2012 08:34:10PM 3 points [-]

Hm. You quoted a question I asked, and then proceeded to not answer it in any way. The question was:

How is providing you with a bonus to something you've already done, manipulating you to do something you don't "wish to do"?

Instead of answering that question, you supplied various generalizations whose referents in physical reality I can't ascertain. Please give an example of a situation where somebody being, say, happy that you did something, means that they are manipulating you to do something you don't "wish to do" (your previous words).

Comment author: TheOtherDave 22 June 2012 09:13:15PM 4 points [-]

Well, I'm not wedrifid, but OK.

Suppose there's a crisis at work, and in response to that crisis I step in and solve a problem.
Suppose, as part of solving that problem, I take some steps (X) that I don't enjoy doing and don't wish to do again.
Suppose my boss notices that I did X and was effective at it and decides that she wants me to do X more regularly, and being familiar with the uses of positive reinforcement decides to hand me a large bonus at our next status meeting. Further, she praises me to the skies in public for having done X, and does so in a way that communicates the (entirely accurate) message that my continuing to receive such praise is contingent on my continuing to do X.

I assert that, in this scenario, my boss is applying positive reinforcement techniques with the goal of increasing my likelihood of doing X, by providing me with a bonus to something I've already done, where X is something I don't wish to do.

Do you agree?

As to whether, in so doing, she's manipulating me... (shrug) I've already had that discussion once too often this week. If our only remaining point of disagreement about that scenario is whether the word "manipulating" properly applies to it, I'm happy to leave that point unresolved.

Comment author: pjeby 23 June 2012 01:25:41AM 0 points [-]

I assert that, in this scenario, my boss is applying positive reinforcement techniques with the goal of increasing my likelihood of doing X, by providing me with a bonus to something I've already done, where X is something I don't wish to do.

So? Are you saying this is a bad thing? That's what I'm asking wedrifid. Are you offended by said boss doing this?

Ironically, in your scenario, your boss is actually elevating your status: trying to please you in order to obtain a consent that in principle could be had by simply ordering you to do more X. So I don't think it's analagous to the situation that upsets wedrifid here.

Comment author: TheOtherDave 23 June 2012 01:37:05AM 4 points [-]

So?

So, you asked for "an example of a situation where somebody being, say, happy that you did something, means that they are manipulating you to do something you don't "wish to do"," and I gave you one.

Apparently, you also wanted an example where the person isn't also elevating my status in the process, isn't trying to please me, and isn't trying to get me to agree to something that they could order me to do. I didn't realize that, sorry.

No, I can't think of any coherent examples where someone tries to use positive reinforcement to alter my behavior by doing something that doesn't please me.

Tapping out now.

Comment author: wedrifid 23 June 2012 02:32:50AM 1 point [-]

Tapping out now.

As am I. I refer any interested observers to the previous comments by myself, TheOtherDave, Vaniver and others, as well as the details of the originally quoted example, including the emphasis on successive approximation. I expect that everyone who wishes to understand will from existing comments and that further engagement would be both futile and constitute a reward of an interaction style which is undesirable.

Comment author: Swimmer963 22 June 2012 08:16:02PM 1 point [-]

What would you see as the difference between a) the story described, and b) a wife who kisses her husband because it makes her happy when he does helpful, nice things, of which putting laundry in the hamper is one, and her automatic response to this surge happiness is "thank you, you're an amazing man!" [kiss]? The latter includes most of the same actions on the part of the wife, and probably occurs in a lot of healthy relationships.

My aversion to hostile takeover of internal motivations is much stronger than my desire for the affections of any particular individual.

Are there some internal motivations that you are less protective of than others? For example, if someone tried to condition me to be less averse to harming people, I would have a pretty big reaction, because that particular internal motivation is sacrosanct to me. But preferences for levels of tidiness...meh. I barely consider that an internal motivation, and definitely not a facet of who I am...it's just a habit, and I don't really care about changing it in either direction.

Is the difference with you that you consider all of your motivations to be a sacrosanct part of who you are? Or just that you place a higher value on your autonomy, and being the one 100% entirely responsible for all of your decisions?

Comment author: TheOtherDave 22 June 2012 09:19:22PM 7 points [-]

It may be worth sharing, anecdotally, that years ago my husband expressed annoyance with me over the fact that I only ever rubbed his back while he was doing dishes, and it made him feel much like how wedrifid describes.

This utterly bewildered me, so I agreed to pay attention to the behavior and see what was going on. Pretty quickly it became clear to me that this was absolutely true, for reasons I wasn't entirely clear on myself, although my working theory was it was the only time that I'd regularly walk past him while he was hunched over in that particular posture, which apparently served as a "give me a backrub" signal for me, for whatever reason.

My response to this was to start giving him random backrubs at other times, which solved the problem.

My point being that (a) being annoyed by this sort of behavior is not at all unique to wedrifid, and (b) whether the behavior pattern is intentional doesn't necessarily matter very much. (I don't mean to suggest that it doesn't matter to wedrifid; actually, they have made it somewhat clear that it's part of what they're objecting to.)

Comment author: Swimmer963 22 June 2012 09:47:16PM 3 points [-]

The main lesson I'm taking from your anecdote is "people are complicated, everyone is complicated in a different way, and for almost any action or behaviour X, there will be a person somewhere who finds it awful." It's hard to guess at the relative numbers without doing a poll, but I'm guessing there's a range of people who wouldn't care if their significant other used physical affection as a reward (or who would even like it, because "yay, more total physical affection!"), and there's a range of people who would find it mildly to extremely unpleasant.

Comment author: TheOtherDave 22 June 2012 09:51:28PM 2 points [-]

I'm guessing there's a range of people who wouldn't care if their significant other used physical affection as a reward (or who would even like it, because "yay, more total physical affection!"), and there's a range of people who would find it mildly to extremely unpleasant.

Yup, that's consistent with my experience.

Comment author: shminux 21 June 2012 10:06:59PM *  3 points [-]

Lessons learned:

  • continue to mentally /ignore people and posts I don't care for on IRC and online forums

  • never comment on bad posts or explain my downvote on LW

  • be more generous with upvoting good contributions and give a short praise when warranted.

Comment author: tgb 23 June 2012 01:44:32PM 5 points [-]

The difference between explaining bad posts and punishing misbehaving dolphins is that the explaining is done for the purpose of the other readers, not just as a punishment.

Comment author: Vaniver 22 June 2012 05:32:19AM 7 points [-]

never comment on bad posts or explain my downvote on LW

This is not quite justified; this is a post on how to use positive reinforcement, not how to use punishment.

Comment author: shminux 22 June 2012 06:02:39AM 1 point [-]

When a dolphin does something wrong, the trainer doesn't respond in any way.

(from the link)

Comment author: Vaniver 22 June 2012 06:30:12AM 3 points [-]

Dolphins are more difficult to punish usefully than humans; for one, they're less likely to understand English.

Moving to object-level advice: I agree that not responding to bad comments or posts is generally a good idea. I think that responding to downvote explanation requests is a good idea about half of the time. Unsolicited downvote explanation is typically done to sway bystander opinion as well as inform the poster, and so deserves its own treatment.

Comment author: dbaupp 22 June 2012 06:26:41AM 3 points [-]

never comment on bad posts or explain my downvote on LW

I think this should be "never downvote".

Comment author: Nornagest 22 June 2012 10:58:03PM *  2 points [-]

I think this should be "never downvote".

Seems to me that a downvote would associate negative valance with both the act of posting on LW and with whatever their specific mistake is, with the latter being stronger. So no vote and a comment with a mixture of praise and criticism is probably the stronger play if you're looking to improve someone's writing or fix some technical mistake while keeping them as a contributor, but a downvote is still effective if all you care about is seeing fewer posts of that kind.

Comment author: CharlieSheen 21 June 2012 02:36:32PM 2 points [-]

We have enough happy death spirals here.

Comment author: wedrifid 21 June 2012 03:22:19PM 2 points [-]

We have enough happy death spirals here.

Who is happy about what?

Comment author: CharlieSheen 21 June 2012 03:28:22PM -2 points [-]

Leave sleeping mind killers lie.

Comment author: wedrifid 21 June 2012 03:35:46PM *  7 points [-]

Your unsubstantiated assertion is rejected. There is nothing that fits that label here. There are things that people like to say that everyone else is in a happy death spiral about but they are too powerfully skeptical to be one of the gullible crowd. This is useless cheap signalling that is a net detriment.

-3 M&Ms for all instances of vague self-reinforcing negativity.

Comment author: CharlieSheen 21 June 2012 03:40:38PM *  4 points [-]

Very well I'll be explicit, I simply wanted to avoid a flame war. Most obvious example:

  • Relationship advice.

Now give me my M&Ms back.

Comment author: wedrifid 21 June 2012 03:54:04PM 2 points [-]

Very well I'll be explicit, I simply wanted to avoid a flame war. Most obvious example:

Relationship advice.

That isn't a Happy Death Spiral. It is a disgraceful mindkiller, sure. But it isn't remotely happy, isn't encouraged by universal reward and absence of criticism. It certainly isn't treated with or caused by the kind of positive feedback Luke's post advocates.

Now give me my M&Ms back.

You can have one back - but being fundamentally confused about what it is you are trying to criticize is only a weak mitigating factor.

Comment author: CharlieSheen 21 June 2012 03:59:08PM *  1 point [-]

That isn't a Happy Death Spiral. It is a disgraceful mindkiller, sure. But it isn't remotely happy, isn't encouraged by universal reward and absence of criticism. It certainly isn't treated with or caused by the kind of positive feedback Luke's post advocates.

Do you remember the online dating profile optimization thread? LessWrong went in Vladimir_M's words "healing crystal equivalent". That thread was a happy death spiral.

Also if you recall the critics in the relationship threads are getting tired and frustrated and just aren't showing up any more, someone even wrote out a full comment to that effect! Evaporative cooling dude. Sure we haven't had a relationship thread since Luke's part I., but its only a matter of time before someone brings it up and the critics won't be there any more.

I only bother because I'm a Charlie Sheen.

Comment author: paper-machine 21 June 2012 03:53:32PM 1 point [-]

Does he have to vomit the M&M's back up?

I really hope that's not the procedure.

Comment author: Vaniver 21 June 2012 04:02:41PM 5 points [-]

Incidentally, chewing M&Ms and then spitting them out is a moderately effective way to wean yourself off of chocolate cravings.

Comment author: [deleted] 21 June 2012 06:23:43PM 2 points [-]

Any suggestions for sugar specifically? I like chocolate and can get it in low-sweet, high-theobromine form, but shaking off sugar cravings would do me a world of good.

Comment author: Vaniver 21 June 2012 06:29:48PM 4 points [-]

From my incomplete understanding of taste psychology, sugar is one of the instinctual taste preferences, whereas things like chocolate are learned taste preferences that are possible to unlearn. I've found that sugar/salt/fat cravings have been useful signals about the quality of my diet, and so would recommend taking a hard look at your diet before trying to alter those signals. (They could be mistuned, but I don't have any advice on how to correctly tune them.)

Comment author: Eliezer_Yudkowsky 21 June 2012 03:02:03PM -1 points [-]

Whatever it is that rationalists are supposed to use instead of death spirals, we don't have enough of it until everything is funded. GO TEAM HAPPINESS!

Comment author: gwern 21 June 2012 04:11:36PM 15 points [-]

'My Little SIAI: Positive Reinforcement is Magic'?

Comment author: CharlieSheen 21 June 2012 03:14:02PM 5 points [-]

No.

Comment author: Strange7 21 June 2012 04:06:16PM 2 points [-]

How long has it been since you had a post that stabilized at net negative votes?

Comment author: wedrifid 21 June 2012 04:11:22PM 7 points [-]

29 March 2012

Comment author: [deleted] 21 June 2012 01:13:19AM 2 points [-]

Excellent article. I wonder if reinforcement could be used to speed up rationality training? I would love to see a study done on that.

Comment author: lukeprog 21 June 2012 01:42:13AM 4 points [-]

I wonder if reinforcement could be used to speed up rationality training?

Almost certainly. CFAR is doing this to some extent at their minicamps, but much more can be done. There is tons of room for rationality training via videogames, for example. Raytheon is developing some debiasing video games for military officers.

Comment author: [deleted] 21 June 2012 01:46:24AM 2 points [-]

Now that is exciting as hell. I've always felt that rationality was a potential competitive advantage for organizations that wasn't being utilized. Way to go.

Comment author: drethelin 21 June 2012 03:41:33PM 0 points [-]
Comment author: wedrifid 22 June 2012 11:59:31AM *  15 points [-]

http://en.wikipedia.org/wiki/Love_bombing

If this genuinely looks like love bombing then it could be an indication that you need more affection in your life to recalibratethe the base rate.

Comment author: faul_sname 24 June 2012 01:45:58AM 4 points [-]

LWers do many cultish things, but I think it's safe to say that's not one of them.

Comment author: Desrtopa 24 June 2012 01:58:33AM 2 points [-]

LWers do many cultish things

How many?

Comment author: faul_sname 24 June 2012 04:44:27AM 5 points [-]

At least 3:

Specifically: foster a distrust of what outsiders say, quotes a lot of stuff by a self-appointed charismatic leader, and emphasize a single solution (rationality) for a large number of problems.

Notable also are the large number of cultish things LWers don't do, such as aggressive recruiting (or really, any recruiting at all).

Comment author: Desrtopa 24 June 2012 05:22:14AM 4 points [-]

quotes a lot of stuff by a self-appointed charismatic leader

I wouldn't exactly call Eliezer a self appointed leader. The community basically accreted around him. If he disavowed being the leader, I think we'd say he was being dishonest or fooling himself.

Not that this is a distinction from cults, the same would probably be true of most of them, I just think it's not quite accurate as a characterization.

Oh, also I think most cult leaders probably have more charisma off the internet.

Comment author: faul_sname 24 June 2012 05:31:52AM 4 points [-]

Oh, probably. I hear Luke has more real-life charisma... Though he kind of kills the "fosters a distrust of outside sources" with the amount he cites outside sources.

Comment author: wedrifid 24 June 2012 06:19:57AM *  9 points [-]

Oh, probably. I hear Luke has more real-life charisma... Though he kind of kills the "fosters a distrust of outside sources" with the amount he cites outside sources.

Quite a lot of charisma, but nothing near the level a cult leader would need to pull off a personality cult. (Although he could probably make up for this if he really wanted to by spending a few weeks reading up research on cult formation then applying it systematically as a 'how to' guide.)

Comment author: Swimmer963 24 June 2012 08:41:09AM 3 points [-]

Quite a lot of charisma, but nothing near the level a cult leader would need to pull off a personality cult. (Although he could probably make up for this if he really wanted to by spending a few weeks reading up research on cult formation then applying it systematically as a 'how to' guide.)

I would like to see Lukeprog post an article on that topic. It would be fascinating.

Comment author: wedrifid 24 June 2012 09:05:49AM 5 points [-]

I would like to see Lukeprog post an article on that topic. It would be fascinating.

Fascinating but suboptimal signalling.

Comment author: sketerpot 22 June 2012 12:47:01AM *  4 points [-]

You realize that almost all people express appreciation or displeasure routinely, right? It's a normal and reasonable part of human interaction, and it's a skill that someone can try to improve without needing to feel too conflicted. Love bombing is far more extreme than anything that this post even touched on. So, while we're linking to things, here's one:

http://lesswrong.com/lw/md/cultish_countercultishness/

Comment author: Viliam_Bur 22 June 2012 10:33:55AM 5 points [-]

Love bombing is just a tool -- its morality depends on how it is used. In a typical situation it is used to ruin the person's natural resistance towards groups that exploit them; that is obviously evil.

A different thing would be to use love bombing with the person's explicit consent, as a reinforcement for things the person values, and for nothing else. Preferably for a limited time specified in advance. It could be a great tool to overcome akrasia.

Comment author: MarkusRamikin 22 June 2012 11:01:31AM 4 points [-]

love bombing with the person's explicit consent

That sounds even more creepy. I like it.

Comment author: Gastogh 21 June 2012 10:26:57AM 1 point [-]

On Skype with Eliezer, I said: "Eliezer, you've been unusually pleasant these past three weeks. I'm really happy to see that, and moreover, it increases my probability than an Eliezer-led FAI research team will work. What caused this change, do you think?"

Eliezer replied: "Well, three weeks ago I was working with Anna and Alicorn, and every time I said something nice they fed me an M&M."

Made me smile. Thanks for sharing.

Comment author: Viliam_Bur 21 June 2012 10:59:22AM 7 points [-]

Hopefully now that the experiment is over, they will return to the original schedule of giving M&Ms for new HPMoR chapters. Seriously, people are suffering here. :D

Comment author: Benquo 21 June 2012 01:54:18PM 12 points [-]

Too infrequent. They need to start by giving him an M&M every time he thinks about writing more HPMoR.

Comment author: gwern 21 June 2012 04:10:48PM 9 points [-]

But then when he starts actually writing, Eliezer will become diabetic!

Comment author: Benquo 21 June 2012 05:59:41PM *  6 points [-]

Shush, don't give away the plan!

But seriously, one can always increase the reward threshold once the first behavior has been firmly established.

Comment author: wedrifid 21 June 2012 04:31:54PM 5 points [-]

But then when he starts actually writing, Eliezer will become diabetic!

If he gets into flow quickly he could be safe. That would mean he is writing more HPMoR but not thinking about writing HPMoR.

Comment author: hvass 21 June 2012 04:22:17AM 1 point [-]

Thanks, Luke! I've always enjoyed this sequence. (It's funny that I was tempted to include a note that I would've been happier if you contributed to the sequence more often, but let's stick with the praise for now. :-)

Comment author: roland 22 June 2012 10:58:11PM *  -2 points [-]

Edit: relevant quotes from the post:

When trying to maintain order in a class, ignore unruly behavior and praise good behavior (Madsen et al. 1968; McNamara 1987).

To help someone improve at dance or sport, ignore poor performance but reward good performance immediately, for example by shouting "Good!" (Buzas & Allyon 1981) The reason you should ignore poor performance if you say "No, you're doing it wrong!" you are inadvertently punishing the effort. A better response to a mistake would be to reinforce the effort: "Good effort! You're almost there! Try once more."

Reward opinion-expressing to get people to express their opinions more often

Now that we all know this, shouldn't we abolish downvotes? From my personal experience the emotional impact of a downvote is extremely frustrating and not helpful at all. The message I get from a downvote is "You are wrong!" or "What you said doesn't agree with the group consensus so we will punish you for it!". I don't see this as constructive in any sense.

Comment author: RichardKennaway 23 June 2012 07:48:41AM *  1 point [-]

The message I get from a downvote is "You are wrong!" or "What you said doesn't agree with the group consensus so we will punish you for it!".

The message I get from a downvote is "Someone did not like this." Obviously, that person is wrong. :-)

ETA: -2! Two people did not like this! I die. My brain turns into maggots which burst from my skull and multiply until they devour the world. All die. O the embarrassment.

Comment author: Jonathan_Graehl 22 June 2012 11:17:27PM 0 points [-]

I think downvotes are generally useful to other readers (though it's odd that the parent suggestion has one as I type), but I agree that people should be protected from the discouraging effect of an early, single downvote. So, why not postpone displaying the negative score to the user for long enough for possible upvotes to counter? (I don't volunteer to implement this).

Comment author: TimS 22 June 2012 11:51:46PM 6 points [-]

The fact that reinforcement can be very effective in changing frequency of behavior doesn't say that punishment should never be used to change the frequency of behavior.

Reinforcement is useful for increasing frequency of behavior. When decreased frequency of behavior is desired, punishment is the type of intervention to use. (For applied behavior analysis, those are the definitions of reinforcement and punishment).

Comment author: Jonathan_Graehl 22 June 2012 11:56:21PM 2 points [-]

Sure. Although I wasn't clear about this, I had in mind the common case of a non-punishing downvoter who simply disagrees with the comment (or wants to see less of its ilk) without saying why. In case punishment is the desired effect, you're right - immediate is better.

Comment author: TheOtherDave 23 June 2012 01:43:11AM 4 points [-]

Be aware that some people upvote comments "back to zero" that they wouldn't otherwise upvote. (Some other people consider this bad practice.)

Comment author: hrishimittal 22 June 2012 07:58:22AM 1 point [-]

What expert timing, Luke! Just two days ago, I came across the fascinating practice of clicker training for horses - http://www.theclickercenter.com, while reading Kathy Sierra's old blog - http://headrush.typepad.com/creating_passionate_users/2006/03/clicker_trained.html.

My only problem is that I need to train my own behaviour rather than someone else's. I'm going to try to use these techniques on myself, although I'm not sure if that's supposed to work.

Comment author: [deleted] 21 June 2012 04:12:59PM 0 points [-]

Maybe it's because behaviorist techniques like reinforcement feel like they don't respect human agency enough. But if you aren't treating humans more like animals than most people are, then you're modeling humans poorly.

But treating human beings, especially adults, like animals is characteristically unethical. Applying some system of reinforcement where someone has asked you to effectively treat their behavior is innocuous enough, as is of course treating yourself.

But generally manipulating the behavior of other people by means other than convincing them that they should behave in a certain way seems to me to be almost definitional of a dark art. If that's not controversial, then I think this article should be qualified appropriately: never do this to other people without their explicit consent.

Comment author: Vaniver 21 June 2012 06:20:18PM 9 points [-]

But treating human beings, especially adults, like animals is characteristically unethical.

It seems to me like the flow is in the reverse direction: many unethical manipulations involve treating adults like animals. But people who skillfully use positive reinforcement are both more pleasant to be around and more effective- which seems like something ethical systems should point you towards, not away from.

Comment author: [deleted] 21 June 2012 06:28:39PM 2 points [-]

That's a fair point: I may have been treating a conditional like a bi-conditional. I think my sense of the matter is this: if a friend told me that he spent a lot of our time together thinking through ways to positively reinforce some of my behaviors, even to my benefit, I would become very suspicious of him. I would feel that I'd been treated as a child or a dog. His behavior would seem to me to be manipulative and dishonest, and I think I would feel this way even if I agreed that the results of his actions were on the whole good and good for me.

Do you think this sort of reaction on my part would be misguided? Or am I on to something?

Comment author: Vaniver 21 June 2012 06:51:20PM *  11 points [-]

I agree with you that your autonomy is threatened by the manipulations of others. But threats only sometimes turn into harm- distinguishing between manipulations you agree with and disagree with is a valuable skill.

Indeed, there's a general point that needs to be made about human interaction, and another about status, but first a recommendation: try to view as many of your actions as manipulations as possible. This will help separate out the things that, on reflection, you want to do and the things that, on reflection, you don't want to do. For example:

if a friend told me that he spent a lot of our time together thinking through ways to positively reinforce some of my behaviors, even to my benefit, I would become very suspicious of him. I would feel that I'd been treated as a child or a dog. His behavior would seem to me to be manipulative and dishonest,

Emphasis mine. The reaction- of calling his behavior manipulative and dishonest- feels like it punishes manipulation, which you might want to do to protect your autonomy. But it actually punishes honesty, because the trigger was your friend telling you! Now, if your friend wants to change you, they'll need to try to do it subtly. Your reaction has manipulated your friend without his explicit consent- and probably not in the direction you wanted it to.

So, the general point: human social interaction is an incredibly thorny field, in part because there are rarely ways to learn or teach it without externalities. Parents, for example, tell their children to share- not because sharing is an objective moral principle, but because it minimizes conflict. As well, some aspects of human social interaction are zero sum games- in which people who are skilled at interaction will lose if others get better at interaction, and thus discourage discussions that raise general social interaction skills.

The status interpretation: generally, manipulation increases the status of the manipulator and decreases the status of the manipulated. Resistance to manipulation could then be a status-preserving move, and interest in manipulation could be a status-increases move. What articles like this try to do is lower the status effects of manipulation (in both directions)- Luke proudly recounts the time Eliezer manipulated him so that he could better manipulate Eliezer. If being molded like this is seen more positively, then resistance to being molded (by others in the community) will decrease, and the community will work better and be happier. As well, I suspect that people are much more comfortable with manipulations if they know how to do them themselves- if positive reinforcement is a tool used by creepy Others, it's much easier to dislike than if it's the way you got your roommate to finally stop annoying you.

Comment author: wedrifid 21 June 2012 06:59:35PM 7 points [-]

distinguishing between manipulations you agree with and disagree with is a valuable skill.

This, with extra emphasis!

Comment author: [deleted] 21 June 2012 07:36:11PM 3 points [-]

I think it's misguided personally. You're already being manipulated this way by your environment whether or not you realize it.

Comment author: [deleted] 21 June 2012 07:51:48PM 2 points [-]

You're already being manipulated this way by your environment whether or not you realize it.

Well, I'm claiming that this kind of manipulation is often, even characteristically, unethical. Since my environment is not capable of being ethical or unethical (that would be a category mistake, I think) then that's not relevant to my claim.

Comment author: TheOtherDave 21 June 2012 05:00:27PM 8 points [-]

But treating human beings, especially adults, like animals is characteristically unethical.

This statement without context is clearly incorrect; there are all sorts of behaviors we can ethically execute with respect to both humans and other animals. I understand that what you and the OP both mean to connote is particular behaviors which we restrict in typical contexts only to non-human animals, but if you're going to label them as unethical when applied to humans it helps to specify what behaviors and context those are.

manipulating the behavior of other people by means other than convincing them that they should behave in a certain way seems to me to be almost definitional of a dark art.

That's a little more specific, but not too much, as I'm not really sure what you mean by "convincing" here.

That is, if at time T1 I don't exhibit behavior B and don't assert that I should exhibit B, and you perform some act A at T2 after which I exhibit B and assert that I should exhibit B, is A an act of convincing me (and therefore OK on your account) or not (and therefore unethical on your account)? How might I test that?

never do this to other people without their explicit consent

This, on the other hand, is clear. Thank you.
I disagree with it strongly.

Comment author: TimS 21 June 2012 05:49:33PM *  2 points [-]

Eliezer replied: "Well, three weeks ago I was working with Anna and Alicorn, and every time I said something nice they fed me an M&M."

That story doesn't trouble you at all?

For most people, there's lots of low hanging fruit from trying to recognize when they are reinforcing and punishing behaviors of others. Also, positive reinforcement is more effective at changing behavior than positive punishment.

But that doesn't mean that we should embrace conditioning-type behavior-modification wholesale. I'm highly doubtful that conditioning responses are entirely justifiable by decision-theoretic reasons. And "not justifiable by decision theoretic reasons" is a reasonable definition of non-rational. Which implies that relying on those types of processes to change others behaviors might be unethical.

Comment author: TheOtherDave 21 June 2012 06:51:07PM 4 points [-]

Does it trouble me at all? I suppose. Not a huge amount, but some. Had Esar said "Doing this to people without their consent is troubling" rather than "never do this to other people without their explicit consent" I likely wouldn't have objected.

My response to the rest of this would mostly be repeating myself, so I'll point to here instead.

More generally, "conditioning-type behavior-modification" isn't some kind of special category of activity that is clearly separable from ordinary behavior. We modify one another's behavior through conditioning all the time. You did it just now when you replied to my comment. Declaring it unethical across the board seems about as useful as saying "never kill a living thing."

Comment author: philh 21 June 2012 10:12:05AM 0 points [-]

I think next time I go shopping, I'll buy a pack of M&Ms, and take one whenever I make a git commit.

Comment author: thomblake 21 June 2012 02:17:20PM 2 points [-]

Careful not to over-reinforce! Think of the commit logs!

Comment author: Caspian 18 July 2013 01:04:50PM 1 point [-]

I just read Don't Shoot The Dog, and one of the interesting bits was that it seemed like getting trained the way it described was fun for the animals, like a good game. Also as the skill was learnt the task difficulty level was raised so it wasn't too easy. And the rewards seemed somewhat symbolic - a clicker, and being fed with food that wasn't officially restricted outside the training sessions.

Thinking about applying it to myself, having the reward not be too important outside the game/practise means I'm not likely to want to bypass the game to get the reward directly. Having the system be fun means it's improving my quality of life in that way in addition to any behaviour change.

I haven't done much about ramping up the challenge. How does one make doing the dishes more challenging?

But I did make sure to make the rewards quicker/more frequent by rewarding subtasks.

Comment author: tsakinis 28 September 2012 12:55:14AM 1 point [-]

Wow, thanks for this great article that was the final piece of information that tipped me over towards getting my shit together. Within 10 minutes after reading it and browsing the comments, I was on my bicycle going to buy small treats I like, that I now give myself for every achieved small goal (~2-10 min of work).

I now wonder though if maybe I should give myself another reinforcer when starting to work with a new goal, otherwise maybe I will only strive for finishing as fast as possible, but starting with a new small goal won't be that much reinforced? Maybe this is my mind trying to get more candy though, so I would be thankful for outside perspective.