= 27d567bc2d48d9f40e9ccc20ea370b15)
The Power of Reinforcement
Part of the sequence: The Science of Winning at Life
Also see: Basics of Animal Reinforcement, Basics of Human Reinforcement, Physical and Mental Behavior, Wanting vs. Liking Revisited, Approving reinforces low-effort behaviors, Applying Behavioral Psychology on Myself.
Story 1:
On Skype with Eliezer, I said: "Eliezer, you've been unusually pleasant these past three weeks. I'm really happy to see that, and moreover, it increases my probability than an Eliezer-led FAI research team will work. What caused this change, do you think?"
Eliezer replied: "Well, three weeks ago I was working with Anna and Alicorn, and every time I said something nice they fed me an M&M."
Story 2:
I once witnessed a worker who hated keeping a work log because it was only used "against" him. His supervisor would call to say "Why did you spend so much time on that?" or "Why isn't this done yet?" but never "I saw you handled X, great job!" Not surprisingly, he often "forgot" to fill out his worklog.
Ever since I got everyone at the Singularity Institute to keep work logs, I've tried to avoid connections between "concerned" feedback and staff work logs, and instead take time to comment positively on things I see in those work logs.
Story 3:
Chatting with Eliezer, I said, "Eliezer, I get the sense that I've inadvertently caused you to be slightly averse to talking to me. Maybe because we disagree on so many things, or something?"
Eliezer's reply was: "No, it's much simpler. Our conversations usually run longer than our previously set deadline, so whenever I finish talking with you I feel drained and slightly cranky."
Now I finish our conversations on time.
Story 4:
A major Singularity Institute donor recently said to me: "By the way, I decided that every time I donate to the Singularity Institute, I'll set aside an additional 5% for myself to do fun things with, as a motivation to donate."
The power of reinforcement
It's amazing to me how consistently we fail to take advantage of the power of reinforcement.
Maybe it's because behaviorist techniques like reinforcement feel like they don't respect human agency enough. But if you aren't treating humans more like animals than most people are, then you're modeling humans poorly.
You are not an agenty homunculus "corrupted" by heuristics and biases. You just are heuristics and biases. And you respond to reinforcement, because most of your motivation systems still work like the motivation systems of other animals.
A quick reminder of what you learned in high school
- A reinforcer is anything that, when it occurs in conjunction with an act, increases the probability that the act will occur again.
- A positive reinforcer is something the subject wants, such as food, petting, or praise. Positive reinforcement occurs when a target behavior is followed by something the subject wants, and this increases the probability that the behavior will occur again.
- A negative reinforcer is something the subject wants to avoid, such as a blow, a frown, or an unpleasant sound. Negative reinforcement occurs when a target behavior is followed by some relief from something the subject doesn't want, and this increases the probability that the behavior will happen again.
What works
- Small reinforcers are fine, as long as there is a strong correlation between the behavior and the reinforcer (Schneider 1973; Todorov et al. 1984). All else equal, a large reinforcer is more effective than a small one (Christopher 1988; Ludvig et al. 2007; Wolfe 1936), but the more you increase the reinforcer magnitude, the less benefit you get from the increase (Frisch & Dickinson 1990).
- The reinforcer should immediately follow the target behavior (Escobar & Bruner 2007; Schlinger & Blakely 1994; Schneider 1990). Pryor (2007) notes that when the reward is food, small bits (like M&Ms) are best because they can be consumed instantly instead of being consumed over an extended period of time.
- Any feature of a behavior can be strengthened (e.g., its intensity, frequency, rate, duration, persistence, its shape or form), so long as a reinforcer can be made contingent on that particular feature (Neuringer 2002).
Example applications
- If you want someone to call you, then when they do call, don't nag them about how they never call you. Instead, be engaging and positive.
- When trying to maintain order in a class, ignore unruly behavior and praise good behavior (Madsen et al. 1968; McNamara 1987).
- Reward originality to encourage creativity (Pryor et al. 1969; Chambers et al. 1977; Eisenberger & Armeli 1997; Eisenberger & Rhoades 2001).
- If you want students to understand the material, don't get excited when they guess the teacher's password but instead when they demonstrate a technical understanding.
- To help someone improve at dance or sport, ignore poor performance but reward good performance immediately, for example by shouting "Good!" (Buzas & Allyon 1981) The reason you should ignore poor performance if you say "No, you're doing it wrong!" you are inadvertently punishing the effort. A better response to a mistake would be to reinforce the effort: "Good effort! You're almost there! Try once more."
- Reward honesty to help people be more honest with you (Lanza et al 1982).
- Reward opinion-expressing to get people to express their opinions more often (Verplanck 1955).
- You may even be able to reinforce-away annoying involuntary behaviors, such as twitches (Laurenti-Lions et al. 1985) or vomiting (Wolf et al. 1965).
- Want a young infant to learn to speak more quickly? Reinforce their attempts at vocalization (Ramely & Finkelstein 1978).
- More training should occur via video games like DragonBox, because computer programs can easily provide instant reinforcement many times a minute for very specific behaviors (Fletcher-Flinn & Gravatt 1995).
For additional examples and studies, see The Power of Reinforcement (2004), Don't Shoot the Dog (2006), and Learning and Behavior (2008).
I close with Story 5, from Amy Sutherland:
For a book I was writing about a school for exotic animal trainers, I started commuting from Maine to California, where I spent my days watching students do the seemingly impossible: teaching hyenas to pirouette on command, cougars to offer their paws for a nail clipping, and baboons to skateboard.
I listened, rapt, as professional trainers explained how they taught dolphins to flip and elephants to paint. Eventually it hit me that the same techniques might work on that stubborn but lovable species, the American husband.
The central lesson I learned from exotic animal trainers is that I should reward behavior I like and ignore behavior I don't. After all, you don't get a sea lion to balance a ball on the end of its nose by nagging. The same goes for the American husband.
Back in Maine, I began thanking Scott if he threw one dirty shirt into the hamper. If he threw in two, I'd kiss him. Meanwhile, I would step over any soiled clothes on the floor without one sharp word, though I did sometimes kick them under the bed. But as he basked in my appreciation, the piles became smaller.
I was using what trainers call "approximations," rewarding the small steps toward learning a whole new behavior...
Once I started thinking this way, I couldn't stop. At the school in California, I'd be scribbling notes on how to walk an emu or have a wolf accept you as a pack member, but I'd be thinking, "I can't wait to try this on Scott."
...After two years of exotic animal training, my marriage is far smoother, my husband much easier to love.
Next post: Rational Romantic Relationships Part 1
Previous post: The Good News of Situationist Psychology
My thanks to Erica Edelman for doing much of the research for this post.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
= f037147d6e6c911a85753b9abdedda8d){kind=logo}
Comments (467)
It's probably worth noting that the original article, which lukeprog quoted, ended with this:
This actually bothers me less than the original, simply because the stereotype of "properly raised wife having to train her lower-status husband to act appropriately" is a VERY common social meme, whereas "husband training wife" is something I generally only see in the context of physical abuse (which, given the lack of violence, this obviously isn't).
Is there a cultural meme I'm missing here that makes THIS version the more offensive one? o.o
"Woman Training Man" is generally presented as funny with no negative ramifications. "Husband training wife" is presented in the context of either physical abuse, emotional abuse, or as part of a widespread societal trend of women being "domesticated" which is now generally considered distasteful. If this had been phrased "husband training wife", it wouldn't pattern match to "funny, harmless joke", it'd pattern-match to either abuse or societal oppression. (The abuse angle wouldn't necessarily be accurate, but for many people it would come to mind before the "mirror-image-of-the-woman-training-man" concept did).
So whether it actually makes sense, the example would produce negative affect in many people.
No, it sounds like you're aware of the relevant cultural meme.
"wife training lower-status husband" is a cultural meme
"man abusing woman" is a very strong meme, and "man <something> woman" pattern-matches it
I agree with all of those statements, and am left with the sense that you were trying to convey an additional message that I didn't quite get.
Just an observation of sexism in our society. We are hypersensitive about anything negative that happens to women (it is a great opportunity for signalling moral superiority above people who are not outraged), while misfortunes of low-status males are just funny (signalling care about them is low-status).
How exactly does this happen? How exactly appears the paradox that this unequal reaction is percieved as fair, while complaining about it can be so easily labeled as sexist?
There is an obvious evolutionary explanation (low-status males are expendable, there is no advantage for high-status males or any-status females to care about them), but how does the algorithm feel from inside? First, there is a rationalization that problems of low-status males are either not real, or could (and should) be easily avoided by them, so if they don't avoid the situations, they obviously deserve the consequences. (Unless they are members of some minority, in which case it is OK to express moral outrage about the opression of given minority.) Second, we are hyper-sensitivised by feminism about everything related to women, because even the smallest joke means that you are a supporter of patriarchy and rape culture, which makes you a complice in every abuse and murder and whatever. There are no innocent jokes about women. Saying your wife "thank you" for doing something nice for you is just a first step on a slippery slope of evil male behavior. (And no, there is no female privilege, and if you have a misunderstood word, go read feminism 101 until you accept it.)
There. Sorry for the mindkilling, I don't know how to write it better without spending too big part of a weekend online.
EDIT: related video
I seem to recall having seen at least one introduction to feminism which did acknowledge that there are forms of female privilege (e.g. children usually end up with the mother after divorces), even though far fewer than forms of male privilege (their list was about an order of magnitude shorter). (This made me find that introduction much more credible, as otherwise it would have failed Policy Debates Should Not Appear One-Sided.)
I would have more respect for such introduction, too, for pretty much the same reasons.
There are several such, but they don't tend to inspire quite as strong a reaction as the ones the OC is reacting to.
OK. Thanks for being explicit.
Interesting I have seen research that suggests a major difference in perceptions between men and women. Men tend to assess the average woman as, well, average in overall attractiveness. Women tend to assess about 80% of men as below average. So in a monogamous society women tend to think they have settled too low.
Such a gap in perceptions would make sense in a polygamous society where a few men at the top have most of the women - so the women marry up, and end up perceiving this as normal. From my reading, most hunter gatherer societies were polygamous.
Have some tact, man. My post was fine, but you... you are a god damned sexist.
Bit of a tangent, but if you ever run across someone for whom this doesn't seem to work, check the hypothesis that they don't parse praise as a positive reinforcer. I don't know how common this is, but I actually have to make a conscious effort to keep it from acting as a mild punishment in most cases when it's applied to me. (Ditto M&Ms in the given context, I expect. Attention Bad.)
You are correct that there are many kinds of reinforcers, and it's important to make sure that the one you choose to use is something the receiver will desire.
-Learning and Behavior, p149
Furthermore at least one person I know (er, myself) picks up on any sort of test-like or game-like or we're-judging-you-so-you-better-not-screw-up-like context and starts acting in extremely confusing/uninformative/atypical/misleading ways so as not to be seen as the kind of person who is easily manipulable (there are probably other motivations involved too). Any incentive structure I'm put under thus has to somehow take this into account, even e.g. the LessWrong karma system. Explicitly manipulative socially mediated praise/M&Ms would strike my brain as outright evil and would stand some chance of being inverted entirely. That said I don't get the impression this sort of defense mechanism is very common.
Excellent insight. Downvoted.
So you are saying that, to change your mode of behavior, all one has to do is create a judging context? That would actually make you very easy to manipulate..
I experience this as common, but I suspect it's because of a small number of exceptionally vocal "manipulation is evil!" types in my life, rather than a larger number of typically vocal ones.
Yes, the situation is usually not so easy that behavior is just a result of inputs, like this:
output := f(input)
People have minds, and a mind is an environment, different for different people. The real equation would be more like this:
[mind1, output] := f([mind0, input])
For example many people like attention of others, but some people may be trained (for example by a previous abuse) that attention of others is usually followed by pain. For them, a positive reinforcement by giving them attention wouldn't work, because the important things is not the attention per se, but what it means for them.
On a meta level, for someone even the idea of "learning" or "improving" or "changing" may be already associated with pain, so they will resist any such process if they notice it. A human mind can be messed up rather easily.
This becomes painfully common (and obvious to any observant third-party that knows these concepts) for subjects that students "are just not made for", such as large amounts of students that "just don't get" maths. They've been trained in so many ways to associate actual learning (especially the actions taken when attempting to learn a concept) with negativity that it becomes obviously so much more rewarding to just guess the teacher's password, and so they are positively reinforced into doing everything they can to avoid mental modeling and seek password-guessing through aggregation and correlation of symbol-data. In most cases I've observed, they become experts at the skill of subconsciously forming "truth-tables" of teachers' passwords through brute-force trial-and-error tactics. What's more, this tactic, which they've been trained to do and learned so well and associate so much with positive feedback, often feeds itself into a vicious circle through several possible methods, which makes getting out (or, for that matter, even realizing that it's there and you need to get out of it) so much more difficult than if that behavior had been blocked immediately when it first appeared.
When I realized that, I've started to feel sad for every student I see showing signs of spending hours upon hours studying and memorizing and headsmashing against the same math problems "until they finally understand them", when in truth they haven't really gained anything worthwhile (IMO) from the experience.
I'd have to say that it shouldn't be that common. Most people want to be praised.
Most people want to be sincerely praised. Someone who reads this post and applies it poorly is going to be saying praise while their body language says something else entirely. Or acting out of character for themselves, leading the reinforcee to suspect that the praise is insincere. Or they may go around praising seemingly everything, causing the reinforcee to interpret the praise as meaningless noise.
There are lots of ways for using praise as reinforcement to go wrong, and if someone is in one of those environments for long enough they will end up being conditioned to interpret praise as neutral or negative.
I suspect it is common enough that when you observe that praising someone doesn't reinforce their behavior or makes them uncomfortable, you should consider that they might have an unusual aversion to praise.
And also, that you might just be really bad at it. ;-)
This was my problem for quite a while: believing that I ought to praise people, while alieving that there wasn't anything to praise and that they didn't deserve it, due to all their obvious imperfections.
This, as you can imagine, produced sub-optimal results. ;-)
Yep. It's not a situation you're likely to come across often, but when you do, it's worth having the alternate theory available to check.
I would expect the rate at which people run counter to "usual" reinforcers to be far less than the rate at which people claim to run counter to "usual" reinforcers. Humans are not naturally very good at reflection of certain sorts.
Having said that I imagine reinforcing techniques would work brilliantly on me whether I knew I was being trained or not. I'd rather know I was being trained (and therefore know what I was being trained to do) and I would therefore wish to reinforce the trainer's openness about the process by having it work better when s/he is open. But even when I didn't realize it, I have every reason to believe that I am not some sort of exception to human neurobiology.
Yes, which is why I said 'someone for whom this doesn't seem to work', not 'someone who claims that this doesn't work on them' - though of course in the latter case it's at least polite to humor them.
I also didn't say that reinforcing techniques don't work on me - I've never run into anyone for whom that was even remotely plausible, in fact. Just, you have to use things that don't squick me out as positive reinforcers, and overt praise and rewards aren't in that category.
Good post! Thank you for writing it Luke =)
Thanks for reinforcing Luke! And it's great that you applied the theory so quickly!
Yay recursive reinforcement!
Why, thanks! It's helpful to hear you say that!
I think I'm going to be ill if this continues.
I see what you did there!
"god this is even more phygish than just that quote about eliezer getting fed mnms"
what's "phygish"?
rot13 it
That strikes me as goofy, not phygish.
Reason #228 I'm crazy and irrational: Without conscious attention to the reinforcement process, my behaviors are selected for reinforcement almost at random. The process selecting behaviors for reinforcement has tons of steps in it like "Did I happen to glance in the direction of the bag of M&Ms right now?" instead of "Is the thing I'm doing now something I want to reinforce?"
(nods) For my own part, it's frequently worse than random... when I don't attend to what I'm doing, I frequently berate or otherwise punish myself for attempts to achieve a target that fall short of that target, and I'm more likely to do that the more I value achieving the target. Which is a great way to extinguish the behaviors I value.
I suspect it's very difficult to design the right reinforcement strategy. It's easy to reward something that seems related to the goal, but can gradually become a replacement for the goal.
For example rewarding success and punishing failure reinforces choosing only trivial tasks, which prevents learning new things. Rewarding starting new things reinforces starting new tasks without finishing them, also choosing tasks for being new, not being useful. Etc.
Rational thinking about consequences, and changing the strategy when necessary, cannot be avoided. So perhaps this should be reinforced. But how do we distinguish between genuine rationality and signalling? Yeah, rationalists should win, but by rewarding success and punishing failure... see the previous paragraph.
Anyway, many people do worse than random, so some reinforcement can be used to improve the situation.
EDIT: Another problem: I suspect that any reinforcement inevitably goes meta. When I get a reward for doing X, I will do X more, but I will also like the reinforcement mechanism more. When I get punished for doing Y, I will do Y less, but I will also hate the reinforcement mechanism and rationalize why I must get rid of it.
I suspect that people prefer wireheading, except in cases when it becomes too obvious that it is wireheading. If I am allowed to choose my reinforcement mechanisms, I will probably unknowingly slowly optimize them towards wireheading. If someone else chooses my reinforcement mechanisms, I suspect they will choose it to optimize their utility function instead of mine.
Yoiks! This may well be why my procrastination at work has increased and increased over the decades. I almost always (habitually?) feel like my efforts are not good enough, will be criticized negatively.
(nods) That's a pretty common result of relying on punishment to shape behavior.
I like this article because it is reasonably short, but very clear and highly actionable.
This compliment is particularly effective because it's specific, verifiable, and true. I've never been very good at accepting vague compliments -- I tend to get embarrassed and self-conscious -- but more specific compliments are really nice.
This explanation of why the complementary comment on the article was effective is itself effective, because it gives specific reasons why the complement is unlikely to evoke the embarrassment sometimes associated with more vague complements.
Thank you Luke for this beautifully written post.
A while ago I saw a kindly waitress give my friend's two year old daughter a small cookie in a restaurant. Various emotions flickered across her tiny face, and then she made a decision, accompanied by a small smile.
She broke the cookie into three pieces and gave them to her brothers. Completely unprompted.
I couldn't believe my eyes. I asked my friend, who is a lecturer in experimental psychology, whether altruism was normal amongst very young siblings.
He looked a bit smug and said "Well we put a lot of reinforcement into that."
I hadn't really thought about what that meant until now. Your clear writing has made it obvious.
As a result of your post, I think I'm going to try deliberately modifying some of my own behaviours this way, and maybe try the techniques on some friends. (The first time, by the way, that I've changed my behaviour as a result of reading less wrong, rather than just treating it as philosophical crack.)
For friends it seems that sincere praise / avoiding criticism would be good, but what would you recommend as rewards to self? I'm pretty sure that nicotine and pizza slices would work for me, but I'm also sure that those aren't things I want to do more of.
Don't underestimate the power of praise as a self-reward. It feels really goofy to explicitly praise myself -- especially to do it out loud -- but that doesn't mean it doesn't work.
IME, the biggest problem with self-reward, whatever the mechanism, is that it requires quite a lot of discipline to differentially reward the thing I want to reinforce at all consistently.
The only time I ever really maintained that discipline for any length of time was when I was recovering from brain damage, when continued focus on self-improvement was the single most important thing in my life for about 18 months. In my real life, I just don't care that much. YMMV.
Recruiting allies to reward me works better for me.
M&Ms, one piece at a time -- they are small enough. (It would probably be good if you stop eating them in all other circumstances, but that is not big sacrifice.)
Or try a symbolic reward. For example put on your table two glass boxes, put 100 stones in first one, and every time you want to reward yourself, move one stone from the first box to the second one, and congratulate yourself on progress. When all stones are in the second box, give yourself a big reward (pizza or whatever), change the boxes, and start again. (This way the reward is still linked to pizza, but it is less pizza. And you see your progress all the time.)
I've noticed in pilates classes with one specific teacher you get positive feedback in one specific situation - when you're having trouble, and have just barely managed something basic. This leads to the association that whenever you get positive comments you know you're doing badly.
Yeah, there's kind of a perceptual/patternmatching arms race going on there -- if you're too blatant about it, or the intended recipient of the reinforcement is just that perceptive, then they're reading the script too and it won't have the intended result. It could backfire (as in your example; semantically-positive reinforcement becomes pragmatically-negative), or send undesirable information ("you wouldn't have put it that way unless something were up, and that gives me a clue"), or open you to counter social-engineering scripts if the part knows what they're doing.
In my case I'm not terribly perceptive, but there's a lot of repetition of the same situation to give you a clue.
If that's the case (and it seems like it is), then reinforcing yourself is going to be almost impossible, because you will by definition know the reinforcement script.
Reinforcing effort only in combination with poor performance wasn't the intent. Pick a better criterion that you can reinforce with honest self-praise. You do need to start off with low enough standards so you can reward improvement from your initial level though.
Eagerly awaiting "The Power of Punishment".
Anecdotally, punishment seems to be a good guilt-releaser, while guilt is dysthymic. Punishment may be effective at snapping someone out of a blue funk and getting them to be responsive to rewards. Guilty people reject rewards. (The above may work better if you are kinked that way.)
I'm curious about the anecdotes. I feel like I'm reading travellers' tales of the weird customs of a distant tribe.
What about self punishment?
Every time you do it, you condition yourself against doing it.
If and only if it's negatively self reinforcing. Which it might not be, if it's serving some purpose.
-- http://www.rcpsych.ac.uk/mentalhealthinfo/problems/depression/self-harm.aspx
Particularly good for demonstrating to observers that you have more status and power than the person you are punishing.
meh. downvoted.
(just joking)
(demonstrating to observers / demonstrating to self / demonstrating to punished; status / power / resources / justification / need / etc; person / cognitive subsystem / institution / problem representation / etc)
The lead article conflates two process: habits and incentives. The very term "reinforcement" dates back to before the distinction was well-understood. Only in the last decade has it been known that habit operates from a neurology distinct from incentives. (The habit mechanism is in a much older part of the brain.) Only the first story, Yudkowsky and the jellybeans, deals clearly with reinforcement of habit. The others are probably primarily adjustment of incentives.
In using habit and incentive, different rules apply. Incentives require that the subject discern the contingency. The processes Skinner studied as "reinforcement" are mostly about incentives. You adjust schedules of reinforcement to alter the organism's expectancies. For incentive effects, consistent reinforcement is not usually best, as the results are subject to extinction soon after the organism stops getting the reward.
Habits, on the other hand, are blind. The organism doesn't need to see any contingency. Yudkowsky continued to be nice even after he no longer received the jellybeans. To form habits, as opposed to incentive structures, consistency is key.
In short, as a general rule, you want consistency to reward habits and considerable randomness to create lasting incentives.
But the difference extends also to the ethical questions raised. Altering others' incentives for our own benefit is part of ordinary human interaction. If his colleagues surreptitiously timed the offer of jellybeans to Yudkowsky when he acted nice, this is something else; the ethical reason is that Yudkowsky need not recognize what he's being rewarded for to be affected by the jellybeans.
Both habit and incentive are "powerful." But they're powerful for different reasons, in different ways; and to apply them effectively and ethically requires different procedures.
How do you tell which things you want to reinforce are habits (and should therefore be reinforced consistently) and which things are incentives?
I'd think a habit is something that just goes on as long as nothing happens to disrupt it. You no longer need to reinforce it.
But the only difference in the process of creating to two is the amount of consistency? That doesn't seem quite right.
I must confess I don't really understand the way 'incentive' is used in [deleted]'s post. Isn't an incentive usually a reward you use to get someone to do something? When I give my cat a treat I to get her cat to come when I call her, the treat is the incentive. I didn't create the incentive - the cat already liked the treats. All I did is get her to associate me calling her with getting a treat.
Can anyone here point me to the relevant scholarly literature discussing the differences between habits and incentives? I tried Google and Google Scholar but failed to find any paper or survey article that explicitly contrasts these two processes.
That's why I tried to stay positive when talking about the new SI website. Especially with technical changes like that, the (vocal) negative response can be overwhelming.
Yup. When reading through the comments about the new website, I could feel my effort being punished.
Perhaps you could have somebody read them for you and summarize them in a non-critical way, thus creating a reinforcement shield.
Alternately, you could adapt what internet marketing "personalities" do, and promote doing: practice celebrating criticism. One marketer (I forget which one) described making a practice of throwing his hands in the air and shouting "Woo!" when he received a criticism via email.
(Background: "personality" marketers promote by writing emotionally charged material that's intended to divide their audience into people who either love or hate them. Thus, the presence of hate mail is evidence that their strategy is working. They will then often publicize the hate mail, in order to stir up the emotions of the people on the opposite side of the debate. Talk radio hosts, bloggers, political commentators, etc. also use these strategies, even if they're not always considered "marketers" in a traditional sense. Whether you consider this "dark arts" is largely a political question, since the LW sequences use these tactics also. Whether he knows it or not, Eliezer is a personality marketer in this sense, it's just that he's not as efficiently monetizing the results. ;-) )
Sorry about that.
It seems to me that if humans were emotionless utility maximizers, we would prefer hearing criticism over praise, the same way programmers purchase more utility by fixing bugs in their programs than polishing features that already work. I suspect criticism is generally more valuable from a pure decision theoretic perspective.
I wonder if there is an effective way to buy encouragement and criticism separately. Also, it's hard to know exactly how best to encourage folks. In theory it's possible that making a new website is not the best use of SI's resources, which suggests reinforcement would not be optimal. But we still may want to reinforce you towards the more general behavior of taking steps to achieve your organizational goals. So what's the best response?
Maybe someone can develop some general guidelines for reinforcing/criticizing people, similar to what the nonviolent communication people came up with. (When {observable event} happened, I felt {feeling} because I need/value {underlying need that felt unmet or value that felt jeopardized}. Would you be willing to {specific request that person could do} in the future?) E.g. check to see if the person was acting with good intentions and reinforce them for those if they existed, check for super goals you endorse and reinforce them for working to accomplish those, check to see if the person could just have easily have sat around doing nothing and reinforce them for expending effort if this was the case, etc.
I think optimally criticism would have lots more reinforcers associated with it: people should be reinforced for requesting, giving, and receiving criticism because these are all activities that are naturally aversive but actually have high expected value.
So, I wholeheartedly endorse the following actions of yours: attempting to maximize humanity's collective utility function, working on the super goal of AGI safety, actually doing stuff, and deliberately gathering critical feedback. Go Luke!
Criticism > Praise > Nothing. The problem is, people default to "Criticize or stay quiet", and so I tend to value praise highly, as it's culturally much rarer.
Also, if it's a matter of opinion (rather than an actual code bug), praise can actively offset criticism (1 person dislikes it, but the other 99 users all love the new UI... probably not wise to revert!)
I am slightly surprised to hear this. I perhaps expected slightly less emotional involvement with the effort and more of a "<Website minion/>, Go! Fix!" feeling.
What happened is that (1) I felt my effort being punished, and then (2) I sent an email to Nickolai or Kamil asking them to fix X.
Great work Nickolai or Kamil, if either of you read lesswrong at all. The website is a much needed improvement! ;)
I've noticed (while being such a minion) that when making such change requests yourself manage to do so with a frame that minimises a criticism vibe or 'effort punishment' feelings. I would pay many, many M&Ms for that effort in careful phrasing.
I tried to do the same. Although, I was probably significantly less successful than I'd liked to have been (sorry Luke, Nickolai, Kamil and anyone else who'd made an effort!).
Also, given lukeprog's comment, this unfortunately appears to be a case of history repeating itself: matt had a similarly negative experience when LW was redesigned a little while ago.
That circumstance is somewhat different in nature. While as far as I know nobody wanted matt to experience negative affect the discouragement of 'effort' was actually a perceived instrumental good, given an expectation that more effort would produce undesired outcomes.
I note that this relies on beliefs at the time. In that context users had to make the prediction "If a website administrator implements detrimental changes when previous discussion had already explained why such a thing was not desired and a prediction had been made that a change to the website would probably be bad, what is the probability that future 'effort' will be beneficial?" The answer is very, very low. The emotional distress matt experience was his social instincts warning him that interfering with the tribe when they do not want you to is a dangerous act - especially when that interference is to (in effect) institute a prohibition against something they could previously do.
It turns out, however, that matt is a superior human being to the typical person in his role. While his ego did cause him to act more defensive than optimal and seemed to cause him to experience emotional distress it did not cripple his ability to respond to user feedback or cause him to lash out with actions against the users as many would. The undesired change was eventually fixed, as were the few bugs that were introduced.
I expect users to drastically update how they would respond to matt if he made future website upgrades due to having more information about matt. He definitely deserves a lot of rewarding for going ahead and doing the bugfixes and implementing 'retraction then deletion' despite having received discouragement. He lives (here) in Melbourne. Perhaps I should give him a packet of M&Ms if I run in to him at one of our meetups!
I read this post last night. I was in the office late, not because I had a great deal to do, but because I was procrastinating. After reading it, I asked my friend to give me a quick call to say congratulations in a half an hour if I'd finished all the work. It took me 10 minutes to finish! :)
But that's probably more of a public commitment effect.
True. But I bet if coffeespoons makes this a routine thing, they'll eventually find themselves enjoying work more.
Nice post SIAI! Have an $5 donation!
I tried a similar reinforcement technique on myself but it didn't stick because I couldn't find a reliable trigger condition for dispensing the reward.
Does this mean that we should stop punishing ourselves for procrastination?
My personal experience strongly suggests that "stop punishing yourself for X" helps avoid X, for most if not all X. For instance, becoming a vegetarian was much easier when I didn't try to go cold turkey, but rather was fine with the fact that I would succumb to the lure of eating meat every now and then. When I did, I felt a little guilty, but then shrugged and thought that I'd try better the next time. I still fall victim to that temptation occasionally, but it's much more rare now than it used to be.
This might have something to do with the fact that if you punish yourself for trying and failing, you stop wanting to try in the first place, as it becomes associated with the negative emotions. Also, accepting and being okay with the occasional failure makes you treat it as a genuine choice where you have agency, not something that you're forced to do against your will.
See also It's okay to be (at least a little) irrational.
Perhaps this is why I like Autofocus better than GTD. "It is fine to have incomplete tasks in your task list".
Also, non-punishment for failures may be one of the distinctions between play-like work and work-like work.
I think I even have work-like play where a game stops being fun. And yes, play-like work is what I want to achieve.
In case of work-like play, I have a resolution: stop playing immediately. It doesn't mean quitting the game for good, but rather "end the session now, if a game permits that". Also, this is why I generally don't play games that punish me for leaving early (e.g. WoW raids, DOTA2).
Anecdote: I've made suggestions to someone on how he might optimize the time he spends writing for his various projects, and more than once he's responded that a given strategy would make it feel too much like work (I don't remember off hand if he said explicitly that this would be an instrumental problem, or if that was only implied). I'm not really sure how I feel on how I might go about applying this concept, mostly because of my extremely vague definitions of work / play, but I do find that having certain restrictions--something as simple as paper size, for example--tends to make it much easier to work on something. (I wrote a shortstory by specifying what it would need to fit in, and measuring books I'd made in the same format years earlier; I made a large number of maps for a game by using a format restricted to 32 tiles across, etc. I haven't found good ways to apply this strategy to most of what I try to do, though.).
If I recall my high school psychology class correctly, you can get a stronger and more persistent effect by secretly rolling a dice and note the number, and when Eliezer says that many nice things, give him an M&M, roll the dice again for a new target number of nice things.
That's true and false. Intermittent reinforcement gets a more robust effect than continual reinforcement, yes, but randomly intermittent reinforcement isn't as effective as setting the reward threshold higher as the behavior becomes more common... e.g., rewarding only the 10% nicest things.
I want to design a reinforcement schedule in one of our apps. Can anyone link me to some specific guidelines on how to optimise this?
(Reinforce exactly what % of successes (30%? 26%? 8%?)? Reinforce performances in the top 10% of past performances (or the top 12%, or the top 8%?)? How does time factor (if the user hasn't used the app for a week, should I push a reinforcer forward?)?)
I can't, but if you find anything concise and useful, I'd love to hear about it myself.
My rule of thumb is to set the threshold so as to reinforce the top 20% or so of performances, and arrange performance frequencies so I'm reinforcing 2-3 times/minute during active training periods. But that's not based on anything.
I'll also note that reinforcing higher-tier performances more strongly works really well (though is hard to do consistently by hand), as do very intermittent "jackpots" (disproportional and unpredictable mega-rewards).
When the threshold is "something nice", there's going to be randomness in the reinforcement anyway.
Some previous discussion about this form of conditioning.
I got a demonstration of how true this is yesterday when, during my taekwondo class, I was paired up with one of the senior black belt students, who has some but not a lot of experience teaching. He was supposed to be fixing up my poomsae (same thing as a kata in karate) and each time he watched me do it, I would finish and he would immediately launch into a description of what I was doing wrong. His feedback was pretty useful–specific, with demonstrations of exactly what to change in order to do it right–but without any prelude of "yay, good job!" or even "okay, the punches were way better that time...now let's work on the stances", I found myself getting really discouraged. Reminding myself that I wasn't actually doing worse than usual, that he just had a different teaching style, helped a little... But my subconscious brain still decided to feel resentful and unenthusiastic, no matter how counterproductive that might be towards my actual goal of improving my poomsae.
As a swimming instructor, I do make sure to dole out a LOT of praise, but I'm wondering if I should push it even further...
I'm not sure a lot of praise is a good idea since that would lower its effectiveness as a reinforcer.
Well, a lot of non-specific praise would water down the value of non-specific praise as a reinforcer, but taking the time to pick out more specific elements that are good/improving would probably reduce discouragement.
I think one of the things I forget most as an instructor is how easy it is to get discouraged, especially when you're being taught by someone who seems to be able to do all of it effortlessly. There's also the element of "I already know I'm doing it wrong! I just can't get my body to listen to my brain!" Instructors who don't acknowledge this and give praise for trying or noticing that I'm doing it wrong are a major source of discouragement for any new physical skill I try to learn.
Any advice on getting one's body or one's student's body to be more cooperative?
Break complex movements down into lots of simple movements ("drills") and practice them individually, a lot...then string together the first two simple movements and practice that sequence a lot...then the first three in sequence...etc. Also, don't start by teaching/trying to learn the full complex movement in the first place–always start with the simplest possible subset, master that, and then worry about the next step.
Very much second this. The most useful thing I've learned about learning, is how to break down a complex action in to multiple simpler ones that I can drill independent of each other.
Excellent point. I stand corrected.
I think you do have a valid point... However, in my experience, most instructors err way on the side of "too little praise" and don't have to worry about using it too much and lowering its effectiveness. And most humans I know have a brain setup where after hearing "good job on X" ten times, hearing it an eleventh time is still really reinforcing. So you'd have to really go to extremes to praise them too much...
I was originally going to say I don't like excessive praise, but thinking about it, what I actually dislike are two things:
1) False praise. I really hate it when it's obvious someone is formatting EVERYTHING they say to match a script (the one that annoys me most is the "sandwich" model of praise-critique-praise. It's great for blows that need to be softened, but if you soften everything, then every mistake becomes equally trivialized).
2) Wasting my time. If the feedback boils down to "You did exactly as well as you did last time" then (a) I probably know this and (b) you can say approximately that without spending 2 minutes extolling my virtues. I'm usually impatient to get back to actually doing the activity. If I'm not impatient, it means I'm either seriously discouraged or don't value the activity at all - either way, unless it's my actual job, I'm unlikely to care about feedback at that point.
These apply to critiques even more-so than praise, but the style of "make everything 90% praise and act like mistakes are just this mild little thing" is a pattern I recognize very quickly, and find extremely discouraging, since it means I'm no longer receiving feedback that actually honestly represents how well I'm doing.
I also hate being given "points for effort" unless the person is correctly reinforcing "You'll probably need to repeat this drill 500 times before you have it down correctly, so be patient with yourself" (saying this when others are figuring it out in 5-10 drills is clearly lying, and will again seriously impact all feedback from that source >.>)
Well, it sounds like you're someone who, if you already know how you did on something, don't need people to shower on praise unless it conveys new information. Am I right that you would find specific praise, or praise on something that you genuinely didn't know whether you'd done well on, less annoying?
As someone who regularly takes more repetitions of a drill to learn a certain (physical) skill than the average person, I damn well like getting points for effort–they're likely to be the only points I get for a while, and I tend to get seriously discouraged watching other people learn stuff easily when I'm struggling with it. I agree that someone who says "be patient, everyone needs to do this 500 times to get it right", when that's not the case, is not being helpful...but a simple "good effort, you will improve on this, don't worry" is a) not lying, and b) helps with the discouragement factor.
This is indeed key, thank you for putting it more concisely than I
It varies. "Be specific" is usually better, but "be brief" is also often important to me. A slow break-down of specifics is important if I don't know how to improve. A brief summary is fine if I'm improving on my own and really just need to get more repetitions. These days I'm usually aware of which one I need, and can ask for it. Previously, I'd just get frustrated if I needed more specific advice, because communication is exhausting, and learning is exhausting, and the combination of the two sucked.
I've found that it varies - if I have things down and just need to drill, I'll often be entirely content off in a corner repeating something mindlessly with minimal feedback (an occasionally "good effort, you will improve on this, don't worry" is very rewarding, but we're talking every 15-30 minutes)
Basically, I can enjoy drilling. I actually find it a ton of fun with most skills I've gotten good at - the skills I fail to improve are usually the ones where I don't enjoy drilling, and thus... don't drill. The assumption that just because I'm stuck repeating something a lot, I must need encouragement... tends to de-motivate me, because it says "hey, you're slow and abnormal and so I'm going to focus a lot on fixing you", which has a lot of bad connotations for me.
I strongly suspect your teaching style would not annoy me (or that you'd quickly adapt it around me), but a lot of people get stuck in the meme of "ALWAYS follow this script" and start completely ignoring body language cues that indicate a particular student DOESN'T do well with a certain script.
As an example: I hate, HATE the "positive - negative - positive" sandwich. It means I associate positive feedback with a lead-in to "something bad", and so any compliment is now a threat to me, a sign I did something else wrong. I also pattern-match fairly well, and once I figure out when it's a sandwich vs a genuine compliment, I'll get impatient to know why I was REALLY brought in to talk (i.e. what I did wrong).
This sounds like a challenging situation. How were you able to move past this in order to be able to ask for more specific feedback when you needed it?
You are very lucky to be content in this kind of situation. I wish I could be more content.
I think I almost have a good connotation around this kind of situation. There are at least two areas (singing as a strong example, and competitive swimming as a weaker example) where I started out pretty awful. I could have compared myself to the people starting out at the same skill level as me...but that would have been pretty pointless. Other people who were as tone deaf as I was at age 11 just didn't try learning to sing. So I made my reference group the people who were doing solos in my choir. After a few years, I think most people actually forgot that they had originally considered me "slow and abnormal." I started to get the comment "well, obviously someone with your natural musical talent..." Ha. Right. But I did succeed in proving, to myself if not anyone else, that if I put myself into situations where I am "slow and abnormal" compared to everyone else, I will make much bigger improvements than if I stick with the activities where I'm already stronger than average.
It's not really exciting to say it, but: 1) I learned to identify, internally, what my emotions correspond to (most critically, if I'm frustrated, it's probably because I'm practicing the wrong thing)
2) I've memorized a few phrases that tend to garner the feedback I need ("Can you be more specific?", "Can you break that down in to smaller pieces?", "I feel like there's some little piece I'm missing that would make this all click together", and "can you demonstrate slowly and narrate what you're doing?")
3) Most important, I have a strong CONCEPT of "this technique is actually a series of smaller techniques that I can drill separately". It's very hard to ask someone to break something down in to simpler steps when you're stuck thinking about it as a single step. And I've broken things down often enough that I can communicate the idea to an instructor who doesn't have it as a concept.
3rd one also helps me evaluate things in advance: "this skill is beyond me - I will need to do something smaller and simpler first, otherwise I'll feel totally overwhelmed and have trouble learning." The tricky bit is usually just finding smaller pieces, but that's where an instructor is useful :)
Hmm, I wonder if providing a lot of negative reinforcement on some attribute of them you don't care about would make the positive reinforcements more effective on the things you do care about.
Example: trying to teach someone math, and praising them at everything they do right with the math, including trying, but complain abut their physique, fashion choices, hygiene, etc. Especially timing those unrelated complaints to when they seem less focused on the math but subtly enough they don't consciously notice the correlation.
Not that this isn't a bad idea for other unrelated reasons...
There's a couple of factors here worth keeping in mind.
One is that classical conditioning continues to work, even when I'm concentrating on operant conditioning. So one result of this strategy is that my target will come to associate me with aversive stimuli, which will in turn reduce the effectiveness of my attempts at reinforcement. They will similarly associate the teaching sessions and math with those stimuli, which may be counterproductive.
Another is that a target consciously noticing my attempts at conditioning changes the whole ball game, in ways I don't entirely understand and I'm not sure are entirely understood. Sometimes it's a huge win. Sometimes it's a huge lose. Staying subtle is more predictable, if I can do it, but of course it's not always possible to avoid detection, and sometimes it's better to admit to my attempts at conditioning than to be caught out at them. The safest move is to first establish a social context where my attempts at conditioning can be labelled "manners," such that any attempt to call me out on them is inherently low-status, but that's not always possible either.
When using praise signals as reinforcers for systems, like some humans, who are capable of skepticism about my motives, it helps to be seen to use expensive signals. (Attention often works well, which is one reason Internet trolls are so persistent.) Of course, that typically means I have to invest resources into my conditioning efforts.
In general, the approach I endorse is to maintain (and adjust as needed) a consistent threshold of evaluation, ignore behavior that falls below that threshold, reward behavior that clears it, and resist the temptation to go meta about the process.
Sounds like an interesting idea for an experiment, although it would probably violate ethical guidelines. :P
The example you give is either punishment of the other attributes or negative reinforcement of the desired behavior (if you look at it from the perspective of taking away the aversive stimulus only when the math is done.)
Thanks for pointing out this particular low-hanging fruit.
I wonder if they had just (re-)watched this Big Bang Theory episode.
Hmm, I better keep this in mind at all times when dealing with my family.
Daniel Kahneman in Thinking, Fast and Slow:
There reason for that lies in regression to the mean when training (example of flight instructors in the israel airforce):
Since positive reinforcement is so counterintuitive: don't forget to reward yourself for rewarding somebody for good behaviour! :)
Speaking of regression to the mean, that seems to be one topic that wasn't really covered in the sequences that really should have been.
My wife, if pulling that kind of stunt, would quickly find that her affections were shunned and her thanks were met with clear contempt (after she was asked politely not to do that the first time). It is almost certainly not in her interests to produce a pavlovian association between her affections and attempts to control me against my wishes. My aversion to hostile takeover of internal motivations is much stronger than my desire for the affections of any particular individual.
This would be entirely different if I had made a prior agreement regarding shirts and hampers. Making it motivationally easier and more enjoyable to do things I am willing to do is to be encouraged.
Seriously? You'd shun your wife because she said thank you? i.e.
Some people react quite viscerally to the awareness that another party is trying intentionally to steer their behavior in any way. It seems to just be a massive squick button for some (indeed, I notice that most randomly-selected people who are made aware of explicit attempts to condition behavior react with discomfort at minimum); for others, there seems to be a correlation with triggers gained from abusive interactions earlier in life; a few I knew who reacted strongly showed strong indications of sociopathy and seemed to instinctively feel violated if someone else successfully, or even just obviously, tried to affect their behavior in a deliberate manner toward some end (a normal part of cognition and social interaction for them directed at others).
(No, I said I would shun kisses delivered under those circumstances. No cutting and pasting of my keywords for the sake of hyperbole thanks.)
If people use their affection in a way that is obviously intended to systematically manipulate me to do things that I do not, in fact, wish to do then yes, of course those instances of affection I will shun. While I know some people are more tolerant to that kind of blatant disrespect I would expect you to at least be able to comprehend the subset of people that will not.
I'm afraid that all women who want kisses to serve the role of doggy treats within our relationship are out of luck. I have yet to experience a problem with having that policy. My model of myself predicts that rewarding hostile-to-my-interests-reward-training with increased compliance or acceptance would leave me with relationships that were far less satisfying and in particular far less enjoyment of displays of affection.
So, I have to ask: do you in fact have a wife?
The phrases "of course" and "blatant disrespect" imply a shared frame of reference that doesn't seem to be in evidence. While it might be considered rude to you, it's pretty much human nature. The phrase "thank you" is, as near as I can tell, pretty much entirely meant as a positive reinforcer.
So, having established that we have different frames of reference, can you go in to WHAT behaviors bother you? Is it the use of specific actions as reinforcers ("thank you" is okay but kissing is not?) or is it just the deliberate (as opposed to socialized and subconscious) application of these techniques? Or something else that I'm missing?
Since positive reinforcement can only be applied after you already do a thing, then presumably, you at least wished to do it once. So, how is providing you with a bonus to something you've already done, manipulating you to do something you don't "wish to do"?
Caveat: I don't know why the husband in question doesn't just put his damn clothes in the hamper. Doesn't the idea of having soiled clothes lying around repulse him anyway? Especially when sharing the space with another. I mean... ewww. But now back to assuming the target behavioral territory is not already granted by the obvious shelling point or prior arrangement.
It seems you wish to unilaterally accept rewarding behavior as positive. I don't. I have no trouble detecting when rewards are being used as "approximations" towards a behavioral landscape that I clearly don't want or, especially, have previously declared that I would not accept. I am also able to predict - by reference to past experience and knowledge of my own preferences - that encouraging that reward pattern gives undesired outcomes. As Vaniver mentioned, an important skill to develop is the ability to detect the difference between desired and undesired manipulations.
As a somewhat separate issue, excessive use of physical affection (kisses, hugs, sex) as a "reward" for good behavior changes the experience of those activities - and not in a good way.
Could you elaborate on that? I'm entirely okay with physical affection being used as a "reward", as long as it's also clear that the person genuinely wants affection with me, and initiates it "just because" too (actually I'd probably be entirely okay with a strictly reward-based system of affection, as long as it was explicit...)
You seem to be assuming, in the example, that the husband doesn't WANT to be modified to put away his laundry. Is that correct?
If so, is it correct that your objection is "you're manipulating me in to a state I don't desire" rather than simply "you're manipulating me"? Given that you PERSONALLY find soiled clothes disgusting, would you PERSONALLY appreciate reinforcement that helped you overcome such a habit?
Yes.
Yes.
Hm. You quoted a question I asked, and then proceeded to not answer it in any way. The question was:
Instead of answering that question, you supplied various generalizations whose referents in physical reality I can't ascertain. Please give an example of a situation where somebody being, say, happy that you did something, means that they are manipulating you to do something you don't "wish to do" (your previous words).
Well, I'm not wedrifid, but OK.
Suppose there's a crisis at work, and in response to that crisis I step in and solve a problem.
Suppose, as part of solving that problem, I take some steps (X) that I don't enjoy doing and don't wish to do again.
Suppose my boss notices that I did X and was effective at it and decides that she wants me to do X more regularly, and being familiar with the uses of positive reinforcement decides to hand me a large bonus at our next status meeting. Further, she praises me to the skies in public for having done X, and does so in a way that communicates the (entirely accurate) message that my continuing to receive such praise is contingent on my continuing to do X.
I assert that, in this scenario, my boss is applying positive reinforcement techniques with the goal of increasing my likelihood of doing X, by providing me with a bonus to something I've already done, where X is something I don't wish to do.
Do you agree?
As to whether, in so doing, she's manipulating me... (shrug) I've already had that discussion once too often this week. If our only remaining point of disagreement about that scenario is whether the word "manipulating" properly applies to it, I'm happy to leave that point unresolved.
The question is not whether positive reinforcement is effective in changing your behavior. The question is whether kisses are positive reinforcement in particular contexts.
Suppose your spouse says, "Please pick up my prescription from the store" and you don't want to, but you do it anyway. When you get back, spouse says "Thanks for dealing with that." Do you really think continued experiences like that won't increase the frequency of the behavior "Run an errand even when I don't want to"?
I think it depends a lot on her intention. If she says 'thank you' for the purposes of positive reinforcement, I mean if she thinks about her 'thank you's' that way, then I think she's being manipulative.
If she says 'thank you' to say what those words mean, namely, that she's grateful, then even if this does have the effective positive reinforcement there's nothing wrong about her behavior.
I find the idea of endorsing manipulative behavior if and only if I remain unaware of the fact that it's manipulative behavior deeply troubling.
It strikes me as similar to saying that hurting people is OK as long as I don't know I'm hurting them. No, it isn't. If hurting people is not OK, then it follows that I ought not hurt people, and learning to recognize when I'm hurting people is part of that, and I ought to learn to recognize it. The behavior doesn't suddenly become "not OK" the moment I learn to recognize it... it never was OK, and now I know it and can improve.
Conversely, if hurting people is OK, then it's OK whether I know I'm doing it or not.
The same goes for manipulating people. Whether I know I'm doing it or not isn't the determiner of whether I'm doing good or ill.
To my mind, the determiner of whether I'm doing good or ill is whether, when I'm done doing it, we're all better off or worse off.
I agree with your point, but I think that "manipulate" needs to be tabooed. If we define manipulate as "acts that tend to change the behavior of others" then I agree with your implicit point that it is impossible to interact with others without changing their behaviors, and there is nothing wrong with thinking about how I would like someone else to behave when considering how I interact with them.
That said, there are connotations of manipulate as the word is ordinarily used that are not captured by the way you (and I) are using the word.
Sure. I'm perfectly happy to drop the word altogether and instead talk about changing the behavior of others.
Awareness of side effects isn't equivalent to intentionality. You can thank someone to express genuine feelings of gratitude. If you wouldn't do that in a counterfactual world in which the gratitude was absent, then I wouldn't call that behavior intentionally manipulative regardless of whether you know about positive reinforcement.
Suppose I am not in the habit of expressing gratitude when people do nice things for me. Never mind why... maybe I was raised wrong. For whatever reason, I'm not in that habit. I feel gratitude, certainly, I just don't express it.
Then one Monday, I learn that expressing gratitude to people for doing nice things for me will increase the odds that they will do it again. Suppose I want people to do nice things for me, and I therefore conclude that I ought to expressing gratitude when people do nice things for me, in order to get them to do it more, and I therefore start expressing gratitude when people do nice things for me, whether I feel gratitude or not.
Then on Wednesday, I learn that this only works when I genuinely do feel gratitude... when I express gratitude I don't actually feel, I get bad results. (Again, it doesn't matter why. Maybe I'm a lousy liar.) So I stop expressing gratitude when people do nice things for me when I don't feel gratitude, but I continue doing so when I do, since that still gets me stuff I want.
If I've understood you correctly, you would call me intentionally manipulative on Tuesday, but not on Thursday. I'm happy to restrict the term "intentionally manipulative" to Tuesday behavior and not Thursday behavior, if that makes communication easier, though I don't use those words that way myself.
Regardless of what words we use, presumably we agree that on both Tuesday and Thursday, I am doing something with the intention of causing changes in other people's behavior, and am doing so without their awareness or consent. Yes?
Do you endorse this on Tuesday?
Do you endorse this on Thursday?
For my own part, I find the idea of endorsing that behavior on Thursday but not on Tuesday deeply troubling, for many of the reasons I listed before.
Obvious remark is obvious: you might disapprove of the behavior on Tuesday because it involves lying.
If you don't know you're manipulating someone, you're not manipulating someone. Manipulation is an intentional behavior, like lying, or congratulating, or taking a vow. Knowing what you're doing is part of doing it.
Yeah, I pretty much disagree with this statement completely.
That's... incredible to me. Do you disagree that there is such a category (i.e. actions you have to know you're doing in order to be doing them at all), or that manipulation falls under it?
I disagree that manipulation falls under it.
This exchange may be helpful to understand TheOtherDave's point.
I do accept this kind of reinforcement from my significant other, assuming that:
Actually I consider it very useful, and for me it would be a waste not to use this kind of cheap "external willpower". YMMV.
Note that I consider the reinforcement you are describing to be entirely different in kind (not "this kind"). The boundaries around the kind I accept are approximately the same as yours:
I go by what my intuition tells me but when formalizing those intuitions something similar is generated.
I make a point of rewarding desired reinforcement (while attempting 'extinction' on less desirable influence tactics like nagging or punishment.)
I supposed the reason why the husband in the story didn't put his clothes in the hamper was that he was too lazy to do that, not that he (terminally) valued that the clothes stayed outside the hamper.
Having a terminal value for clothes outside the hamper isn't the point. It is whether given the negotiated relationship boundaries and typical behaviors as they currently are the person being modified would prefer "status quo except I do <influenced behavior> more" over "status quo".
"Too lazy" can be left out of such considerations. That doesn't distinguish between akrasia and considered intent not to do the thing (for whatever reason). For most part judgements like "too lazy" are just another method of attempting influence - usually a method that is inferior to reinforcement.
Well, making judgments like "too lazy" can also provide valuable social cover for other kinds of reinforcement (or punishment), within communities where deliberately altering the behavior of others is seen as unacceptable unless I can frame it as being for their benefit.
More generally, motivated speculation about other people's best interests (including but not limited to positing that they possess unexpressed "terminal values" that happen to align better with what I seem to want than with what they seem to want) can be a very useful way to ignore people's stated preferences without feeling (or being seen by third parties as) indebted to them.
What would you see as the difference between a) the story described, and b) a wife who kisses her husband because it makes her happy when he does helpful, nice things, of which putting laundry in the hamper is one, and her automatic response to this surge happiness is "thank you, you're an amazing man!" [kiss]? The latter includes most of the same actions on the part of the wife, and probably occurs in a lot of healthy relationships.
Are there some internal motivations that you are less protective of than others? For example, if someone tried to condition me to be less averse to harming people, I would have a pretty big reaction, because that particular internal motivation is sacrosanct to me. But preferences for levels of tidiness...meh. I barely consider that an internal motivation, and definitely not a facet of who I am...it's just a habit, and I don't really care about changing it in either direction.
Is the difference with you that you consider all of your motivations to be a sacrosanct part of who you are? Or just that you place a higher value on your autonomy, and being the one 100% entirely responsible for all of your decisions?
It may be worth sharing, anecdotally, that years ago my husband expressed annoyance with me over the fact that I only ever rubbed his back while he was doing dishes, and it made him feel much like how wedrifid describes.
This utterly bewildered me, so I agreed to pay attention to the behavior and see what was going on. Pretty quickly it became clear to me that this was absolutely true, for reasons I wasn't entirely clear on myself, although my working theory was it was the only time that I'd regularly walk past him while he was hunched over in that particular posture, which apparently served as a "give me a backrub" signal for me, for whatever reason.
My response to this was to start giving him random backrubs at other times, which solved the problem.
My point being that (a) being annoyed by this sort of behavior is not at all unique to wedrifid, and (b) whether the behavior pattern is intentional doesn't necessarily matter very much. (I don't mean to suggest that it doesn't matter to wedrifid; actually, they have made it somewhat clear that it's part of what they're objecting to.)
The main lesson I'm taking from your anecdote is "people are complicated, everyone is complicated in a different way, and for almost any action or behaviour X, there will be a person somewhere who finds it awful." It's hard to guess at the relative numbers without doing a poll, but I'm guessing there's a range of people who wouldn't care if their significant other used physical affection as a reward (or who would even like it, because "yay, more total physical affection!"), and there's a range of people who would find it mildly to extremely unpleasant.
Yup, that's consistent with my experience.
Well, the whole thing where he is standing up against the sink with his back to you but his hands were busy and he couldn't turn around (to engage in other forms of affection) seems like the obvious guess.
"Don't Shoot the Dog" remains my favorite book for these sorts of anecdotes, as well as some of the theory and a lot of the practice. I recommend it.
So, reinforcement with M&Ms doesn't translate into an addiction for extrinsic rewards and the reduction of intrinsic motivation?
I'm missing something here, I know.
One could attempt to fight that by reducing the number or frequency of M&Ms eaten over a long period of time, essentially weaning one's self off of extrinsic rewards.
I think it's still hard to privilege if that kind of effect exists in the first place.
I am probably unusual in this regard, but I think I would find both approaches equally aggravating. If someone points out that I've made a mistake, anything other than a concise detailing of exactly how what I did differs from what I was supposed to do, is just going to irritate me. Also, my brain tends to interpret being ignored as a signal that I'm doing correctly.
Is this because of the "damn it, I know I made a mistake, you telling me I did doesn't help!" effect? I get that too... A good thought experiment is that if I was making a type of mistake that I couldn't automatically tell I was making on my own, I would prefer it to be pointed out, even if not in a concise detailed fashion–the idea of not knowing that I'm making a mistake is kind of scary. What would your reaction be in that situation?
No, I react the same way whether I was previously aware of my mistake or not. I only experience that effect when I'm told to do something I am already doing.
Pragmatically, we as humans, just barely over the threshold into sapient intelligence, make mistakes we're not aware of constantly. If we didn't, we wouldn't need a superintelligence to fix the world; we'd have already done it ourselves. So finding the concept scary seems kind of pointless.(Sort of like being hydrophobic about the water in one's own body.) However, I would, of course, rather be aware of my mistakes than not.
But none of this is really on the topic, which was that the listed reinforcements don't seem even remotely applicable to humans in a universal way.
My actions have impacts on others. In general, I prefer to help other people or at least not harm them–however, I may harm someone by mistake, and I really don't want this to happen. If I make a mistake once and I realize it–fine, hopefully no harm done, I won't do it again. If I make a mistake and I don't know about it, well, maybe no harm done that time in particular, but I'm likely to keep making this mistake over and over, and possibly the first time I'll find out is when there is harm done. I think that justifies finding it scary.
Lessons learned:
continue to mentally /ignore people and posts I don't care for on IRC and online forums
never comment on bad posts or explain my downvote on LW
be more generous with upvoting good contributions and give a short praise when warranted.
This is not quite justified; this is a post on how to use positive reinforcement, not how to use punishment.
(from the link)
Dolphins are more difficult to punish usefully than humans; for one, they're less likely to understand English.
Moving to object-level advice: I agree that not responding to bad comments or posts is generally a good idea. I think that responding to downvote explanation requests is a good idea about half of the time. Unsolicited downvote explanation is typically done to sway bystander opinion as well as inform the poster, and so deserves its own treatment.
The difference between explaining bad posts and punishing misbehaving dolphins is that the explaining is done for the purpose of the other readers, not just as a punishment.
I think this should be "never downvote".
Seems to me that a downvote would associate negative valance with both the act of posting on LW and with whatever their specific mistake is, with the latter being stronger. So no vote and a comment with a mixture of praise and criticism is probably the stronger play if you're looking to improve someone's writing or fix some technical mistake while keeping them as a contributor, but a downvote is still effective if all you care about is seeing fewer posts of that kind.
This seems to contradict the very powerful effect of learning from failure and corrective feedback. See http://www.wired.com/wiredscience/2011/10/why-do-some-people-learn-faster-2/ for an accessible overview.
I'd conjecture this works better when someone can already perform the desired behavior and wants to form a habit, whereas learning from failure comes in when new information needs to be stored and reorganized.
That article especially seems to demonstrate the critical importance of choosing what you reinforce, and how your a teacher's model of what they are reinforcing may differ from the students.
I was about to reply "hmm, I wonder how you could reward someone for making an effort rather than just for succeeding, or reward them for noticing when they make a mistake." Then I read the article, and realized that that's basically what it talks about.
Yeah, failures are important. But the natural tendency, whether teaching others or trying to change our own behaviour, is to correct and criticize failures–which is basically negative reinforcement and trains people to stop trying because failing is so painful. The interesting new point in the article is that positively reinforcing for success, if done in a certain way (the "wow you're smart!" group of kids) can actually have the same effect as negatively reinforcing for failure.
More generally, a positive motivation often contains an implicit negative motivation -- a threat of not receiving the same reward next time. ("What pushes you forward, holds you back.")
Telling someone they are smart implies that the teacher has ability to judge smart and stupid students based on their work. So if tomorrow the work is not good enough, the same student could be judged as stupid. This could also happen if the student tries something new, where they obviously cannot have as good results as when they stick with what they already know well.
Telling someone they work hard avoids this danger somehow. Maybe because it contains an actionable advice what to next time to achieve the reward -- so it feels more under control, less threatening.
Maybe the secret is in finding a motivation that feels under control, but not too much to allow cheating. Maybe it's a moving target; I suspect that given enough time, some children in the experiment would find ways to appear working hard without doing the hard work.
Yeah, working hard is something that isn't associated with a fixed mindset in the same way that intelligence is. A lot of people see intelligence as something that you either have or you don't.
We have enough happy death spirals here.
Who is happy about what?
Excellent article. I wonder if reinforcement could be used to speed up rationality training? I would love to see a study done on that.
Almost certainly. CFAR is doing this to some extent at their minicamps, but much more can be done. There is tons of room for rationality training via videogames, for example. Raytheon is developing some debiasing video games for military officers.
Now that is exciting as hell. I've always felt that rationality was a potential competitive advantage for organizations that wasn't being utilized. Way to go.
I just read Don't Shoot The Dog, and one of the interesting bits was that it seemed like getting trained the way it described was fun for the animals, like a good game. Also as the skill was learnt the task difficulty level was raised so it wasn't too easy. And the rewards seemed somewhat symbolic - a clicker, and being fed with food that wasn't officially restricted outside the training sessions.
Thinking about applying it to myself, having the reward not be too important outside the game/practise means I'm not likely to want to bypass the game to get the reward directly. Having the system be fun means it's improving my quality of life in that way in addition to any behaviour change.
I haven't done much about ramping up the challenge. How does one make doing the dishes more challenging?
But I did make sure to make the rewards quicker/more frequent by rewarding subtasks.
Wow, thanks for this great article that was the final piece of information that tipped me over towards getting my shit together. Within 10 minutes after reading it and browsing the comments, I was on my bicycle going to buy small treats I like, that I now give myself for every achieved small goal (~2-10 min of work).
I now wonder though if maybe I should give myself another reinforcer when starting to work with a new goal, otherwise maybe I will only strive for finishing as fast as possible, but starting with a new small goal won't be that much reinforced? Maybe this is my mind trying to get more candy though, so I would be thankful for outside perspective.
Have you been trying this? Any luck?
It worked with similar effectiveness as other techniques I implement - that means only until I have done enough to feel good about myself (2-5 productive days)...
What expert timing, Luke! Just two days ago, I came across the fascinating practice of clicker training for horses - http://www.theclickercenter.com, while reading Kathy Sierra's old blog - http://headrush.typepad.com/creating_passionate_users/2006/03/clicker_trained.html.
My only problem is that I need to train my own behaviour rather than someone else's. I'm going to try to use these techniques on myself, although I'm not sure if that's supposed to work.
Note that there are many circumstances when it is right to criticise. For instance group brainstorming exercises are more productive if the participants criticise each others ideas.
Does this still work if I reinforce myself? Every time I read 5 lesswrong articles in a day, I give myself a reward. Or every time i have a cigarette, I kick a brick wall with no shoes on. If i was consistent with this for a long time, would it work?
Totally. The wall will fall over in 20 years, tops!
The actual answer is maybe - it works for some but not others. A common point of failure is that people just train themselves to cheat and take the reward anyway. I'm not sure what the response rate is when full compliance to the reward schedule is assumed.
It can. Basically the failure modes are the same as when reinforcing others. In particular, it's common to fail to maintain consistent thresholds of self-reward.
Made me smile. Thanks for sharing.
Hopefully now that the experiment is over, they will return to the original schedule of giving M&Ms for new HPMoR chapters. Seriously, people are suffering here. :D
Too infrequent. They need to start by giving him an M&M every time he thinks about writing more HPMoR.
But then when he starts actually writing, Eliezer will become diabetic!
Shush, don't give away the plan!
But seriously, one can always increase the reward threshold once the first behavior has been firmly established.
If he gets into flow quickly he could be safe. That would mean he is writing more HPMoR but not thinking about writing HPMoR.
Thanks, Luke! I've always enjoyed this sequence. (It's funny that I was tempted to include a note that I would've been happier if you contributed to the sequence more often, but let's stick with the praise for now. :-)
Lots of cool videos of operant conditioning and animals are here (also click on the other chapters).
related: http://gettingstronger.org/2012/01/hormesis-and-the-limbic-brain/
"Reprogramming the amygdala. This is the indirect way to re-program the hypothalamus, by altering the amygdaloid reward circuitry that feeds it. There are a number approaches to achieving this, some of which I’ve outlined in previous articles, but all of them fall generally under the umbrella of classical or Pavlovian conditioning. There are a few basic strategies:
"
Any non-sugary food related reinforcer?
Coffee/tea. Twice in the past week, during extremely busy shifts at work, one of the doctors has decided to buy Tim Hortons for all the nurses. I can't think of any other single event that has made me as happy (albeit for a short 5-minute time period.)
Perhaps it's just the unexpected generosity.
I remember a few years ago I was reading about positive psychology's various findings, such as that spending money on experiences with friends or family made one happier than spending money on an object for oneself; so I tried out buying donuts or bagels for a few university clubs. Everyone seemed much happier, for what was a relatively trivial amount of money invested (<$10 in every case).
(I don't remember noticing that the donut people were happier than the bagel people.)
Cheese would work well as a reinforcement for me. Also the proverbial carrot. And probably lots of other things.
Dark chocolate?
This post may have the highest upvotes per comment I've ever seen. Anyone got access to the database want to confirm that?