LESSWRONG
LW

Comment Permalink

When it is possible to change the situation, emotion directed the right way acts as reinforcement signal, and helps to learn the correct behavior (and generally to focus on figuring out a way of improving the situation). Attaching the right amount of right emotions to the right situations is an indispensable tool, good for efficiency and comfort.

The piece you may have missed is that even if the situation can be changed, it is still sufficient to use a positive reinforcement to motivate action, and in human beings, it is generally most useful to use positive reinforcement to motivate positive action.

This is because, on the human platform at least, positive reinforcement leads to exploratory, creative, and risk-taking behaviors, whereas negative reinforcement leads to defensive, risk-avoidance, and passive behaviors. So if the best way to change a situation is to avoid it, then by all means, use negative reinforcement.

However, if the best way to change the situation is to engage with it, then negative emotions and "shoulds" are your enemy, not your friend, as they will cause your mind and body to suggest less-useful behaviors (and signals to others).

Vladimir_Nesov15y00

IAWYC, modulo the use of "should": at least with connotations assumed on Less Wrong, it isn't associated with compulsion or emotional load, it merely denotes preference. "Ought" would be closer.

See in context

16 A social norm against unjustified opinions?

by Kaj_Sotala

29th May 2009

1 min read

161

16

A currently existing social norm basically says that everyone has the right to an opinion on anything, no matter how little they happen to know about the subject.

But what if we had a social norm saying that by default, people do not have the right to an opinion on anything? To earn such a right, they ought to have familiarized themselves on the topic. The familiarization wouldn't necessarily have to be anything very deep, but on the topic of e.g. controversial political issues, they'd have to have read at least a few books' worth of material discussing the question (preferrably material from both sides of the political fence). In scientific questions where one needed more advanced knowledge, you ought to at least have studied the field somewhat. Extensive personal experience on a subject would also be a way to become qualified, even if you hadn't studied the issue academically.

The purpose of this would be to enforce epistemic hygiene. Conversations on things such as public policy are frequently overwhelmed by loud declarations of opinion from people who, quite honestly, don't know anything on the subject they have a strong opinion on. If we had in place a social norm demanding an adequate amount of background knowledge on the topic before anyone voiced an opinion they expected to be taken seriously, the signal/noise ratio might be somewhat improved. This kind of a social norm does seem to already be somewhat in place in many scientific communities, but it'd do good to spread it to the general public.

At the same time, there are several caveats. As I am myself a strong advocate on freedom of speech, I find it important to note that this must remain a *social* norm, not a government-advocated one or anything that is in any way codified into law. Also, the standards must not be set *too* high - even amateurs should be able to engage in the conversation, provided that they know at least the basics. Likewise, one must be careful that the principle isn't abused, with "you don't have a right to have an opinion on this" being a generic argument used to dismiss any opposing claims.

DisagreementPublic DiscourseSocial & Cultural DynamicsStrong Opinions Weakly Held

Personal Blog

16

Mentioned in

15Kaj Sotala's Posts

A social norm against unjustified opinions?

New Comment

161 comments, sorted by

top scoring

Click to highlight new comments since: Today at 2:41 AM

Some comments are truncated due to high volume. (⌘F to expand all)Change truncation settings

[-]JamesCole15y200

A currently existing social norm basically says that everyone has the right to an opinion on anything, no matter how little they happen to know about the subject.

...what if we had a social norm saying that by default, people do not have the right to an opinion on anything?

I get what you're saying, but I don't think that's quite the problem.

The real problem is the social norm that says "you aren't allowed to be critical of someone else's view because everyone has the right to an opinion".

The italicized bit is the problem. I think everyone should have the right to an opinion, but also that everyone should have the right to be able to express criticisms of other's opinions.

(I think the "you can't criticise other's views" thing stems from relativism).

5JamesCole15y

put another way, I think the problem is a norm that says "the right to have an opinion means the right to not have it criticised"

5Eliezer Yudkowsky15y

I think this is a good distinction, and anyone somehow trying to shift social norms (perhaps within a subcommunity) might be well-advised to shift the norms in order: First, teach people that others have a right to criticize their opinion; then, teach them that they have no right to an opinion.

3alvarojabril15y

"teach them that they have no right to an opinion." I know people throw the term around (I try not to), but this is maybe the most fascist thing I've seen on this board. They have no right to an opinion? You might want to rephrase this, as many of my opinions are somewhat involuntary.

2Eliezer Yudkowsky15y

http://www.overcomingbias.com/2006/12/you_are_never_e.html

6Alicorn15y

It seems that in this article, Robin is co-defining "opinion" with "belief". This isn't, exactly, incorrect, but I don't think it maps completely onto the common use, which may be causing misunderstanding. If I say "it's my opinion that [insert factual proposition here]", then Robin's remarks certainly apply. But if it's my opinion that chocolate chip cookie dough ice cream is delicious - which is certainly a way people often use the word "opinion" - then in what way might I not be entitled to that? Unless I turn out to be mistaken in my use of the term "chocolate chip cookie dough ice cream", or something, but assume I'm not.

1JGWeissman15y

Robin was clear about what he meant by "opinion". From his first paragraph, with emphasis added: Though I agree that it can cause problems to use "opinion" in an unusual way, even in the context of explicitly stating one's unusual definition, when people are going to quote the conclusion as a slogan out of the clarifying context. On the other hand, "You are entitled to your utility function but not your epistemology" would not make an effective slogan. (Well maybe, if it has enough "secret knowledge" appeal to motivate people to figure out what it means.)

1alvarojabril15y

Thank you. An opinion is a thought. What does it mean to say that you are not entitled to a thought?

6AdeleneDawner15y

In this case, it means that you're not entitled to refuse to change a belief that's been proven wrong. If you think "everyone likes chocolate ice cream", and I introduce you to my hypothetical friend Bill who doesn't like chocolate ice cream, you're not entitled to still believe that 'everyone' likes chocolate ice cream. You could still believe that 'most people' like chocolate ice cream, but if I was able to come up with a competent survey showing that 51% of people do not like chocolate ice cream, you wouldn't be entitled to that belief, either, unless you could point me to an even more definitive study that agreed with you. Even the belief "I like chocolate ice cream" could be proven false in some situations - peoples' tastes do change over time, and you could try it one summer and discover that you just don't enjoy it any more. It also implies that you're supposed to go looking for proof of your claims before you make them - that you're not 'entitled' to have or spread an opinion, but instead must earn the right by doing or referencing research.

0AdeleneDawner15y

(And I agree with the two posters in the other comment-branches who pointed out that it's a poor wording.)

5JamesCole15y

That article is entitled "You Are Never Entitled to Your Opinion" and says: I don't think Robin really means that people aren't entitled to their opinions. I think what he really means is people aren't allowed to say "I'm entitled to my opinion" - that is, to use that phrase as a defense. There's a big difference. When people use that defense they don't really mean "I'm entitled to have an opinion", but instead "I'm entitled to express my opinion without having it criticised". In other words "I'm entitled to my opinion" is really a code for "all opinions are equally valid and thus can't be criticised".

3JamesCole15y

That said, I do think it is valid to say "I am entitled to an opinion" in situations where your right to expression is being attacked. I'm not saying you always do have a right to freely and fully express yourself. But in situations when you do have some measure of this, it can be unfairly stomped on. For example, you might be in a business meeting where you should be able to have input on a matter but one person keeps cutting you off. Or say you're with friends and you're outlining your view on some topic and, though you're able to get your view out there, someone else always responds with personal attacks. Sometimes people are just trying to shut you down.

3loqi15y

I don't see how "I'm entitled to my opinion" is a particularly optimal or meaningful response to these situations. What about "it's unfair not to give me a chance to express my position" in the former situation, and "concluding I'm an asshole because I'm pro-X isn't justified" in the latter?

3JGWeissman15y

Right, "opinion" is so overloaded with meaning that in order to determine if the use of "I'm entitle to my opinion" or "You are not entitled to your opinion" is virtuous, one should taboo "opinion", and probably "entitled" as well, and express the thought in way that is specific to the situation, such as in your examples. And of course, having gone through the mental exercise of validating that what you say makes sense, you should give everyone else the benifet of this thought process and actually communicate the alternate form, so they also can tell if it is virtuous.

2AdeleneDawner15y

Agreed, absolutely. I have nothing against hearing about people's half-baked theories - something about the theory or their logic may turn out to be useful, or give me an idea about something else, even if the theory is wrong. But it'd be nice to be able to ask "so why do you think that?" without risking an unpleasant reaction. It might even lead me to figure out that some idea that I would have otherwise dismissed is actually correct!

8Annoyance15y

Most people don't derive their conclusions from reasons. They establish conclusions, then go searching for 'reasons' to cite. Asking for the reasons for the conclusion, in a way that indicates the conclusion ought to follow from them, is perceived by most people as an attack. The only way not to risk receiving an unpleasant reaction is to avoid talking to such people.

5AdeleneDawner15y

Yes, but maybe if there was a social norm such that if I asked that and they couldn't answer, they would take the social-status hit, instead of me, they wouldn't act that way.

3Annoyance15y

Social pressure is pretty much the only thing that can force normal people to acknowledge failures of rationality, in my experience. In a milieu in which a rationalization of that failure will be accepted or even merely tolerated, they'll short-circuit directly to explaining the failure away rather than forcing themselves to acknowledge the problem. Yeah, it'd be nice, but it's probably not going to happen.

2AdeleneDawner15y

Yes, I was giving people too much credit again, wasn't I?

2Annoyance15y

It took me years to even recognize that I was doing that, and I still haven't managed to stop completely. One obstacle: as long as they aren't expected to produce obvious results to meet your expectations, people really, really like being given too much credit. And they really, really dislike being given precisely enough credit when they're nothing special, even if it lets them off the hook. Many of my social 'problems' began once I recognized that other people didn't think like I did, and were usually profoundly stupid. That's not a recognition that lends itself to frictionless interaction with others.

3alvarojabril15y

This little tidbit highlights so much of what's wrong with this community: "Many of my social 'problems' began once I recognized that other people didn't think like I did, and were usually profoundly stupid. That's not a recognition that lends itself to frictionless interaction with others." You'd think a specimen of your gargantuan brainpower would have the social intelligence to handily conceal your disdain for the commonfolk. Perhaps it's some sort of signaling?

5AdeleneDawner15y

I think you're underestimating the degree of social intelligence required. To pull that off while still keeping the rationalistic habits that such people find offensive, you'd have to: * Recognize the problem, which is nontrivial, * Find a way of figuring out who falls on which side of the line, without tipping people off, * Determine all of the rationalistic habits that are likely to offend people who are not trying to become more rational, * Find non-offensive ways of achieving those goals, or find ways of avoiding those situations entirely, * Find a way not to slip up in conversation and apply the habits anyway - again, nontrivial. Keeping this degree of focus in realtime is hard. You'd also probably have to at least to some degree integrate the idea that it's 'okay' (not correct, just acceptable) to be irrational into your general thought process, to avoid unintentional signaling that you think poorly of them. If anything, irrational people are more likely to notice such subtle signals, since so much of their communication is based on them.

5pjeby15y

Or, you could just treat the existence of irrationality as a mere fact, like the fact that water freezes or runs downhill. Facts are not a matter of correctness or acceptability, they just are. In fact (no pun intended), assigning "should-ness" to facts or their opposites in our brains is a significant force in our own irrationality. To say that people "should" be rational is like saying that water "should" run uphill - it says more about your value system than about the thing supposedly being pointed to. Functionally, beliefs about "should" and "should not" assign aversive consequences to current reality - if I say water "should" run uphill, then I am saying that is is bad that it does not. The practical result of this is to incur an aversive emotional response every time I am exposed to the fact that water runs downhill -- a response which does not benefit me in any way. A saner, E-prime-like translation of "water should run uphill" might be, "I would prefer that water ran uphill". My preference is just as unlikely to be met in that case, but I do not experience any aversion to the fact that reality does not currently match my preference. And I can still experience a positive emotional response from, say, crafting nice fountains that pump water uphill. It seems to me that a rationalist would experience better results in life if he or she did not experience aversive emotions from exposure to common facts... such as the fact that human beings run on hardware that's poorly designed for rationality. Without such aversions, it would be unnecessary to craft complex strategies to avoid signaling them to others. And, equally important, having aversive responses to impersonal facts is a strong driver of motivated reasoning that's hard to detect in ourselves!

1Vladimir_Nesov15y

Good summary; the confusion of treating natural mindless phenomena with intentional stance was addressed in the Three Fallacies of Teleology post. When it is possible to change the situation, emotion directed the right way acts as reinforcement signal, and helps to learn the correct behavior (and generally to focus on figuring out a way of improving the situation). Attaching the right amount of right emotions to the right situations is an indispensable tool, good for efficiency and comfort.

1pjeby15y

The piece you may have missed is that even if the situation can be changed, it is still sufficient to use a positive reinforcement to motivate action, and in human beings, it is generally most useful to use positive reinforcement to motivate positive action. This is because, on the human platform at least, positive reinforcement leads to exploratory, creative, and risk-taking behaviors, whereas negative reinforcement leads to defensive, risk-avoidance, and passive behaviors. So if the best way to change a situation is to avoid it, then by all means, use negative reinforcement. However, if the best way to change the situation is to engage with it, then negative emotions and "shoulds" are your enemy, not your friend, as they will cause your mind and body to suggest less-useful behaviors (and signals to others).

0Vladimir_Nesov15y

IAWYC, modulo the use of "should": at least with connotations assumed on Less Wrong, it isn't associated with compulsion or emotional load, it merely denotes preference. "Ought" would be closer.

4pjeby15y

It's true that in technical contexts "should" has less emotional connotation; however even in say, standards documents, one capitalizes SHOULD and MUST to highlight the technical, rather than colloquial sense of these words. Banishing them from one's personal vocabulary greatly reduces suffering, and is the central theme of "The Work" of Byron Katie (who teaches a simple 4-question model for turning "shoulds" into facts and felt-preferences). Among a community of rationalists striving for better communication, it would be helpful to either taboo the words or create alternatives. As it is, a lot of "shoulds" get thrown around here without reference to what goal or preference the shoulds are supposed to serve. "One should X" conveys no information about what positive or negative consequences are being asserted to stem from doing or not-doing X -- and that's precisely the sort of information that we would like to have if we are to understand each other.

0loqi15y

Agreed. Even innocuous-looking exceptions, like phrases of the form, "if your goal is to X, then you should Y", have to make not-necessarily-obvious assumptions about what exactly Y is optimizing.

0Vladimir_Nesov15y

Avoiding existing words is in many cases a counterproductive injunction, it's a normal practice when words get stolen for terms of art. Should refers to a sum total of ideal preference, the top level terminal goal, over all of the details (consequences) together. Should may require a consequentialist explanation for instrumental actions, or a moral argument for preference over consequences.

0AdeleneDawner15y

Agreed. This is one of the major themes of some (most?) meditation practices and seems to be one of the most useful.

0[anonymous]15y

I seriously doubt we're capable of not associating it with those things, though. I think of "should" and "ought" as exactly synonymous, btw.

0[anonymous]15y

Thanks to both of you for expressing so clearly what I failed to, and with links!

0AdeleneDawner15y

That's just what I was trying to get at. Thanks for the clarification.

0alvarojabril15y

The problems you cite in bullets are only nontrivial if you don't sufficiently value social cohesion. My biggest faux pas have sufficiently conditioned me to make them less often because I put a high premium on that cohesion. So I think it's less a question of social intelligence and more one of priorities. I don't have to keep "constant focus" - after a few faux pas it becomes plainly apparent which subjects are controversial and which aren't, and when we do come around to touchy ones I watch myself a little more.

2AdeleneDawner15y

I thought I would get away with that simplification. Heh. Those skills do come naturally to some people, but not everyone. They certainly don't come naturally to me. Even if I'm in a social group with rules that allow me to notice that a faux pas has occurred (not all do; some groups consider it normal to obscure such things to the point where I'll find out weeks or months later, if at all), it's still not usually obvious what I did wrong or what else I could do instead, and I have to intentionally sit down and come up with theories that I may or may not even have a chance to test.

1alvarojabril15y

Right, I get that people fare differently when it comes to this stuff, but I do think it's a matter of practice and attention more than innate ability (for most people). And this is really my point, that the sort of monastic rationality frequently espoused on these boards can have politically antirational effects. It's way easier to influence others if you first establish a decent rapport with them.

0AdeleneDawner15y

I don't at all disagree that the skills are good to learn, especially if you're going to be focusing on tasks that involve dealing with non-rationalists. I think it may be a bit of an over generalization to say that they should be a high priority for everyone, but probably not much of one. I do have a problem with judging people for not having already mastered those skills, or for having higher priorities than tackling those skills immediately with all their energy, though, which seems to be what you're doing. Am I inferring too much when I come to that conclusion?

0alvarojabril15y

Look, this whole thread started because of Annoyance's judgment of people who have higher priorities than rationality, right? Did you have a problem with that? All I'm saying is that this community in general gives way too short shrift to the utility of social cohesion. Sorry if that bothers you.

0AdeleneDawner15y

Quote, please? Most of what he said condenses to "people who are not practicing rationality are irrational", which is only an insult if you consider 'irrational' to be an insult, which I didn't see any evidence of. I saw frustration at the difficulty in dealing with them without social awkwardness, but that's not the same. Have I missed something?

0alvarojabril15y

Yes, and most of what I said reduces to "Annoyance is not practicing rationality with statements like "'social cohesion is one of the enemies of rationality.'" You said you had a "problem" with my contention and then I pointed out that Annoyance had made a qualitatively similar claim that hadn't bothered you. Aside from our apparent disagreement on the point I don't get how my claim could be a problem for you. I think I've made myself clear and this is getting tiresome so I'll invite you to have the last word.

2AdeleneDawner15y

I hope I'm not the only one who sees the irony in you refusing to answer my question about your reasoning, given where this thread started. I guess the best option now is to sum this disagreement up in condensations. For simplicity's sake, I'm only going to do comments on the branch that leads directly here. I'm starting with this comment. * JamesCole: Quoted hypothetical social-norm suggestion, disagreed, offered altenate suggestion suggestion, offered supporting logic. * JamesCole: Restated supporting logic. * Me: Agreed, offered more support. * Annoyance: counterargument: "Most people are not interested enough in being rational for that suggestion to work; they'll find a way around it, instead" * Me: disagreement with Annoyance - I was wrong * Annoyance: Pointed out my mistake * Me: "Oh, right" * Annoyance: "That is a common mistake, and one that I haven't fully overcome yet, which means I still have trouble communicating with people who are not practicing rationality" (probably intended to make me feel better) * You: "I object to the above exchange; you're just masking your prejudice against irrational people by refusing to communicate clearly with them" * Me: "Actually, it's not a refusal, it's just hard." * You: "No, it's not hard, and refusal to do it means that you don't value social cohesion." with a personal example of it not being hard. * Me: "Okay, you got me. It's only hard for some people." * You: "Okay, it is hard for some people, but it's still learnable, and harmful to the cause of rationality if you present yourself as a rationalist without having those skills." * Me: "They're good to learn, but I think you're over-valuing them, and judging people for not sharing your values." * You: "Why are you complaining about me being judgmental when you didn't complain about Annoyance being judgmental?", plus what appears to be some social-signaling stuff intended to indicate that I'm a bad person because I don't care about social cohesion. I

0AdeleneDawner15y

Hmm, might you have been referring to this? That's not a judgment against less intelligent people; it's a judgment against all of us, himself included. I recognize it as being the more rational decision in the situation I mentioned here as one that I'm failing at from a rationalist standpoint, and am not going to bother challenging his rational view on a rational forum when the best defense I can think of is "yes, but you shouldn't say that to the muggles".

-2Annoyance15y

Social cohesion is one of the enemies of rationality. It's not necessarily so in that it's not always opposed to it, but it is incompatible with the mechanisms that bring it about and permit it to error-correct. It tends to reinforce error. When it happens to reinforce correctness, it's not needed, and when it doesn't, it makes it significantly harder to correct the errors.

0alvarojabril15y

"When it happens to reinforce correctness, it's not needed" Can you elaborate? I'll note that rationality isn't an end. My ideal world state would involve a healthy serving of both rationality and social cohesion. There are many situations in which these forces work in tandem and many where they're at odds. A perfect example is this site. There are rules the community follows to maintain a certain level of social cohesion, which in turn aides us in the pursuit of rationality. Or are the rules not needed?

0Annoyance15y

Why can't it be? How is that demonstrated?

0alvarojabril15y

It's demonstrated by the fact that you can up/down vote and report anyone's posts, and that you need a certain number of upvotes to write articles. This is a method of policing the discourse on the site so that social cohesion doesn't break down to an extent which impairs our discussion. These mechanisms "reinforce correctness," in your terms. So I'll ask again, can we do away with them? I don't think humanity follows obviously from rationality, which is what I meant about rationality being a means rather than an end.

-2Annoyance15y

You're assuming a fact not in evidence.

0alvarojabril15y

So you tell me what you think they're for, then.

0AdeleneDawner15y

Those rules are rarely discussed outright, at least not comprehensively. I'm pretty sure if I started posting half of my comments in pig-Latin or French or something, for no apparent reason, and refused to explain or stop, I'd be asked to leave fairly quickly, though. That all communication will be in plain English unless there's a reason for it not to be is one example. I'm sure there are others.

0JoeShipley15y

I disagree. It is rational to exploit interpersonal communication for clarity between persons and comfortable use. If the 'language of rationality' can't be understood by the 'irrational people', it is rational to translate best you can, and that can include utilizing societal norms. (For clarity and lubrication of the general process.)

2AdeleneDawner15y

Yes, I agree - my point was that the skill of translating is a difficult one to acquire, not that it's irrational to acquire it.

3JoeShipley15y

Oh, I'm sorry I misunderstood you. Yeah, it can be tiring. I'm a fairly introverted person and need a good amount of downtime between socialization. I guess I was projecting a little -- I use to think social norms were garbage and useless, until I realized neglecting their utility was irrational and it was primarily an emotional bias against them in never feeling like I 'fit in'. Sometimes it feels like you never stop discovering unfortunate things about yourself...

2JoeShipley15y

I agree here: Reading stuff like this totally makes me cringe. I don't know why people of above average intelligence want to make everyone else feel like useless proles, but it seems pretty rampant. Some humility is probably a blessing here, I mean, as frustrating as it is to deal with the 'profoundly stupid', at least you yourself aren't profoundly stupid. Of course, they probably think given the same start the 'profoundly stupid' person was given, they would have made the best of it and would be just as much of a genius as they are currently. It's a difficult realization, when you become aware you're more intelligent then average, to be dropped into the pool with a lot of other smart people and realize you really aren't that special. I mean, in a world of some six billion odd, if you are a one-in-a-million genius, that still means you likely aren't in the top hundred smartest people in the world and probably not in the top thousand. It kind of reminds me of grad school stories I've read, with kids who think they are going to be a total gift to their chosen subject ending up extremely cynical and disappointed. I think people online like to exaggerate their eccentricity and disregard for societal norms in an effort to appeal to the stereotypes for geniuses. I've met a few real geniuses IRL and I know you can be a genius without being horribly dysfunctional.

2AdeleneDawner15y

Rationality and intelligence are not the same thing - I've seen plenty of discussions here despairing about the existence of obviously-intelligent people, masters in their fields, who haven't decided to practice rationality. I also know people who are observably less intelligent than I am, who practice rationality about as well as I do. One major difference between people in that latter group, and people who are not practicing rationality, no matter what the irrational peoples' intelligence levels are, is that those people don't get offended when someone points out a flaw in their reasoning, just as I don't get offended when they, or even people who are not practicing rationality, point out a flaw in mine. People who are less intelligent will probably progress more slowly with rationality, as with any mental skill-set, but that's not under discussion here. The irrational unwillingness to accept criticism is.

0JoeShipley15y

Being called 'profoundly stupid' is not exactly a criticism of someone's reasoning. (Not that anybody was called that.) I think we're objecting to this because of how it'll offend people outside of the 'in group' anyway. Besides that, As much as we might wish we were immune to the emotional shock or glee at our thoughts and concepts being ridiculed or praised. I think it would be a rarity here to find someone who didn't. People socializing and exchanging ideas is a type of system -- It has to be understood and used effectively in order to produce the best results -- and calling, essentially, everybody who disagrees with you 'profoundly stupid' is not good social lubrication.

5AdeleneDawner15y

You appear to be putting words into my mouth, but I'm currently too irritated to detangle this much beyond that point. "Giving people too much credit" was a reference to peoples' desire to be rational. I tend to assume that that's significantly above zero in every case, even though the evidence does not seem to support that assumption. This is a failure to be rational on my part. (I doubt I'll fix that; it's the basis for most of my faith in humanity.) I make no such assumption about intelligence (I do not assume that people want to be more intelligent than they are), and make a conscious effort to remove irrational biases toward intelligent people from my thought process when I encounter them. I have been doing so for years, with a significant degree of success, especially considering that I was significantly prejudiced against less intelligent people, before I realized that it was wrong to hold that view. I have also put significant effort into learning how to bridge both of those communication gaps, and the skills required in each case are different. When I'm simply dealing with someone who's less intelligent, I moderate my vocabulary, use lots of supporting social signaling, make smaller leaps of logic, and request feedback frequently to make sure I haven't lost them. (Those skills are just as useful in regular conversation as they are in explaining things.) When I'm dealing with someone who's not practicing rationality, I have to be very aware of their particular worldview, and only thoughtfully challenge it - which requires lots of complicated forethought, and can require outright lies. The lack of either of those sets of communication skills will make dealing with the relevant people difficult, and can lead to them thinking poorly of you, whether you actually are prejudiced against them or not. Assuming that someone who does not have one of those sets of skills is prejudiced does not, in practice, work - there's a very high risk of getting a false-positiv

0Annoyance15y

A person who is 'thinking' irrationally can only be challeneged to the degree that they're being rational. If they eschew rationality completely, there isn't any way to communicate with them. What have you actually accomplished, if you use social signals to get someone to switch their concept-allegiances?

3AdeleneDawner15y

I thought we'd already defined "practicing rationality" as "intentionally trying to make rational decisions and intentionally trying to become more rational". Whether we had or not, that was what I meant by the term. Someone can be being somewhat rational without 'practicing' rationality, and to the degree that they can accurately predict what effects follow what causes, or accomplish other tasks that depend on rationality, every person I know is at least somewhat rational. Even animals can be slightly rational - cats for example are well known for learning that the sound of a can opener is an accurate sign that they may be fed in the near future, even if they aren't rational enough to make stronger predictions about which instances of that sound signal mealtime. While social signaling can be used on its own to cause someone to switch their allegiances to concepts that they don't value especially highly, that's not the only possible use of it, and it's not a use I consider acceptable. The use of social-signaling that I recommend is intended to keep a person from becoming defensive while 'rationality-level appropriate' rational arguments are used to actually encourage them to change their mind.

-1Annoyance15y

No, only if you rationally try to make rational decisions and rationally try to become more rational. If you're acting irrationally, you're not practicing rationality, in the same way that you're not practicing vegetarianism if you're eating meat.

0AdeleneDawner15y

I wrote this rant before I saw the thing above. I'm not deleting it, because someone may find this useful, but the issue has been resolved. :)

1Douglas_Knight15y

You should expand this into a top-level post. Communication is difficult and I think most people could use advice about it. As it stands, it sounds like broad strokes which are obviously good ideas, but probably hard to implement without more details.

1AdeleneDawner15y

I've been considering it, actually, for my own use if not to post here. I think it'd be useful in several ways to try to come up with actual wordings for the tricks I've picked up.

-2Annoyance15y

Isn't it obvious? Almost everyone is a "useless prole", as you put it, and even the people who aren't have to sweat blood to avoid that fate. Recognizing that unpleasant truth is the first step towards becoming non-useless - but most people can't think usefully enough to recognize it in the first place, so the problem perpetuates itself. I know I'm usually a moron. I've also developed the ability to distinguish quality thinking from moronicity, which makes it possible for me to (slowly, terribly slowly) wean myself away from stupid thinking and reinforce what little quality I can produce. That's what makes it possible for me to occasionally NOT be a moron, at least at a rate greater than chance alone would permit. It's the vast numbers of morons who believe they're smart, reasonable, worthwhile people that are the problem.

3AdeleneDawner15y

I was reading around on the site today, and I think I've figured out why this attitude sends me running the other way. What clued me in was Eliezer's description of Spock in his post "Why Truth? And...". Eliezer's point there is that Spock's behavior goes against the actual ideals of rationality, so people who actually value rationality won't mimic him. (He's well enough known that people who want to signal that they're rational will likely mimic him, and people who want to both be and signal being rational will probably mimic him in at least some ways, and also note that the fact that reversed stupidity is not intelligence is relevant.) It may come as a shock, but in my case, being rational is not my highest priority. I haven't actually come up with a proper wording for my highest priority yet, but one of my major goals in pursuing that priority is to facilitate a universal ability for people to pursue their own goals (with the normal caveats about not harming or overly interfering with other people, of course). One of the primary reasons I pursue rationality is to support that goal. I suspect that this is not an uncommon kind of reason for pursuing rationality, even here. As I mentioned in the comment that I referenced, I've avoided facing the fact that most people prefer not to pursue rationality, because it appears that that realization leads directly to the attitude you're showing here, and I can reasonably predict that if I were to have the attitude you're showing here, I would no longer support the idea that everyone should have as much freedom as can be arranged, and I don't want to do that. Very few people would want to take the pill that'd turn them into a psychopath, even if they'd be perfectly okay with being a psychopath after they took the pill. But there's an assumption going on in there. Does accepting that fact actually have to lead to that attitude? Is it impossible to be an x-rationalist and still value people?

8byrnema15y

This is something I’ve thought a lot about. I’m worried about the consequences of certain negative ideologies present here on Less Wrong, but, actually, I feel that x-rationality, combined with greater self-awareness, would be the best weapon against them. X-rationality -- identifying facts that are true and strategies that work -- is inherently neutral. The way you interpret those facts (and what you use your strategies for) is the result of your other values. Consider, to begin with, the tautology that 99.7% of the population is less intelligent than 0.3% of the population, by some well-defined, arbitrary metric of intelligence. Suppose also, that someone determined they were in the top 0.3%. They could feel any number of ways about this fact: completely neutral, for example, or loftily superior, or weightily responsible. Seen in this way, feeling contempt for "less intelligent" people is clearly the result of a worldview biased in some negative way. Generally, humanity is so complex that however anyone feels about humanity says more about them than it does about humanity. Various forces (skepticism and despair; humanism and a sense of purpose) have been vying throughout history: rationality isn’t going to settle it now. We need to pick our side and move on … and notice which sides other people have picked when we evaluate their POV. I always find it ironic, when 'rationalists' are especially misanthropic here on Less Wrong, that Eliezer wants to develop a friendly AI. Implicit with this goal -- built right in -- is the awareness that rationality alone would not induce the machine to be friendly. So why would we expect that a single-minded pursuit of rationality would not leave us vulnerable to misanthropic forces? Just as we would build friendliness into a perfectly logical, intelligent machine; we must build friendliness into our ideology before we let go of “intuition” and other irrational ways we have of “feeling” what is right, because they contain our hum

0saturn15y

If we assume he has goals other than simply being a self-abasing misanthrope, the attitude Annoyance is showing is far from rational. Arbitrarily defining the vast majority of humans as useless "problems" is, ironically, itself a useless and problematic belief, and it represents an even more fundamental failure than being Spocklike -- Spock, at least, does not repeatedly shoot himself in the foot and then seek to blame anything but himself.

1AdeleneDawner15y

I've pretty much figured that out. If nothing else, Annoyance is being an excellent example of that right now. Next question: Is it something about this method of approaching rationality that encourages that failure mode? How did Annoyance fall off the path, and can I avoid doing the same if I proceed? I'm starting to think that the answer to that last question is yes, though.

2MrHen15y

While I find conversations with Annoyance rather void, I would encourage you to not try and lift (him ?) up as an example of falling off the path or entering failure modes. If you care about the question I would make a post using generic examples. This does a few things: * Gets you away from any emotional responses to Annoyance (both in yourself and anyone else). * Provides a clear-cut example that can be picked apart without making this entire thread required reading. It also cleans up many straw men and red herrings before they happen, since the specifics in the thread are mostly unneeded with relation to the question you have just asked. * Brings attention to the core problem that needs to be addressed and avoids any specific diagnoses of Annoyance (for better or worse)

0AdeleneDawner15y

That's very good advice. However, I'm not going to take it today, and probably won't at all. It seems more useful at this point to take a break from this entirely and give myself a chance to sort out the information I've already gained. I'll definitely be interested in looking at it, in a few days, if someone else wants to come up with that example and continue thinking about it here.

0MrHen15y

I would agree. I pass. The discussion of that topic would be interesting to me but writing the article is not. I have too many partial articles as it is... :P

-4Annoyance15y

A logically incorrect statement. An attitude is rational if it consistently and explicitly follows from data gathered about the world and its functioning. As there are other consequences from my behavior other than the one you so contemptuously dismiss, and you have no grounds for deciding what my goals are or whether my actions achieve them, your claim is simply wrong. Trivially so, in fact. It's not arbitrary. The rational thing to do when confronted with a position you don't understand is ask yourself "Why did that person adopt that position?"

2saturn15y

If your actions accomplish your goals, fine. However, it's safe to say most of the people here don't want to be Annoyances, and it's important to point out that your behavior does not reflect a requirement or implication of rationality. If you disagree, I hope you will explicitly list the assumptions leading to your belief that it's a good idea to treat people with condescension.

0Vladimir_Nesov15y

This is of low value, if the answer doesn't come easily.

-1Annoyance15y

Easy answers are rarely worthwhile. Worthwhile questions are rarely answered easily.

0Vladimir_Nesov15y

Search for an answer requires the question to be worthwhile, which is far from prior expectation for the research of inane-sounding positions people hold.

0pjeby15y

If you want to convince someone of something, it's generally a good idea to understand why they believe what they believe now. People generally have to be convinced out of one belief before they can be convinced into another, and you can't refute or reframe their evidence unless you know what the evidence is. Even if their reasoning is epistemologically unsound, if you know how it's unsound, you can utilize the same type of reasoning to change their belief. For example, if someone only believes things they "see with their own eyes", you would then know it is a waste of time to try to prove something to them mathematically.

0Vladimir_Nesov15y

I agree, but in this case the benefit comes not from the expectation of finding insight in the person's position, but from the expectation of successful communication (education), which was not the motivation referred in Annoyance's comment.

-6Annoyance15y

3alvarojabril15y

Annoyance, you're still dodging the question. Joe didn't ask whether or not in your opinion everyone is a useless prole, he asked why it's useful to make people feel like that. Your notion that "social cohesion is the enemy of rationality" was best debunked, I think by pjeby's point here: http://lesswrong.com/lw/za/a_social_norm_against_unjustified_opinions/rrk more flies with honey and all that.

-2Annoyance15y

I don't want to catch flies.

2alvarojabril15y

Annoyance, your argument has devolved into inanity. If you don't want to popularly cultivate rationality then you disagree with one of the core tenets of this community. It's in the second paragraph of the "about" page: "Less Wrong is devoted to refining the art of human rationality - the art of thinking. The new math and science deserves to be applied to our daily lives, and heard in our public voices." Your circular word games do no good for this community.

1Annoyance15y

Or perhaps simply the recognition that it's sometimes impossible to fluff other people's egos and drive discussion along rational paths at the same time. If people become offended when you point out weaknesses in their arguments - if they become offended if you even examine them and don't automatically treat their ideas as inherently beyond reproach - there's no way to avoid offending them while also acting rationally. It becomes necessary to choose.

6Psychohistorian15y

Really? Have you tried, maybe, just not pointing out the weaknesses in their arguments? Mightn't that be the rational thing to do? Just a polite smile and nod, or a gentle, "Have you considered some alternative?" Or even, "You may well be right." (This is true of pretty much any non-contradictory statement.) Or there are many different ways to argue with someone without being confrontational. Asking curious-sounding questions works fairly well. It's generally easy to recognize how well a person will react to an argument against him. If you have basic people skills, you'll be able to understand what type of argument/approach will communicate your point effectively, and when you simply don't have a chance. The idea that it's necessary to offend people to act rationally seems completely absurd (at least in this context). If it's going to offend them, it's going to accomplish the opposite of your goal, so, rationally, you shouldn't do it. This whole discussion reminds me of the Dave Barry quote that may well have been used earlier on this site: "I argue very well. Ask any of my remaining friends. I can win an argument on any topic, against any opponent. People know this, and steer clear of me at parties. Often, as a sign of their great respect, they don't even invite me." This. Is. Not. Winning.

0AdeleneDawner15y

I was going to say "there are more workarounds than you think", but that's probably my selection bias talking again. That said, there are workarounds, in some situations. It's still not a trivial thing to learn, though.

3StanR15y

It's not just nontrivial, it's incredibly hard. Engaging "system 2" reasoning takes a lot of effort, lowering sensitivity to, and acute awareness of, social cues and signals. The mindset of "let's analyze arguments to find weaknesses," aka Annoynance's "rational paths," is a completely different ballgame than most people are willing to play. Rationalists may opt for that game, but they can't win, and may be reinforcing illogical behavior. Such a rationalist is focused on whether arguments about a particular topic are valid and sound, not the other person's rational development. If the topic is a belief, attempting to reason it out with the person is counterproductive. Making no ground when engaging with people on a topic should be a red flag: "maybe I'm doing the wrong thing." Does anyone care enough for me to make a post about workarounds? Maybe we can collaborate somehow Adelene, I have a little experience in this area.

0pjeby15y

Engaging system 2 is precisely what you don't want to do, since evolutionarily speaking, a big function of system 2 is to function as a decoy/shield mechanism for keeping ideas out of a person. And increasing a person's skill at system 2 reasoning just increases their resistance to ideas. To actually change attitudes and beliefs requires the engagement of system 1. Otherwise, even if you convince someone that something is logical, they'll stick with their emotional belief and just avoid you so they don't have to deal with the cognitive dissonance. (Note that this principle also applies to changing your own beliefs and attitudes - it's not your logical mind that needs convincing. See Eliezer's story about overcoming a fear of lurking serial killers for an example of mapping System 2 thinking to System 1 thinking to change an emotional-level belief.)

1StanR15y

pjeby, sorry I wasn't clear, I should have given some context. I am referencing system 1 and 2 as simplified categories of thinking as used by cognitive science, particularly in behavioral economics. Here's Daniel Kahneman discussing them. I'm not sure what you're referring to with decoys and shields, which I'll just leave at that. To add to my quoted statement, workarounds are incredibly hard, and focusing on reasoning (system 2) about an issue or belief leaves few cycles for receiving and sending social cues and signals. While reasoning, we can pick up those cues and signals, but they'll break our concentration, so we tend to ignore them while reasoning carefully. The automatic, intuitive processing of the face interferes with the reasoning task; e.g. we usually look somewhere else when reasoning during a conversation. To execute a workaround strategy, however, we need to be attuned to the other person. When I refer to belief, I'm not referring to fear of the dark or serial killers, or phobias. Those tend to be conditioned responses--the person knows the belief is irrational--and they can be treated easily enough with systematic desensitization and a little CBT thrown in for good measure. Calling them beliefs isn't wrong, but since the person usually knows they're irrational, they're outside my intended scope of discussion: beliefs that are perceived by the believer to be rational. People are automatically resistant to being asked to question their beliefs. Usually it's perceived as unfair, if not an actual attack on them as a person: those beliefs are associated with their identity, which they won't abandon outright. We shouldn't expect them to. It's unrealistic. What should we do, then? Play at the periphery of belief. To reformulate the interaction as a parable: We'll always lose if we act like the wind, trying to blow the cloak off the traveller. If we act like the sun, the traveller might remove his cloak on his own. I'll think about putting a post togeth

0pjeby15y

My hypothesis is that reasoning as we know it evolved as a mechanism to both persuade others, and to defend against being persuaded by others. Consider priming, which works as long as you're not aware of it and therefore defending against it. But it makes no sense to evolve a mechanism to avoid being primed, unless the priming mechanism were being exploited by our tribe-mates. (After all, they're the only ones besides us with the language skill to trigger it.) In other words, once we evolved language, we became more gullible, because we were now verbally suggestible. This would then have resulted in an arms race of intelligence to both persuade, and defend against persuasion, with tribal status and resources as the prize. And once we evolved to the point of being able to defend ourselves against any belief-change we're determined to avoid, the prize would've become being able to convince neutral bystanders who didn't already have something at stake. The system 1/2 distinctions cataloged by Stanovich & West don't quite match my own observation, in that I consider any abstract processing to be system 2, whether it's good reasoning or fallacious, and whether it's cached or a work-in-progress. (Cached S2 reasoning isn't demanding of brainpower, and in fact can be easily parroted back in many forms once an appropriate argument has been heard, without the user ever needing to figure it out for themselves.) In my view, the primary functional purpose of human of reasoning is to persuade or prevent persuasion, with other uses being an extra bonus. So in this view, using system 2 for truly rational thought is actually an abuse of the system... which would explain why it's so demanding of cognitive capacity, compared to using it as a generator of confabulation and rhetoric. And it also explains why it requires so much learning to use properly: it's not what the hardware was put there for. The S&W model is IMO a bit biased by the desire to find "normative" reasoning (i.e.

0AdeleneDawner15y

Actually, system one can handle a surprising amount of abstraction; I don't have one handy, but any comprehensive description of conceptual synesthesia should do a good job of explaining it. (I'm significantly enough conceptually synesthetic that I don't need it explained, and have never actually needed an especially good reference before.) The fact that I can literally see that the concept 'deserve X' depends on the emotional version of the concept 'should do X', because the pattern for one contains the pattern for the other, makes it very clear to me that such abstractions are not dependent on the rational processing system. It's also noteworthy that synesthesia appears to be a normal developmental phase; it seems pretty likely to me that I'm merely more aware of how my brain is processing things, rather than having a radically different mode of processing altogether.

0pjeby15y

I'd certainly be interested in that. My own definitions are aimed at teaching people not to abstract away from experience, including emotional experience. Certainly there is some abstraction at that level, it's just a different kind of abstraction (ISTM) than system 2 abstraction. In particular, what I'm calilng system 1 does not generally use complex sentence structure or long utterances, and the referents of its "sentences" are almost always concrete nouns, with its principal abstractions being emotional labels rather than conceptual ones. I consider "should X" and "deserve X" to both be emotional labels, since they code for attitude and action towards X, and so both are well within system 1 scope. When used by system 2, they may carry totally different connotations, and have nothing to do with what the speaker actually believes they deserve or should do, and especially little to do with what they'll actually do. For example, a statement like, "People should respect the rights of others and let them have what they deserve" is absolutely System 2, whereas, a statement like "I don't deserve it" (especially if experienced emotionally) is well within System 1 territory. It's entirely possible that my definition of system 1/2 is more than a little out of whack with yours or the original S&W definition, but under my definition it's pretty easy to learn to distinguish S1 utterances from S2 utterances, at least within the context of mind hacking, where I or someone else is trying to find out what's really going on in System 1 in relation to a topic, and distinguish it from System 2's confabulated theories. However, since you claim to be able to observe system 1 directly, this would seem to put you in a privileged position with respect to changing yourself - in principle you should be able to observe what beliefs create any undesired behaviors or emotional responses. Since that's the hard part of mind hacking IME, I'm a bit surprised you haven't done more with the "ea

2AdeleneDawner15y

Yep, it mostly uses nouns, simple verbs, relatedness catgegorizations ('because') , behavior categorizations ('should', 'avoid with this degree of priority'), and a few semi-abstract concepts like 'this week'. Surprisingly, I don't often 'see' the concepts of good or bad - they seem to be more built-in to certain nouns and verbs, and changing my opinion of a thing causes it to 'look' completely different. (That's also not the only thing that can cause a concept to change appearance - one of my closest friends has mellowed from a very nervous shade of orange to a wonderfully centered and calm medium-dark chocolate color over the course of the last year or so.) Hmm... heh, it actually sounds like I just don't use system 2, then. I have and do, actually, and there's very little that's 'undesirable' left in there that I'm aware of (an irrational but so far not problematic fear of teenagers and a rationally-based but problematic fear of mental health professionals and, by extension, doctors are the only two things that come to mind that I'd change, and I've already done significant work on the second or I wouldn't be able to calmly have this conversation with you). The major limitation is that I can only see what's at hand, and it takes a degree of concentration to do so. I can't detangle my thought process directly while I'm trying to carry on a conversation, unless it's directly related to exactly what I'm doing at the moment, and I can't fix problems that I haven't noticed or have forgotten about. I'm going to be putting together a simple display on conceptual synesthesia for my Neuroversity project this week... I'll be sure to send you a link when it's done.

0AdeleneDawner15y

I've been thinking more about this... or, not really. One of the downsides to my particular mind-setup is that it takes a long time to retrieve things from long-term memory, but I did retrieve something interesting just now. When I was younger, I think I did use system two moderately regularly. I do vaguely remember intentionally trying to 'figure things out' using non-synesthetic reasoning - before I realized that the synesthesia was both real and useful - and coming to conclusions. I very distinctly remember having a mindset more than once of "I made this decision, so this is what I'm going to do, whether it makes sense now or not". I also remember that I was unable to retain the logic behind those decisions, which made me very inflexible about them - I couldn't use new data to update my decision, because I didn't know how I'd come to the conclusion or how the new data should fit in. Using that system is demanding enough that it simply wasn't possible to re-do my logic every single time a potentially-relevant piece of data turned up, and in fact I couldn't remember enough of my reasoning to even figure out which pieces of data were likely to be relevant. The resulting single-mindedness is much less useful than the ability to actually be flexible about your actions, and after having that forcibly pointed out by reality a few times, I stopped using that method altogether. There does seem to be a degree of epistemic hygiene necessary to switch entirely to using system one, though. I do remember, vaguely, that one problem I had when I first started using system one for actual problems was that I was fairly easy to persuade - it took a while to really get comfortable with the idea that someone could have an opinion that was well-formed and made sense but still not be something that I would 'have to' support or even take into consideration, for example. Essentially my own concepts of what I wanted were not strong enough to handle being challenged directly, at first. (

0jimrandomh15y

I feel I should jump in here, as you appear to be talking past each other. There is no confusion in the system 1/system 2 distinction; you're both using the same definition, but the bit about decoys and shields was actually the core of PJ's post, and of the difference between your positions. PJ holds that to change someone's mind you must focus on their S1 response, because if they engage S2, it will just rationalize and confabulate to defend whatever position their S1 holds. Now, I have no idea how one would go about altering the S1 response of someone who didn't want their response altered, but I do know that many people respond very badly to rational arguments that go against their intuition, increasing their own irrationality as much as necessary to avoid admitting their mistake.

3StanR15y

I don't believe we are, because I know of no evidence of the following: Perhaps one or both of us misunderstands the model. Here is a better description of the two. Originally, I was making a case that attempting to reason was the wrong strategy. Given your interpretation, it looks like pjeby didn't understand I was suggesting that, and then suggested essentially the same thing. My experience, across various believers (Christian, Jehovah's Witness, New Age woo-de-doo) is that system 2 is never engaged on the defensive, and the sort of rationalization we're talking about never uses it. Instead, they construct and explain rationalizations that are narratives. I claim this largely because I observed how "disruptable" they were during explanations--not very. How to approach changing belief: avoid resistance by avoiding the issue and finding something at the periphery of belief. Assist in developing rational thinking where the person has no resistance, and empower them. Strategically, them admitting their mistake is not the goal. It's not even in the same ballpark. The goal is rational empowerment. Part of the problem, which I know has been mentioned here before, is unfamiliarity with fallacies and what they imply. When we recognize fallacies, most of the time it's intuitive. We recognize a pattern likely to be a fallacy, and respond. We've built up that skill in our toolbox, but it's still intuitive, like a chess master who can walk by a board and say "white mates in three."

0AdeleneDawner15y

This. Exactly this. YES.

2pjeby15y

Tell them stories. If you'll notice, that's what Eliezer does. Even his posts that don't use fiction per se use engaging examples with sensory detail. That's the stuff S1 runs on. Eliezer uses a bit more S2 logic in his stories than is perhaps ideal for a general audience; it's about right for a sympathetic audience with some S2+ skills, though. On a general audience, what might be called "trance logic" or "dramatic logic" works just fine on its own. The key is that even if your argument can be supported by S2 logic, to really convince someone you must get a translation to S1 logic. A person who's being "reasonable" may or may not do the S2->S1 translation for you. A person who's being "unreasonable" will not do it for you; you have to embed S1 logic in the story so that any effort to escape it with S2 will be unconvincing by comparison. This, by the way, is how people who promote things like intelligent design work: they set up analogies and metaphors that are much more concretely convincing on the S1 level, so that the only way to refute them is to use a massive burst of S2 reasoning that leaves the audience utterly unconvinced, because the "proof" is sitting right there in S1 without any effort being required to accept it.

0AdeleneDawner15y

I hadn't actually found the system 1/system 2 meme before this, but it maps nicely onto how I handle those situations. The main trick is to make lots of little leaps of logic, instead of one big one, while pushing as few emotional buttons as you can get away with, and using the emotional buttons you do push to guide the conversation along. An example of that is here. In the original example, telling someone directly that they're wrong pushes all kinds of emotional buttons, and a fully thought out explanation of why is obviously too much for them to handle with system one, so it's going to fall flat, unless they want to understand why they're wrong, which you've already interfered with by pushing their buttons. In my example, I made a much smaller leap of logic - "you're using a different definition of 'okay' than most people do" - which can be parsed by system one, I think. I also used social signaling rather than words to communicate that the definition is not okay, which is a good idea because social signaling can communicate that with much more finesse and fewer emotional buttons pushed, and because people are simply wired to go along with that kind of influence more easily.

1AdeleneDawner15y

No kidding. My sanity-saver ... but obviously not rationality-saver... has been to learn to encourage the people I'm dealing with to be more rational, at least when dealing with me. My inner circle of friends is made up almost entirely of people who ask themselves and each other that kind of question just as a matter of course, now, and dissect the answers to make sure they're correct and rational and well-integrated with the other things we know about each other. That doesn't help at all when I'm trying to think about society in general, though.

0loqi15y

And worse, they can cite completely incoherent "reasons", which can be observed by noting that the sequence resulting from repeated application of "what do you mean by X" basically diverges. It reminds me of the value "bottom" in a lifted type system. It denotes an informationless "result", such as that of a non-terminating computation.

1Ozymandias_King15y

I think the way we conduct debating has become stuck in a bad place. In a debate we want to win quickly, all else equal. But all else is not equal. If you try specifically to be nice, and complement the person on the things they do get right, they have an easier time accepting criticism. In any other social situation than a purely factual debate, would you even think of only being adversarial? This general climate is the aggregate consequence of every debate we have. If the approach is: "Everything about you sucks, now CHANGE!" The reception will not be: "Okay, I will change X and Y, but not Z" but: "My opinions shall be immune to criticism" The internet has enabled this polarization, by making the rationalist crowd (rightfully) more fundamentalist about their epistemic skill. When you see that logic and evidence works to clear up so much confusion and falsity in your beliefs, you think that you can cure the "sick" person of all his diseases in one fell swoop. Thinking of the dilemma as one of opposing "rights" also doesn't help: [My right to criticize your beliefs] vs [Your right to have them not be criticized] When they refuse to listen to your criticism you feel angry about your rights not being respected, rather than sad that you cannot help them towards better beliefs. ---------------------------------------- Disclaimer: The "You" in this comment is the "We as rationalists"

[-]JamesCole15y70

I think a norm is likely to be a product of the solution, not the solution itself.

So the problem is we have a lot of people who don't appreciate what constitutes a reasonable foundation for an opinion. They think they can just say what they feel. To put it one way, they have a poor understanding of the nature of evidence.

I don't think a norm like you describe could have any effect on anyone like that who had a poor understanding of evidence. Those people would just think the norm was wrong or ridiculous.

If they were to come to better understand the nat... (read more)

1Vladimir_Nesov15y

People should be able to consider what they feel, it's valid rational evidence, it may just not be the best that can be done in a given situation, when better evidence is available.

1JamesCole15y

I agree, I probably just didn't explain myself very well. I was just trying to talk about the situations when people express an opinion without really giving any consideration to why they think it is true.

0JGWeissman15y

I would say people should even explain why they think something is true, which would of course force them to consider it. And then, of course, those who disagree can and should explain in detail what they think is wrong.

0Ozymandias_King15y

Do you mean something like: If being merely informed becomes the norm before rational reasoning is a norm, you just end up with the case of more informed political subjects becoming more polarized and more certain of their views. Badly calibrated and worse off than when they started.

[-]Apprentice15y60

In general it would be great if people were more knowledgeable, our conversations were more enlightened etc. etc. - no-one will disagree with that.

But I think there is a dangerous elitist fallacy lurking here somewhere. The validity of your opinion about subject X isn't always strongly dependent on your knowledge of subject X. Let me make up a simple example. We have a choice between getting a red rock or a green rock. Some of us have a utility function that prefers red to green. Others have a utility function that prefers green to red. But a further group... (read more)

0JGWeissman15y

I don't see how it makes sense to reference nationalism or internationalism as terminal values. They are instrumental values, and their effects on a population's happiness/fun/quality of life should be studied and discussed. There are other valid perspectives besides economics for evaluating the issue, such as the sense of community a system promotes, but when people state simply that they value a certain system as a terminal value, their only contribution is the anecdotal data about their own preference. I think the analogy breaks down in that the utility of economic systems can be reduced to the utility of properties about people (the actual entities with values that could be represented by utility functions), which makes more sense than reducing the utility of reflected colors to the utility of the molecular structures that reflect those colors, rather than the utility of the effect of people perceiving those colors. (Of course, even the rock example can have problems, if the molecular structures have bigger impacts on people than perception of color, for example, if green rocks are toxic.)

0Apprentice15y

I don't exactly disagree but you're upping the subtlety a bit. You're arguing that people should not regard certain things as terminal values when making political decisions, things which I think in fact they do regard that way. But if I get to dress my sockpuppets up a bit I can have them say, "Fine, I agree that I'll be a good utilitarian and try to maximize population happiness over time - but I think the long-term benefits of nationalism/integrationism clearly outweigh the marginal effects of short-term economic developments." Then someone could argue against that point - and we would almost certainly have improved the political discourse. But the people with the original elaborate analyses of short-term economic effects would still have been just as wrong in their claimed superiority. In any case, the general point I wanted to make can certainly be made in a formally correct way as long as you don't insist that every rational intelligent being must have the same utility function.

0JGWeissman15y

Certainly, when there is a genuine disagreement in utility functions, it is not reasonable to claim one's own terminal values as privileged (in debate, that is; it is reasonable to personally act on one's own values), but this does not apply when one's stated values are not their terminal values. And it is reasonable to question whether this is so, to try to distinguish the two cases.

[-]JamesCole15y50

I don't think you could rely on people having read a bit about the matter.

The reasons we do or don't believe something aren't so simple.

Behind my belief that quantum physics is a respectable field is not some specific evidence I have that its descriptions of the world are actually true.

My belief in it is derived from of a complex 'web of trust' - I place a certain amount of trust in the channels from which I hear that there is evidence for it, because I trust the social and scientific means by which that evidence is gathered, disseminated and evaluated. ... (read more)

[-]smoofra15y50

The way you build such a norm is by annoyingly pointing out when people violate your proposed norm.

http://en.wikipedia.org/wiki/Consciousness_raising

[-]JamesCole15y40

If we had in place a social norm demanding an adequate amount of background knowledge on the topic before anyone voiced an opinion they expected to be taken seriously, the signal/noise ratio might be somewhat improved.

Unfortunately that'd skew things towards the status quo.

Advances in knowledge often come from taking a very different angle on a problem, by someone who isn't immersed -- and thus not necessarily knowledgeable in -- the existing viewpoints about the problem (e.g. by amateurs or people from different fields).

Ultimately a person's view should be judged just on its own merits.

1AdeleneDawner15y

Actually, counting personal experience as relevant research would counteract this effect in most situations. I'd like to hear a workable way of making that happen, though. In my experience, in any controversial situation, admitting to basing an opinion on personal experience just opens yourself up to personal attacks.

0Ozymandias_King15y

IAWY, but It's easy to overestimate the size of this effect. * We would expect a priori that more information is useful * We often don't know how much the person who succeeds where others fail in fact did know. * "The Wright Brothers succeeding despite lack of experience/knowledge" -story is more easily remembered and spread because it's feels better.

[-]CannibalSmith15y30

Yes, yes, it would be nice.

-1 Porn. The article is nothing but "what if". No suggestion on how do we bring about the goal or even a question to the audience. I didn't get any bit smarter upon reading it.

1AdeleneDawner15y

+1, Promoted useful comments. Sometimes a push to get us started in a direction is a good thing, even if the poster doesn't actually have anything new to add.

[-]LukeParrish15y20

People are entitled to make mistakes, provided they are not overly detrimental to others. What is offensive is not having a mistaken opinion (particularly when this is a freshly formed mistaken impression rather than an entrenched bias), but attempting to spread it far and wide.

Existing safeguards against this include our concept of expertise. More people will listen to someone who has advanced understanding of an area of knowledge, versus a novice. Usually an expert in a subject really can provide better guidance to form valid opinions.

The trouble arises ... (read more)

2JamesCole15y

But that would also mean that nobody but an authority in the religion could criticize the religion. These rules always have to be symmetrical.

2Alicorn15y

It's possible to be an authority on religion without being an authority in the religion, in much someone can be an authority on computers without being one.

1Vladimir_Nesov15y

See Expert At Versus Expert On by Robin Hanson.

[-]Psychohistorian15y20

"Everyone has a right to his/her opinion" is a social standard because it helps people get along. The total solution to this problem is not telling people that they aren't entitled to their opinions, so much as making it so that when you someone's opinion is ill-founded, they say, "Hey, you're right. I should change my opinion!"

Given that, in reality, that almost never happens, the expression exists as a way of maintaining civility. That way the person pointing out that no, the earth is not 6000 years old becomes the "bad guy"... (read more)

3Ozymandias_King15y

How useful are debates in general for changing the opinions of the person you are debating with? Most debates are implicitly or explicitly framed as contests with opponents, a zero-sum game. The right thing might be to focus more on the 3rd party onlookers, some consequences might be: *Seek bigger debates (more viewers) *Feel less sad when your opponent "beats" you, using twisted logic. *Present more and different types of arguments. *Do wrestle pigs, if the debate is entertaining and public. *Focus more on getting new info in a debate, then isolate yourself to perform belief updates when you are not in contest mode anymore. *If the last point applies equally to your opponent, stop before they get annoyed with you. Allow them to perform calm private reasoning later instead. Can someone think of others?

[-]Vladimir_Nesov15y20

Focusing on the listeners, the important thing is for the arguments/opinions to be helpful. Unjustified opinions is one example of unhelpful declaration: there is no point in listening to them. A justified opinion can also be unhelpful, if it isn't understood, or the reason it's justified isn't understood. Or even worse, a justified opinion, or valid knowledge, can have a negative effect on the listener.

Also, a person still needs to hold unjustified opinions on every subject, that's how the decisions are bootstrapped, but voicing these opinions is usually ... (read more)

0JGWeissman15y

Often, someone who presents an incorrect or poorly supported argument can learn from their mistake and sometimes even fix the argument if they are asked for clarification. The solution to information being harmful out of context is not to withhold the information, but to provide the context. Teach people about biases, and that they need to inspect arguments they like for biases as well as arguments they don't like. I don't understand the point you are making here, or the relevance of the link. What do you mean by "how the decisions are bootstrapped"? Perhaps an example would help illustrate what you are talking about.

0Vladimir_Nesov15y

Withholding the information is also a solution. It you can construct a better one for a given situation is a separate issue. I'm talking about priors, or what passes for them at the first step of plausibility elicitation, when you consult your gut feeling on a single question of fact. Even when you decide to seek out the additional info on a decision, you need to start from sufficient expectation in the discovered information improving your decisions. Maybe you are already convinced that Astrology is bunk, and don't need to research the Encyclopedia of Astrology in Twelve Volumes to improve the precision of your conclusion. The decisions like this are done often and without conscious notice, in fact they may determine what does receive conscious attention.

[-]Ozymandias_King15y10

I think the causality has to run: X-Rationalists raise the standards for ordinary rationalists and scientists-> People connected to the scientists raise their standards-> Everyone else

Sort of top down by osmosis rather than decree. Everyone gets slightly better, but most ordinary people won't have to unrealistically become X-Rationalists.

[-]thomblake15y10

I think this problem, if it exists, is aided by the often misguided fact/opinion distinction commonly taught in schools. It's surely useful to some extent in the context of journalism or law, but it leads to all sorts of problems in fields like ethics and science. If 'everbody's entitled to an opinion' and there are bright lines between fact and opinion, then people will feel indignant for their opinions being declared false.

Example:
"Abby: I think it's okay to set cats on fire"
"Bob: But that's clearly wrong; (insert argument here)"
&qu... (read more)

7AdeleneDawner15y

Actually, I have a workaround for the kind of situation you used as your example: Abby: I think it's okay to set cats on fire Me: You have a very strange definition of 'okay'. (Accompanied by appropriate social signaling that it's not the good kind of strange - depending on the person I may just say outright that I don't approve of it.) Abby's statement is actually true, assuming she's not joking or being sarcastic or something odd like that. What she thinks is not correct, if she's using a definition of 'okay' that this society would consider normal, but that's not a very good assumption to make in situations like this, and pointing out the incongruity will get farther than trying to argue that a true statement is false.

[-]Ozymandias_King15y00

I agree that the end result would be valuable, but I think that changing norms for a whole society would be very hard.

Although it might still be easier than the converse of raising the rationality of a whole society: being informed has higher status in society than being rational. It is more related to being a professor, journalist or talking head, whereas rationality is more associated to being a nerd, scientist or economist.

Moderation Log

161Comments