Well, this is exactly the problem. First step is that people say random things just because they reflect the emotions they have at the moment. Second step is that later they sometimes derive logical consequences of what they previously said. Then bad things happen as a result.

Usually there is a boundary; people often have crazy beliefs in far mode, while having quite sensible beliefs in near mode. They can keep talking bullshit as long as it does not concern them directly, but when it becomes personal they can either conveniently forget to apply the bullshit, or have some general excuse such as "but this specific case is different". This can work surprisingly well as long as there is a social norm of not requiring people to actually act on their abstract beliefs.

Nerds usually lack the social skills to follow this strategy (because no one tells them about it explicitly; because applying this strategy to itself means never talking explicitly about it), usually harming themselves as the result, by following the norms that everyone applauds but no one except a few nerds actually follows. Sometimes they harm the others as the result, for example when they take the norm of killing unbelievers literally and become suicidal bombers; while for most of the society the same words in the holy scripture simply mean "boo unbelievers" without any impact on their everyday life.

And then there is the complication that our civilization became too complex and has too many channels where the far-mode beliefs translate into actions without people noticing that the boundary was crossed. For example, a person holding a stupid belief in far mode could never directly act upon the belief, but they might still vote according to the belief -- and then the politician in the office might actually do it.

It is a useful tool of propaganda to channel these emotions into far-mode beliefs that benefit some specific group. For example, any kind of frustration with various failures in coordination problems (what SSC readers would call "Moloch") can be channeled into hate against "Jews" or "capitalism" or "decadent western civilization", which can in turn influence the political orientation of people.

Yep, that's a problem and an additional problem is that this mechanism is often exploited by agitprop and, generally, dark arts at the social scale.

Nerds usually lack the social skills to follow this strategy

I don't think it has anything to do with nerds or social skills. If I had to come up with an expression for what prevents people from applying their abstract beliefs in practice, it would be the trite "common sense" (which, yes, I know, isn't exactly common).

Essentially it's the matter of being able to recognize consequences when they appear in front of your eyes. Most people, thankfully, require a large amount of pushing and shoving to make the transition from "Ethnicity X is bad" to "We will go and set fire to our neighbour's house and throw stones at his children". It's not that avoiding that transition is a social skill, it's more that watching a house burn and children cry has direct emotional impact that you need very high levels of ideological belief to override.

Fair criticism, indeed the list has been inclusive on some of the more philosophical ideas. Obviously I hold that some scientific ideas could be mistaken. However "alternative" medicine cannot be established using the scientific method, how is that wrong to include as irrational? Out of a list of 1229 ideas that is probably one of the most definite nonsense ideas included.

since they don't have an a priori reason to distrust them

Um, they don't..? As in, entirely unfamiliar with Chairman Mao and his ways?

such passion means the Maoist-types are expressing their "true desires"

That assertion strongly smells of straw.

And why would they do that?

The first obvious point is that when learning human values you need a large dataset which isn't biased by going viral on 4chan.

The more interesting question is what happens when we get more powerful AI which isn't just a chatbot. Suppose in the future a powerful Baysian inference engine is developed. Its not an AGI, so there is no imminent singularity, but it does have the advantages of very large datasets and being completely unbiased. Asking it questions produces provably reliable results in many fields (but it is not smart enough to answer "how do I create AGI?"). Now, there are a lot of controversial beliefs in the world, so I would say it is probable that it answers at least one question in a controversial way, whether this is "there is no God" or "there are racial differences in intelligence" or even "I have ranked all religions, politics and philosophies in order of plausibility. Yours come near the bottom. I would say I'm sorry, but I am not capable of emotions.".

How do people react? Since its not subject to emotional biases, it's likely to be correct on highly controversial subjects. Do people actually change their minds and believe it? After the debacle, Microsoft hardcoded Tay to be a feminist. What happens if you apply this approach to the Baysian inference engine? Well, if there is logic like so:

The scientific method is reliable -> very_controversial_thing

And hardcoded:

P(very_controversial_thing)=0

Then the conclusion is that the scientific method isn't reliable.

I the point I am trying to make is that if an AI axiomatically believes something which is actually false, then this is likely to result in weird behaviour.

As a final thought, for what value of P(Hitler did nothing wrong) does the public start to freak out? Any non-zero ammount? But 0 and 1 are not probabilities!

They would perhaps conclude that an AI has no soul?

There are several conspiracy theories about the airport actually. Apparently there are storage bunkers below the main buildings used for "unsavory business". The MKULTRA-Jonestown conspiracy theory says that MKULTRA created the Jonestown cult if I remember correctly :)

Actually I am a bit surprised, the post got two downvotes already. I was under the impression that LW would appreciate it given it being a site about rationality and all.. I've been reading LW for quite some time but I hadn't actually posted before, did I do something horribly wrong or anything?

'men of good will can always come to a reasonable agreement' article of faith

That's an interesting observation given that SJWs are very... forceful about separating everyone into sheep and goats. They have come to heavily depend upon the existence of "the enemy" fighting which constitutes most of their raison d'etre. There are, of course, parallels with the devil, but the machinations of Satan figure much more prominently in Catholicism and are (almost?) completely absent in the UU doctrine.

this is an ancient value with regards to my family/my friends/my tribe... But I'm doubtful about how far beyond that it would extend.

For low-cost help I think pretty far. Imagine yourself travelling in some non-Christian country where you are clearly not a native (say, China for most people here). You had a minor accident and you are standing at the side of the road over, say, a broken bike and bleeding from a minor gash. You think random strangers won't stop and help you?

I do also think a lot of Social Justice thinking started out as a genuine desire to help people and make them happier

I've come to realize that my view of SJ is insufficiently steely. Do you happen to know of some text that presents SJ in a reasonable way and:

• Is not written by an idiot
• Is not written for idiots
• Handwaves as little as possible
• Does not descend into the post-modernism morass
• Does not reduce SJ to humanism and/or XIX-century liberalism writ large

?

First, "Social justice" is a broad and very diverse movement of people wanting to reduce the amount of (real or perceived) injustice people face for a variety of reasons (skin color, gender, sexual orientation, place of birth, economical position, disability, ...). Like in any such broad political movement, subparts of the movement are less rational than others.

Overall, "social justice" is still mostly a force of reason and rationality against the most frequent and pervasive forms of irrationality in society, which are mostly religion-based, but yes it varies from subparts of the movement. It is, historically, a byproduct of the Enlightenment after all.

That said, there are several levels of "rationality" and "rationalism", and it might be very rational to make irrational demands.

When you make demands in social and political context, you know your demands will usually not be completely fulfilled. Asking for something "impossible" may be the best way, from a game theoretical point of view, to end up with having something not too far from what you really want - the same way that when you're bargaining the price of an item in an informal market (like in latam or maghreb).

It can also be a powerful way to make people think about a question in novel ways and try to find alternative solutions which aren't part of the hypothesis space they usually wander. "Abolish prisons" may seem an irrational demand, and it's very likely that something "like prison" will be required for a few very dangerous individuals, but it can make people think about possible alternatives to prison, something they don't usually do, and which could very well be used for 90% or even perhaps 99% of people currently in prison.

Of course, making "irrational" demands can also be counterproductive, it can discredit the movement, may you appear to be a lunatic, ... but it's a powerful tool to have in your toolbox when you rationally pursue some deep changes in society.