The line goes "to fathom a strange plot, one technique was to look at what ended up happening, assume it was the intended result, and ask who benefited". But in the real world strange secret complicated Machiavellian plots are pretty rare, and successful strange secret complicated Machiavellian plots are even rarer. So I'd be wary of applying this rule to explain big once-off events outside of fiction. (Even to HPMoR's author!)

I agree Eliezer didn't seem to be trying very hard to suppress information. I think that's probably just because he's a human, and humans get angry when they see other humans defecting from a (perceived) social norm, and anger plus time pressure causes hasty dumb decisions. I don't think this is super complicated. Though I hope he'd have acted differently if he thought the infohazard risk was really severe, as opposed to just not-vanishingly-small.

Cf. AI Risk and the Security Mindset.

Thanks, fixed!

"One might think that the possibility of CEV punishing people couldn't possibly be taken seriously enough by anyone to actually motivate them. But in fact one person at SIAI was severely worried by this, to the point of having terrible nightmares, though ve wishes to remain anonymous."

This paragraph is not an Eliezer Yudkowsky quote; it's Eliezer quoting Roko. (The "ve" should be a tip-off.)

This is evidence that Yudkowsky believed, if not that Roko's argument was correct as it was, that at least it was plausible enough that could be developed in [sic] a correct argument, and he was genuinely scared by it.

If you kept going with your initial Eliezer quote, you'd have gotten to Eliezer himself saying he was worried a blackmail-type argument might work, though he didn't think Roko's original formulation worked:

"Again, I deleted that post not because I had decided that this thing probably presented a real hazard, but because I was afraid some unknown variant of it might, and because it seemed to me like the obvious General Procedure For Handling Things That Might Be Infohazards said you shouldn't post them to the Internet."

According to Eliezer, he had three separate reasons for the original ban: (1) he didn't want any additional people (beyond the one Roko cited) to obsess over the idea and get nightmares; (2) he was worried there might be some variant on Roko's argument that worked, and he wanted more formal assurances that this wasn't the case; and (3) he was just outraged at Roko. (Including outraged at him for doing something Roko thought would put people at risk of torture.)

What better place to discuss possible failure modes of an AI design? [...] Yelling and trying to sweep it under the rug was irresponsible.

There are lots of good reasons Eliezer shouldn't have banned R̶o̶k̶o̶ discussion of the basilisk, but I don't think this is one of them. If the basilisk was a real concern, that would imply that talking about it put people at risk of torture, so this is an obvious example of a topic you initially discuss in private channels and not on public websites. At the same time, if the basilisk wasn't risky to publicly discuss, then that also implies that it was a transparently bad argument and therefore not important to discuss. (Though it might be fine to discuss it for fun.)

Roko's original argument, though, could have been stated in one sentence: 'Utilitarianism implies you'll be willing to commit atrocities for the greater good; CEV is utilitarian; therefore CEV is immoral and dangerous.' At least, that's the version of the argument that has any bearing on the conclusion 'CEV has unacceptable moral consequences'. The other arguments are a distraction: 'utilitarianism means you'll accept arbitrarily atrocious tradeoffs' is a premise of Roko's argument rather than a conclusion, and 'CEV is utilitarian in the relevant sense' is likewise a premise. A more substantive discussion would have explicitly hashed out (a) whether SIAI/MIRI people wanted to construct a Roko-style utilitarian, and (b) whether this looks like one of those philosophical puzzles that needs to be solved by AI programmers vs. one that we can safely punt if we resolve other value learning problems.

I think we agree that's a useful debate topic, and we agree Eliezer's moderation action was dumb. However, I don't think we should reflexively publish 100% of the risky-looking information we think of so we can debate everything as publicly as possible. ('Publish everything risky' and 'ban others whenever they publish something risky' aren't the only two options.) Do we disagree about that?

The wiki article talks more about this; I don't think I can give the whole story in a short, accessible way.

It's true that LessWrongers endorse ideas like AI catastrophe, Hofstadter's superrationality, one-boxing in Newcomb's problem, and various ideas in the neighborhood of utilitarianism; and those ideas are weird and controversial; and some criticism of Roko's basilisk are proxies for a criticism of one of those views. But in most cases it's a proxy for a criticism like 'LW users are panicky about weird obscure ideas in decision theory' (as in Auerbach's piece), 'LWers buy into Pascal's Wager', or 'LWers use Roko's Basilisk to scare up donations/support'.

So, yes, I think people's real criticisms aren't the same as their surface criticisms; but the real criticisms are at least as bad as the surface criticism, even from the perspective of someone who thinks LW users are wrong about AI, decision theory, meta-ethics, etc. For example, someone who thinks LWers are overly panicky about AI and overly fixated on decision theory should still reject Auerbach's assumption that LWers are irrationally panicky about Newcomb's Problem or acausal blackmail; the one doesn't follow from the other.

"As well-argued/thoughtful/reasonable as the SSC stuff" is a pretty hard target to hit on most topics. Could you be more specific than "feminist writing", e.g., talk about a certain claim SSC makes that you'd like to see assessed by others?

Popehat's Shirts and Shirtiness strikes me as an especially high-quality post that's SSC-ish while coming out in favor of some ideas you see in feminism. Agenty Duck's Hasty Genderalizations gives a broader argument for worrying about gender bias. Ben Kuhn's On Inclusivity in Less Wrong is an example of a good feminism-relevant response to a specific SSC argument (though I don't necessarily agree 100% with Ben's arguments, and some of them may be dated at this point).

I'm not sure which of these is closest to what you're looking for. I find blogs like Thing of Things, Gruntled and Hinged, and The Unit of Caring useful both for clearly articulating ideas in the feminist memespace and for providing independent confirmation for some SSC views.

I agree it matters in this case, but it doesn't matter whether we use the word "contrarianism" vs. tabooing it.

Also, your summary assumes one of the points under dispute: whether it's possible to be good at arriving at true non-mainstream beliefs ('correct contrarianism'), or whether people who repeatedly outperform the mainstream are just lucky. 'Incorrect contrarianism' and 'correct-by-coincidence contrarianism' aren't the only two possibilities.

One of the most common ways they use the word "contrarian" is to refer to beliefs that are rejected by the mainstream, for whatever reason; by extension, contrarian people are people who hold contrarian beliefs. (E.g., Galileo is a standard example of a "correct contrarian" whether his primary motivation was rebelling against the establishment or discovering truth.) "Aesthetic preference" contrarianism is a separate idea; I don't think it matters which definition we use for "contrarianism".

My typical heuristic for reliable experts (taken from Thinking Fast and Slow I think) is that if experts have tight, reliable feedback loops, they tend to be more trustworthy. Futurism obviously fails this test.

Then it might be that futurism is irrelevant, rather than being expertise-like or bias-like. (Unless we think 'studying X while lacking tight, reliable feedback loops' in this context is worse than 'neither studying X nor having tight, reliable feedback loops.')

Contrarianism isn't really a "field" in itself, and I tend to think of it more as a bias...

Thiel, Yudkowsky, Hanson, etc. use "contrarian" to mean someone who disagrees with mainstream views. Most contrarians are wrong, though correct contrarians are more impressive than correct conformists (because it's harder to be right about topics where the mainstream is wrong).

entirelyuseless made the point that low cryonics use rates in the general population are evidence against the effectiveness of cryonics. James Miller responded by citing evidence supporting cryonics: that cryonicists are disproportionately intelligent/capable/well-informed. If your response to James is just that very few people have signed up for cryonics, then that's restating entirelyuseless' point. "The intellectual quality of some people who have NOT signed up for cryonics is exceptionally high" would be true even in a world where every cryonicist were more intelligent than every non-cryonicist, just given how few cryonicists there are.