Let's go with this analogy. The good thing to do is ask a variety of experts for safety evaluations, run the code through a wide variety of tests, etc. The think NOT to do is keep the code a secret while looking for mistakes all by yourself.

No, if you don't want to use code you don't give the code to a variety of experts for safety evaluations but you simply don't run the code. Having a public discussion is like running the code untested on a mission critical system.

What utility do you think is gained by discussing the basilisk?

and it's certainly not a good habit.

Strawman. This forum is not a place where things get habitually banned.

Somehow, blackmail from the future seems less plausible to me than every single one of your examples. Not sure why exactly.

How plausible do you find TDT and related decision theories as normative accounts of decision making, or at least as work towards such accounts? They open whole new realms of situations like Pascal's Mugging, of which Roko's Basilisk is one. If you're going to think in detail about such decision theories, and adopt one as normative, you need to have an answer to these situations.

Once you've decided to study something seriously, the plausibility heuristic is no longer available.

If a philosophical framework causes you to accept a basilisk, I view that as grounds for rejecting the framework, not for accepting the basilisk.

...

To state it yet another way: to me, the basilisk has the same status as an ontological argument for God. Even if I can't find the flaw in the argument, I'm confident in rejecting it anyway.

Despite the other things I've said here, that is my attitude as well. But I recognise that when I take that attitude, I am not solving the problem, only ignoring it. It may be perfectly sensible to ignore a problem, even a serious one (comparative advantage etc.). But dissolving a paradox is not achieved by clinging to one of the conflicting thoughts and ignoring the others. (Bullet-swallowing seems to consist of seizing onto the most novel one.) Eliminating the paradox requires showing where and how the thoughts went wrong.

I believe he thinks that sufficiently clever idiots competing to shoot off their own feet will find some way to do so.

Sorry, I'll be more concrete; "there's a serious risk" is really vague wording. What would surprise me greatly is if I heard that Eliezer assigned even a 5% probability to there being a realistic quick fix to Roko's argument that makes it work on humans. I think a larger reason for the ban was just that Eliezer was angry with Roko for trying to spread what Roko thought was an information hazard, and angry people lash out (even when it doesn't make a ton of strategic sense).

The basilisk is so a priori implausible that you should be extremely suspicious of fancy arguments claiming to prove it.

So are: God, superintelligent AI, universal priors, radical life extension, and any really big idea whatever; as well as the impossibility of each of these.

Plausibility is fine as a screening process for deciding where you're going to devote your efforts, but terrible as an epistemological tool.

Eliezer thought it was so plausible that he banned discussion of it

If you are a programmer and think your code is safe because you see no way things could go wrong, it's still not good to believe that it isn't plausible that there's a security hole in your code.

You rather practice defense in depth and plan for the possibility that things can go wrong somewhere in your code, so you add safety precautions. Even when there isn't what courts call reasonable doubt a good safety engineer still adds additional safety procautions in security critical code. Eliezer deals with FAI safety. As a result it's good for him to have mindset of really caring about safety.

German nuclear power station have trainings for their desk workers to teach the desk workers to not cut themselves with paper. That alone seems strange to outsiders but everyone in Germany thinks that it's very important for nuclear power stations to foster a culture of safety even when that means something going overboard.

The basilisk is so a priori implausible that you should be extremely suspicious of fancy arguments claiming to prove it.

Finding an idea plausible has little to do with being extremely suspicious of fancy arguments claiming to prove it.

Idea that aren't proven to be impossible are plausible even when there are no convincing arguments in favor of them.

If you're saying 'LessWrongers think there's a serious risk they'll be acausally blackmailed by a rogue AI', then that seems to be false. That even seems to be false in Eliezer's case, and Eliezer definitely isn't 'LessWrong'. If you're saying 'LessWrongers think acausal trade in general is possible,' then that seems true but I don't see why that's ridiculous.

Is there something about acausal trade in general that you're objecting to, beyond the specific problems with Roko's argument?

This is evidence that Yudkowsky believed (...) that at least it was plausible enough that could be developed in a correct argument, and he was genuinely scared by it.

Just to be sure, since you seem to disagree with this opinion (whether it is actually Yudkowsky's opinion or not), what exactly is it that you believe?

a) There is absolutely no way one could be harmed by thinking about not-yet-existing dangerous entities; even if those entities in the future will be able to learn about the fact that the person was thinking about them in this specific way.

b) There is a way one could be harmed by thinking about not-yet-existing dangerous entities, but the way to do this is completely different from what Roko proposed.

If it happens to be (b), then it still makes sense to be angry about publicly opening the whole topic of "let's use our intelligence to discover the thoughts that may harm us by us thinking about them -- and let's do it in a public forum where people are interested in decision theories, so they are more qualified than average to find the right answer." Even if the proper way to harm oneself is different from what Roko proposed, making this a publicly debated topic increases the chance of someone finding the correct solution. The problem is not the proposed basilisk, but rather inviting people to compete in clever self-harm; especially the kind of people known for being hardly able to resist such invitation.