Honestly I was suprised at EY's reaction. I thought he had figured out things like that problem and would tear it to pieces rather than become. Possibly I'm not as smart as him, but even presuming Roko's right you would think Rationalists Should Win. Plus, I think Eliezer has publicly published something similar to the Basilisk, albeit much weaker and without being explicitly basilisk like, so I'd have thought he would have worked out a solution. (EDIT: No, turns out it was someone else who came up with it. It wasn't really fleshed out so Eliezer may not have thought much of it or never noticed it in the first place.)
The fact that people are upset by it could be reason to hide it away, though, to protect the sensitive. Plus, having seen Dogma, I get that the post could be an existential risk...
Plus, having seen Dogma, I get that the post could be an existential risk...
My understanding is that the post isn't the x-risk- a UFAI could think this up itself. The reaction to the post is supposedly an x-risk- if we let on we can be manipulated that way, then a UFAI can do extra harm.
But if you want to show that you won't be manipulated a certain way, it seems that the right way to do that is to tear that approach apart and demonstrate its silliness, not seek to erase it from the internet. I can't come up with a metric by which EY's approach is reasonable.
It might mollify people who disagree with the current implicit policy, and make discussion about the policy easier. Here's one option:
One requirement would be that the policy be no more and no less vague than needed for safety.
Discuss.