Maybe this is a silly question I should already know the answer to and/or not quite appropriate for this thread, but I can't think of anywhere better to ask. Apologies also if it was already directly addressed.
What's the point of suppressing discussion of certain concepts for fear of a superintelligent AI discovering them? Isn't it implicit that said superintelligent AI will independently figure out anything we humans have come up with? Furthermore, what was said in Roko's post will exist forever; the idea was released to the interwebs and its subsequent banning can't undo this.
It's not the attempt to hide an idea from a future AI, but something else entirely. I'll reply (in vague terms) by private message.
EDIT: People keep asking me privately; and after thinking for a while, I'm confident that it's not harmful to post my analogy of why a Topic could be rightly Banned (in addition to the very good reason that it can give people nightmares), without giving away what Banned Topic consists of. (If our benevolent overlords disagree on the wisdom of this, I'm happy to edit or delete my comment, and will gladly accept having it done ...
This is our monthly thread for collecting arbitrarily contrived scenarios in which somebody gets tortured for 3^^^^^3 years, or an infinite number of people experience an infinite amount of sorrow, or a baby gets eaten by a shark, etc. and which might be handy to link to in one of our discussions. As everyone knows, this is the most rational and non-obnoxious way to think about incentives and disincentives.