If the idea is dangerous in the first place (which is very unlikely), it is only dangerous to people who understand it, because understanding it makes you vulnerable.

This strikes me as tautological and/or confusing definitions. I'm happy to agree that the idea is dangerous to people who think it is dangerous, but I don't think it's dangerous and I think I understand it. To make an analogy, I understand the concept of hell but don't think it's dangerous, and so the concept of hell does not bother me. Does the fact that I do not have the born-again Christian's fear of hell mean that they understand it and I don't? I don't see why it should.

I would prefer to never have read about the idea in question.

If you people are this worried about reality, why don't you work to support creating a Paperclip maximizer? It would have a lot of fun doing what it wants to do and everyone else would quickly die. Nobody ever after would have to fear what could possible happen to them at some point.

If you people want to try to turn the universe into a better place, at whatever cost, then why do you worry or wish to not know about potential obstacles? Both is irrational.

The forbidden topic seems to be a dangerous Ugh field for a lot of people here. You have to decide what you want and then follow through on it. Any self-inflicted pain just adds to the overall negative.

Upvoted, agree strongly.

Considering the extraordinary appeal that forbidden knowledge has even for the average person, let alone the exceptionally intellectually curious, I don't think this is a very effective way to warn a person off of seeking out the idea in question. Far from deserving what they get, such a person is behaving in a completely ordinary manner, to exceptionally severe consequence.

Personally, I don't want to know about the idea (at least not if it's impossible without causing myself significant psychological distress to no benefit,) but I've also put significant effort into training myself out of responses such as automatically clicking links to shock sites that say "Don't click this link!"

Look, you have three people all of whom think it is a bad idea to spread this. All are smart. Two initially thought it was OK to spread it.

I see a lot more than three people here, most of whom are smart, and most of them think that Langford basilisks are fictional, and even if they aren't, censoring them is the wrong thing to do. You can't quarantine the internet, and so putting up warning signs makes more people fall into the pit.

I wish I had never come across the initial link on the internet that caused me to think about transhumanism and thereby about the singularity;

I wish you'd talk to someone other than Yudkowsky about this. You don't need anyone to harm you, you already seem to harm yourself. You indulge yourself in self-inflicted psychological stress. As Seneca said, "there are more things that terrify us than there are that oppress us, and we suffer more often in opinion than in reality". You worry and pay interest for debt that will likely never be made.

Look, you have three people all of whom think it is a bad idea to spread this. All are smart.

I read about quite a few smart people who hold idiot beliefs, I only consider this to be marginal evidence.

Furthermore, I would add that I wish I had never learned about any of these ideas.

You'd rather be some ignorant pleasure maximizing device? For me truth is the most cherished good.

If this is not enough warning to make you stop wanting to know more, then you deserve what you get.

BS.

I would add that I wish I had never learned about any of these ideas. In fact, I wish I had never come across the initial link on the internet that caused me to think about transhumanism and thereby about the singularity;

Hmm. It is tricky to go back, I would imagine.

The material does come with some warnings, I believe. For instance, consider this one:

"Beware lest Friendliness eat your soul." - Eliezer Yudkowsky

In fact, I wish I had never come across the initial link on the internet that caused me to think about transhumanism and thereby about the singularity

As I understand, you donate (and plan to in the future) to existential risk charities, and that is one of the consequences of you having come across that link. How does this compute into net negative, in your estimation, or are you answering a different question?

As a decision on expected utility under logical uncertainty, but extremely low confidence, yes. I can argue that it most certainly won't be a bad thing (which I even attempted in comments to the post itself, my bad), the expectation of it being a bad thing derives from remaining possibility of those arguments failing. As Carl said, "that estimate is unstable in the face of new info" (which refers to his own argument, not necessarily mine).