I've spoken over PM with the person I believe Roko was talking about when he said "one person at SIAI was severely worried by this, to the point of having terrible nightmares". They say:
I'm the person Roko was referring to in his original post. I saw this thread talking about "nervous breakdowns" and wanted to post a basic clarification to make sure this isn't because people are reading something into that line that's more sensational than the reality. There have been times when I was anxious about possibilities in the space of TDT, acausal blackmail, and simulationism, not "the Basilisk" particularly, but that general sort of thing. This happened both when I was a visiting fellow at SIAI in 2009-10 and earlier. None of this amounted to anything I'd consider a "nervous breakdown". Nobody observing my behavior or internet posting, say, would have noticed much out of the ordinary. These anxieties have faded over time and haven't been an issue for the last decade or so.
Funny how one person becomes "some" becomes "a seemingly significant portion of the 'rationality community'".
(On the other hand, something in my heart rejoices in the thought that in our community, every single person is considered a significant portion.)
I think the claim is false, though it's hard to prove an absence. I have never heard of anyone having a particularly strong emotional reaction to Roko's basilisk (outside of a "sneering" reaction towards rationalists or something), and almost everyone I've met in the community thinks the thought experiment is not particularly interesting or important or dangerous. I do like the general policy of not talking a ton about infohazards of this type, but that's just a broad reference class, of which Roko's basilisk seems like a non-central example.
The Roko's Basilisk wiki article has a lot of detail, and I recommend looking through it. It also includes these paragraphs:
Other sources have repeated the claim that Less Wrong users think Roko's basilisk is a serious concern. However, none of these sources have yet cited supporting evidence on this point, aside from Less Wrong moderation activity itself. (The ban, of course, didn't make it easy to collect good information.)
Less Wrong user Gwern reports that "Only a few LWers seem to take the basilisk very seriously," adding, "It's funny how everyone seems to know all about who is affected by the Basilisk and how exactly, when they don't know any such people and they're talking to counterexamples to their confident claims."
... if THE CLAIM is true then it brings to mind some potentially unkind questions about the psychological heath of a seemingly significant portion of the 'rationality community'.
So I think we have much stronger evidence of psychological health issues with the rationality community (which I assume is the same thing as the 'rationality community' though I'm uncertain) via things like the LW and SSC surveys. Perhaps you do not trust surveys because of self-report issues? But in that case I'd probably look at proxies like common correlates of mental health issues, or maybe suicide rates within the community and see if it's above baseline.
I'm personally pretty convinced that psychological issues in the rationalist community is substantially above baseline[1], but I would not want to trust a fairly complex causal assignment ascribed to a specific cause, that likely has politically motivated reporting, above more mundane instruments like surveys and other observational data.
[1] The main source of hesitation for me is uncertainty about the baseline; it's plausible but unlikely to me that "normal people" are very crazy in the clinical sense but for some reason this is mostly unreported.
Agreed. On priors I would expect above-baseline rates of mental health issues in the community even in the total absence of any causal arrow from the community to mental health issues (and in fact even in the presence of fairly strong mental health benefits from participation in the community), simply through selection effects. Which people are going to get super interested in how minds work and how to get theirs to work better? Who's going to want to spend large amounts of time interacting with internet strangers instead of the people around them? Who's going to be strongly interested in new or obscure ideas even if it makes the people around them think they're kind of weird? I think people in this community are both more likely to have some pre-existing mental health issues, and more likely to recognise and acknowledge the issues they have.
@Linch: My observations (based on an admittedly limited set of observations and my lack of psychological training) agree with your "I'm personally pretty convinced that psychological issues in the rationalist community is substantially above baseline.
I'm surprised, given the claimed truth-seeking and evidentiary rigor values of 'the rationalist community', that there is not a magiteral data-laden LessWrong essay that addresses "psychological issues in the rationalist community" that is cited when discussion turns to this topic. Can anyone point to such an essay?
What does "magiteral" mean here?
At any rate, you're free to be the change you want to see in the world. :)
It was a dirty job, he thought, but somebody had to do it.
As he walked away, he wondered who that somebody might be.
There's no way to give a citation to Eliezer not saying it. You might get further querying the citations in the article for the claim that he did say it. There are three, but the first does not mention it and the third calls it a rumour. The second quotes a chunk of Eliezer's immediate response to Roko, the one that began "Listen to me very closely, you idiot" (which I remember reading at the time), and then says "Yudkowsky said that Roko had already given nightmares to several LessWrong users and had brought them to the point of breakdown." My memory is not detailed enough to say whether he said that, but I think it quite likely that someone did say something of the sort back then.
If you search LessWrong for "Listen to me very closely, you idiot", there is a hit on that comment by Eliezer, but the comment itself has been deleted. Since there is that hit, the comment is presumably still in the database. Perhaps the LessWrong admins might be willing to give you sight of it? But the result either way wouldn't be a citable source for Wikipedia.
The argument for which EY banned the basilisk is a very complex argument and you need to understand TDT to follow it. Newspapers are in the business of dumbing things down for their readers so that their readers get the sense they understand what goes on. As a result you won't find a good explanation of why EY banned it in any newspaper. As a result you won't find a good explanation in anything that's seen as worthy to be a Wikipedia source.
To my knowledge this claim seems to be almost entirely fabricated as the only text that is vaguely reminiscent of this claim in the original thread is a claim from Roko that “One might think that the possibility of CEV punishing people couldn’t possibly be taken seriously enough by anyone to actually motivate them. But in fact one person at SIAI was severely worried by this, to the point of having terrible nightmares, though ve wishes to remain anonymous.” which along with describing the experience of a grand total of one person only refers to anxiety resulting from an idea related to the premises of the thought experiment and not the thought experiment itself.
The "Berkeley, CA based Rationalist Community Forum" sent you to this a few times. In which EY stated that the infohazard itself, not "emotional brittleness" was the cause. This included a direct statement from EY, whereas the links and sources that suggested "emotional brittleness" were sourced as "rumors."
Be careful of DDCT!
@ Shouperfluous: Would you please reread the post to which you are responding? If you can in fact provide suitable evidence then would you please quote and cite it?
Please note my prior: "No one could provide evidence that THE CLAIM [ Yudkowsky censored mention of Roko's basilisk because it "caused some readers to have nervous breakdowns"] is false and sufficient to meet Wikipedia's evidentiary standards so as [to allow editing Wikipedia so as] to rebut or remove the THE CLAIM".
"No one could provide a citation to a specific denial from Yudkowsky that THE CLAIM did not figure into the decision to censor mention of Roko's basilisk."
I emphasize that the issue here is evidence "sufficient to meet Wikipedia's evidentiary standards". You might not like Wikipedia's evidentiary standards. I might share your dislike. But that is not the point. Wikipedia, not us, sets Wikipedia's evidentiary standard.
As a former Wikipedia admin, I don't think the current basilisk content on Wikipedia meets Wikipedia's evidential standards (or general site standards):
Large parts of Wikipedia are inconsistent with the site's standards. This is why the encyclopedia is a work in progress. (Well, that plus the fact that the world is big and keeps changing, and our knowledge isn't exhaustive.)
If one of the editors thinks that no strong evidence is required to include claims like 'some readers of Roko's post had nervous breakdowns', but thinks strong evidence is required in order to remove that claim once it's on a Wikipedia page, then the editor just doesn't understand how Wikipedia works. You're welcome to quote my claims here on the relevant WP Talk page and start a discussion there if you think there's any disagreement on any of these points.
Regarding 2: Eliezer's original reply to Roko's post was ambiguous. The current WP article version claims the correct interpretation is X; Eliezer disputes this, and says he meant Y instead.
The actual meaning, in Eliezer's telling, is roughly: 'Roko's argument is silly, but there seems to be non-negligible probability (here in 2010) that there's some structurally analogous argument out there that would work, and the field hasn't had time to explore the space and confirm that there's no such argument; when we haven't done that basic due diligence, we shouldn't go around posting about this on the public Internet.'
Which is easy to round off via a game of telephone to 'Roko's basilisk is dangerous', and thereby to 'Eliezer thinks Roko's argument is sound'. But Wikipedia isn't supposed to be in the business of contributing to games of journalistic telephone. It aspires to be more like the 'final draft' of humanity's knowledge -- an extremely vetted encyclopedia, more carefully fact-checked than Britannica, with large conservative boundaries in place to ensure it errs on the side of excluding any information it's even slightly uncertain about. It obviously hasn't reached that ideal, since it's hard to wrangle thousands of volunteer editors spread across millions of articles; but that is what the editorial policies are trying to create.
In this case, both interpretations are perhaps consistent with the text (neither is silly or obviously false). I think a case can be made that Eliezer's claim about what he had in mind makes sense, there's no special reason to doubt it, and it can just be reported on, if the topic is noteworthy enough to justify that level of detail.
... But if there is serious published disagreement about whether Eliezer is making stuff up here, then: In cases like this, as a matter of policy, Wikipedia avoids doing original research / detective work into sussing out that Eliezer's lying and his real meaning was something totally different from what he claims it was. Instead, if the topic is important enough to cover in detail at all, you quote the source text and then report on noteworthy claims about what that text means.
_____________________________________________________
Regarding 3: This wouldn't necessarily be out of place in a long Wikipedia article that includes a bunch of other quotes, so long as the article is detailed enough to provide any context for why Auerbach thinks any of those things. (As is, someone reading WP to learn about LW for the first time won't know what any of this has to do with 'transcending conventional morality' or 'messianic ambitions'; bringing all this up without context is just bad writing.)
However, on its own, this is an obviously axe-grindy choice of quote that makes the Wikipedia article look like an op-ed that's trying to support a critique of LW, rather than looking like an encyclopedia.
My response is a direct refutation of THE CLAIM, in that Yudkowsky explains the rational for the prohibition.
@Shouperfluous — You don't understand the meaning of "direct refutation".
An actual direct refutation of THE CLAIM would be a quote from Yudkowsky to the effect that "my decision to censor mention/discussion of Roko's basilisk was in no way motivated by emotional reactions from members of 'the rationalist community' including but not limited to nervous breakdowns and nightmares.'
You might infer a refutation to the THE CLAIM from the above-linked post but your inference is not a "direct refutation".
Given the latest post about not just Leverage but also CFAR and MIRI, the Wikipedia article now seems to me sufficiently accurate as it stands.
I recently came upon the story of "Roko's basilisk". Heretofore I'd not seen the topic. Reading about it in Wikipedia[1] I saw "Discussion of Roko's basilisk was banned on LessWrong for several years because Yudkowsky had stated that it caused some readers to have nervous breakdowns".[1]
That Yudkowsky censored mention of Roko's basilisk because it "caused some readers to have nervous breakdowns" (henceforth THE CLAIM) seemed very strange to me. So I asked in a Berkeley CA 'rationalist community' forum — "This story of censorship of Roko's basilisk and it causing "nervous breakdowns" strikes me me a bizarre; Is this some sort of joke?".
The answers to this question, some of which were angry and emotional, amounted to:
1) THE CLAIM is false.
2) No one could provide evidence that THE CLAIM is false sufficient to meet Wikipedia's evidentiary standards so as to rebut or remove the THE CLAIM in Wikipedia..
3) No one could provide a citation to a specific denial from Yudkowsky that THE CLAIM did not figure into the decision to censor mention of Roko's basilisk.
In the interest of truth-seeking ...
... if the THE CLAIM is false then I'd like to obtain evidence to that effect and correct Wikipedia.
... if THE CLAIM is true then it brings to mind some potentially unkind questions about the psychological heath of a seemingly significant portion of the 'rationality community'.
Please note that my questions here bears only on THE CLAIM (ie. "caused some readers to have nervous breakdowns") and not any of the issues associated with Roko's basilisk, eg. claims of infohazard.
Even if the "caused some readers to have nervous breakdowns" is a hyperbolic description, with no actual "nervous breakdowns" occurring, the notion that censorship needed to be employed to protect emotionally brittle persons in the 'rationality community' from a thought experiment reflects poorly on the community.
Ref.
{1] https://en.wikipedia.org/wiki/LessWrong#Roko's_basilisk