You started with an intent to associate SIAI with self delusion
I see, he must be one of those innately evil enemies of ours, eh?
My current model of aaronsw is something like this: He's a fairly rational person who's a fan of Givewell. He's read about SI and thinks the singularity is woo, but he's self-skeptical enough to start reading SI's website. He finds a question in their FAQ where they fail to address points made by those who disagree, reinforcing the woo impression. At this point he could just say "yeah, they're woo like I thought". But he's heard they run a blog on rationality, so he makes a post pointing out the self-skepticism failure in case there's something he's missing.
The FAQ on the website is not the place to signal humility and argue against your own conclusions.
Why not? I think it's an excellent place to do that. Signalling humility and arguing against your own conclusions is a good way to be taken seriously.
Overall, I thought aaronsw's post had a much higher information to accusations ratio than your comment, for whatever that's worth. As criticism goes his is pretty polite and intelligent.
Also, aaronsw is not the first person I've seen on the internet complaining about lack of self-skepticism on LW, and I agree with him that it's something we could stand to work on. Or at least signalling self-skepticism; it's possible that we're already plenty self-skeptical and all we need to do is project typical self-skeptical attitudes.
For example, Eliezer Yudkowsky seems to think that the rational virtue of "humility" is about "taking specific actions in anticipation of your own errors", not actually acting humble. (Presumably self-skepticism counts as humility by this definition.) But I suspect that observing how humble someone seems is a typical way to gauge the degree to which they take specific actions in anticipation of their own errors. If this is the case, it's best for signalling purposes to actually act humble as well.
(I also suspect that acting humble makes it easier to publicly change your mind, since the status loss for doing so becomes lower. So that's another reason to actually act humble.)
(Yes, I'm aware that I don't always act humble. Unfortunately, acting humble by always using words like "I suspect" everywhere makes my comments harder to read and write. I'm not sure what the best solution to this is.)
He's also someone with an actual track record of achievement. Could we do with some of those on LW?
When Richard Feynman started investigating irrationality in the 1970s, he quickly begun to realize the problem wasn't limited to the obvious irrationalists.
Uri Geller claimed he could bend keys with his mind. But was he really any different from the academics who insisted their special techniques could teach children to read? Both failed the crucial scientific test of skeptical experiment: Geller's keys failed to bend in Feynman's hands; outside tests showed the new techniques only caused reading scores to go down.
What mattered was not how smart the people were, or whether they wore lab coats or used long words, but whether they followed what he concluded was the crucial principle of truly scientific thought: "a kind of utter honesty--a kind of leaning over backwards" to prove yourself wrong. In a word: self-skepticism.
As Feynman wrote, "The first principle is that you must not fool yourself -- and you are the easiest person to fool." Our beliefs always seem correct to us -- after all, that's why they're our beliefs -- so we have to work extra-hard to try to prove them wrong. This means constantly looking for ways to test them against reality and to think of reasons our tests might be insufficient.
When I think of the most rational people I know, it's this quality of theirs that's most pronounced. They are constantly trying to prove themselves wrong -- they attack their beliefs with everything they can find and when they run out of weapons they go out and search for more. The result is that by the time I come around, they not only acknowledge all my criticisms but propose several more I hadn't even thought of.
And when I think of the least rational people I know, what's striking is how they do the exact opposite: instead of viciously attacking their beliefs, they try desperately to defend them. They too have responses to all my critiques, but instead of acknowledging and agreeing, they viciously attack my critique so it never touches their precious belief.
Since these two can be hard to distinguish, it's best to look at some examples. The Cochrane Collaboration argues that support from hospital nurses may be helpful in getting people to quit smoking. How do they know that? you might ask. Well, they found this was the result from doing a meta-analysis of 31 different studies. But maybe they chose a biased selection of studies? Well, they systematically searched "MEDLINE, EMBASE and PsycINFO [along with] hand searching of specialist journals, conference proceedings, and reference lists of previous trials and overviews." But did the studies they pick suffer from selection bias? Well, they searched for that -- along with three other kinds of systematic bias. And so on. But even after all this careful work, they still only are confident enough to conclude "the results…support a modest but positive effect…with caution … these meta-analysis findings need to be interpreted carefully in light of the methodological limitations".
Compare this to the Heritage Foundation's argument for the bipartisan Wyden–Ryan premium support plan. Their report also discusses lots of objections to the proposal, but confidently knocks down each one: "this analysis relies on two highly implausible assumptions ... All these predictions were dead wrong. ... this perspective completely ignores the history of Medicare" Their conclusion is similarly confident: "The arguments used by opponents of premium support are weak and flawed." Apparently there's just not a single reason to be cautious about their enormous government policy proposal!
Now, of course, the Cochrane authors might be secretly quite confident and the Heritage Foundation might be wringing their hands with self-skepticism behind-the-scenes. But let's imagine for a moment that these aren't just reportes intended to persuade others of a belief and instead accurate portrayals of how these two different groups approached the question. Now ask: which style of thinking is more likely to lead the authors to the right answer? Which attitude seems more like Richard Feynman? Which seems more like Uri Geller?