However, there are ideologies that don't contain shards of infinite utility, or that contain a precautionary principle that guards against shards of infinite utility that crop up. They'll say things like "don't trust your reasoning if it leads you to do awful things" (again, compare that to "shut up and calculate"). For example, political conservatism is based on a strong precautionary principle. It was developed in response to the horrors wrought by the French Revolution.

I don't know how you could read LW and not realize that we certainly do accept precautionary principles ("running on corrupted hardware" has its own wiki entry), that we are deeply skeptical of very large quantities or infinities (witness not one but two posts on the perennial problem of Pascal's mugging in the last week, neither of which says 'you should just bite the bullet'!), and libertarianism is heavily overrepresented compared to the general population.

One of the big black marks on the SIAI/LW is the seldom discussed justification for murder and terrorism that is a straightforward result of extrapolating the locally accepted morality.

No, one of the 'big black marks' on any form of consequentialism or utilitarianism (as has been pointed out ad nauseam over the centuries) is that. There's nothing particular to SIAI/LW there.

LW scares me. It's straightforward to take the reasoning of LW and conclude that terrorism and murder are justified.

Is there any ideology or sect of which that could not be said? Let us recall the bloody Taoist and Buddhist rebellions or wars in East Asian history and endorsements of wars of conquest, if we shy away from Western examples.

Nevermind that there were no actual plans for destroying fabs, and that the whole "terrorist plot" seems to be a collective hallucination.

Nevermind that the author in question has exhaustively argued that terrorism is ineffective.

I got a distinct cultish vibe when I joined, but only from the far-out parts of the site, like UFAI, but not from the "modern rationality" discussions. When I raised the issue on #lesswrong, the reaction from most regulars was not very reassuring: somewhat negative and more emotional than rational. The same happened when I commented here. That's why I am looking forward to the separate rationality site, without the added untestable and useless to me EY's idiosyncrasies, such as the singularity, the UFAI and the MWI.

Banning topics that show unpalatable results of Yudkowsky's philosophy was a bad start.

If you're talking about the basilisk, it wasn't banned because it reflected poorly on Yudkowsky's philosophy, it was banned because it was potentially quite dangerous. Its author was a former visiting fellow at the SI, generally agreed with Yudkowsky, and agreed (IIRC) with the decision to censor the topic.

So is the near perfect certainty that the institute members project on their entirely a priori reasoning regarding the nature of future AI.

The claim that institute members have near perfect certainty is false. You can easily see this in the linked section of the FAQ; this is true to the extent that Aaron seemed to think that the SI folks simply weren't address some criticisms that they semi-agreed with. For that matter, see the most recent post in main on LW.

If the FAQ isn't the place for self-skepticism, perhaps you could point us to other places where it exists?

He did- see my linked comment.

I am sad I missed Aaron's original article before he was pressured into changing it. I probably would have found much to agree with.

It was almost identical; it replaced the paragraph about the Heritage Institute with a link to the SI's FAQ, with a few brief and non-substantive criticisms (that it didn't address many potential criticisms, that it didn't address their full impact, and that it was too short), and then made a terrible analogy between the SI and Uri Geller without any substantive evidence.

Let's try an easier question first. If someone is about to create Skynet, should you stop them?

I hope SI will agree that the FAQ answer you linked is inadequate

As Randaly notes, an FAQ of short answers to common questions is the wrong place to look for in-depth analysis and detailed self-skepticism! Also, the FAQ links directly to papers that do respond in some detail to the objections mentioned.

Another point to make is that SI has enough of a culture of self-skepticism that its current mission (something like "put off the singularity until we can make it go well") is nearly the opposite of its original mission ("make the singularity happen as quickly as possible"). The story of that transition is here.

...you just jinxed it! Now congress is going to pass a new bill forbidding online aids to count towards compulsory education requirements for home schooling, and otherwise hamper the idea by whatever means necessary.

After all, what better propaganda system is there than a bunch of gullible "teachers" who regurgitate everything you tell them to and whom children look up to as absolute authorities?

This Company starts in the United States and ties into existing home school regulations with a self-driven web learning program that requires minimum parental involvement and results in a high school degree.

The nice thing about this is that it works on an existing market, while leveraging the successful tactics discovered through hard work by Coursera & the like to bring advances to the domain.

Of course, techniques designed for university courses may not precisely transfer.

I'm skeptical about 'leveraging' videos from Khan Academy for a for-profit education system. Makes it sound half-baked.

This idea may fit with the general spaced-repetition enthusiasm I am seeing in other proposals.

It cloaks itself as merely a tool to aid homeschool parents, similar to existing mail-order tutoring materials, hiding its radical mission to end high school as we know it.

...And you just blew your cover. :)

"Give your listeners the facts—the Family Facts from the experts at The Heritage Foundation." I'm completely reassured.