Indeed, this possibility isn't discussed enough.

I'm sympathetic to your position here, but precisely what kind of discussion about this possibility do you want more of?

Eliezer believes that building a superhuman intelligence is so dangerous that experimenting with it is irresponsible, and that instead some kind of theoretical/provable approach is necessary before you even get started.

The specific approach he has adopted is one built on a particular kind of decision theory, on the pervasive use of Bayes' Theorem, and on the presumption that what humans value is so complicated that the best way to express it is by pointing at a bunch of humans and saying "There: that!"

SIAI is primarily populated by people who think Eliezer's approach is a viable one.

Less Wrong is primarily populated by people who either think Eliezer's strategy is compelling, or who don't have building a superhuman intelligence as their primary focus in the first place.

People who have that as their primary focus and think that his strategy is a poor one go put their energy somewhere else that operates on a different strategy.

If he's wrong, then he'll fail, and SIAI will fail. If someone else has a different, viable, strategy, then that group will succeed. If nobody does, then nobody will.

This seems like exactly the way it's supposed to work.

Sure, discussion is an important part of that. But Eliezer has written tens of thousands of words introducing his strategy and his reasons for finding it compelling, and hundreds of readers have written tens of thousands of words in response, arguing pro and con and presenting alternatives and clarifications and pointing out weaknesses.

I accept that you're (as you say) unable to read thousands of comments, so you can't know that, but in the nine months or so since I've found this site I have read thousands of comments, so I do know it. (Obviously, you don't have to believe me. But you shouldn't expect to convince me that it's false, either, or anyone else who has read the same material.)

I'm not saying it's a solved problem... it's not. It is entirely justifiable to read all of that and simply not be convinced. Many people are in that position.

I'm saying it's unlikely that we will make further progress along those lines by having more of the same kind of discussion. To make further progress in that conversation, you don't need more discussion, you need a different kind of discussion.

In the meantime: maybe this is a groupthink-ridden cult. If so, it has a remarkable willingness to tolerate folks like me, who are mostly indifferent to its primary tenets. And the conversation is good.

There's a lot of us around. Maybe we're the equivalent of agnostics who go to church services because we're bored on Sunday mornings; I dunno. But if so, I'm actually OK with that.