Not only that, but also that it's a giant scam. Because if that is the case, even at a probability as low as 0.1%, valuable resources would be wasted that could be used to mitigate other existential risks, or being used by someone who follows selfish motives.

You say that as if it's worse than other ways that money could go to waste.

(There are good game-theoretic reasons to act sort of as if you thought that, but they should be made explicit, and explicit consideration of them probably wasn't what motivated your statement.)

It is very easy to dispel any such doubts, all he would have to do is publish some technical paper that manages to survive peer-review, thereby substantiate his claims and prove that he is qualified.

You seem to have the idea that this is all about Eliezer Yudkowsky. In actual fact, he wasn't at the meeting where we came up with the model I described in this article, he's influential but doesn't control SIAI, and the existential risk issue is bigger than SIAI and a lot bigger than any one person. Most of the people involved think AI risk is important based on their own reasoning, not based on trusting Eliezer. Personally, I don't really care whether he's qualified, because I consider myself qualified enough to judge his arguments (or anonymous arguments) directly. What may be throwing you off is that he's extremely visible - he's the public face of SIAI to a lot of people - because he's a prolific writer, and because he optimizes his writing to get lots of people to read it.

Journals are actually very bad for getting read by non-specialists, and Eliezer's specialized his writing skill for presenting to smart laymen, rather than academics. Nevertheless, other authors have written and published papers about AI risk have been published. The issue at hand right now is getting into prestigious machine learning and computer science journals, rather than philosophy journals, so that the right specialists will read them. That's much more difficult, because their editors think of them as having narrow topics that don't include philosophy or futurism.

I don't think that the argument, that people can be smart enough to create an AGI, that can take over the universe in a matter of hours, can be dumb enough not to recognize the dangers posed by such an AGI, is very strong. To fortify that argument you would either have to show that the people working for SIAI are vastly more intelligent than most AGI researchers, in which case they would be more likely to build the first AGI, or that the creation of an AGI, that is capable of explosive recursive self-improvement, demands much less intelligence and insight than which is necessary to recognize risks from AI.

I have some scepticism about that point as well. We do have some relevant history relating to engineering disaasters. There have been lots of engineering projects in history, and we know roughly how many people died in accidents, and as a result of negligence, or were screwed over in other ways.

Engineers do sometimes fail. The Titanic. The Tacoma Narrows Bridge Collapse.

Then there's all the people killed by cars and in coal mines. Society wants the benefits, and individuals pay the price. This effect seems much more significant than accidents to me - in terms of number of deaths.

However, I think that engineers have a reasonable record. In a historical enginnering project with lives at stake, one would certainly not expect failure - or claim that failure is "the default case". The case goes the other way: high technology and machines cause humans to thrive.

Of course, reference class forecasting has limitations, but we should at least attempt to learn from this type of data.

I don't think that the argument, that people can be smart enough to create an AGI, that can take over the universe in a matter of hours, can be dumb enough not to recognize the dangers posed by such an AGI, is very strong. To fortify that argument you would either have to show that the people working for SIAI are vastly more intelligent than most AGI researchers, in which case they would be more likely to build the first AGI, or that the creation of an AGI, that is capable of explosive recursive self-improvement, demands much less intelligence and insight than which is necessary to recognize risks from AI.

One of the major messages which I think you should be picking up from the sequences is that it takes more than just intelligence to consistently separate good ideas from bad ones.