But XiXiDu doesn't understand SIAI's actual object-level claims, let alone the arguments that link them, and he writes AI researchers in a style that looks crankish.
Agreed - some of his questions were cringe-inducing, but overall, I appreciated that series of posts because it's interesting to hear what a broad range of AI researchers have to say about the topic; some of the answers were insightful and well-argued.
I agree that sounding crankish could be a problem, but I don't think Xixidu was presenting himself as writing in LW/SIAI's name. Crankiness from some lesswrongers tarring the reputation of Eliezer's writings is hard to avoid anyway: the main problem is that there's no clear way to refer to Eliezer's writings; "The Sequences" is obscure and covers too much stuff, some of which isn't Eliezer; "Overcoming Bias" worked at the time, and "Less Wrong" is a name that wasn't even used when most of the core Sequences were written, and now mostly refers to the community.
This is a reply to a comment by Yvain and everyone who might have misunderstood what problem I tried to highlight.
Here is the problem. You can't estimate the probability and magnitude of the advantage an AI will have if you are using something that is as vague as the concept of 'intelligence'.
Here is a case that bears some similarity and might shed light on what I am trying to explain:
The use of 'intelligence' is as misleading and dishonest in evaluating risks from AI as the use of 'tech' in Star Trek.
It is true that 'intelligence', just as 'technology' has some explanatory power. Just like 'emergence' has some explanatory power. As in "the morality of an act is an emergent phenomena of a physical system: it refers to the physical relations among the components of that system". But it does not help to evaluate the morality of an act or in predicting if a given physical system will exhibit moral properties.