What WrongBot says. I approve of the project of getting outside views, and I approve of the idea of making more AI researchers aware of AI as a possible existential risk. (From what I've heard, SIAI is quietly doing this themselves with some of the more influential groups.) But XiXiDu doesn't understand SIAI's actual object-level claims, let alone the arguments that link them, and he writes AI researchers in a style that looks crankish.
I can hardly think of a better way to prejudice researchers against genuinely examining their intuitions than a wacky letter asking their opinions on an array of absurd-sounding claims with no coherent structure to them, which is exactly what he presents them with.
XiXiDu doesn't understand SIAI's actual object-level claims...
I understand them well enough f...
This is a reply to a comment by Yvain and everyone who might have misunderstood what problem I tried to highlight.
Here is the problem. You can't estimate the probability and magnitude of the advantage an AI will have if you are using something that is as vague as the concept of 'intelligence'.
Here is a case that bears some similarity and might shed light on what I am trying to explain:
The use of 'intelligence' is as misleading and dishonest in evaluating risks from AI as the use of 'tech' in Star Trek.
It is true that 'intelligence', just as 'technology' has some explanatory power. Just like 'emergence' has some explanatory power. As in "the morality of an act is an emergent phenomena of a physical system: it refers to the physical relations among the components of that system". But it does not help to evaluate the morality of an act or in predicting if a given physical system will exhibit moral properties.