If you weaken the odds of creating general intelligence to around 50-50, then virtually none have given decent counterarguments to 1)-7). The disconnect starts at 8)-11).
How much evidence, that you are right, does the absence of counter-arguments actually constitute?
Quite strong evidence, at least for my position (which has somewhat wider error bars that SIAI's). Most people who have thought about this at length tend to agree with me, and most arguments presented against it are laughably weak (hell, the best arguments against Whole Brain Emulations were presented by Anders Sandberg, an advocate of WBE).
I find the arguments in favour of the risk thesis compelling, and when they have the time to go through it, so do most other people with relevant expertise (I feel I should add, in the interest of fairness, that neuroscientists seemed to put much lower probabilities on AGI ever happening in the first place).
Of course the field is a bit odd, doesn't have a wide breadth of researchers, and there's a definite deformation professionelle. But that's not enough to change my risk assessment anywhere near to "not risky enough to bother about".
Of course the field is a bit odd, doesn't have a wide breadth of researchers, and there's a definite deformation professionelle. But that's not enough to change my risk assessment anywhere near to "not risky enough to bother about".
"risky enough to bother about" could be interpreted as:
(in ascending order of importance)
Someone should actively think about the issue in their spare time.
It wouldn't be a waste of money if someone was paid to think about the issue.
It would be good to have a periodic conference to evaluate the issue
One of the most annoying arguments when discussing AI is the perennial "But if the AI is so smart, why won't it figure out the right thing to do anyway?" It's often the ultimate curiosity stopper.
Nick Bostrom has defined the "Orthogonality thesis" as the principle that motivation and intelligence are essentially unrelated: superintelligences can have nearly any type of motivation (at least, nearly any utility function-bases motivation). We're trying to get some rigorous papers out so that when that question comes up, we can point people to standard, and published, arguments. Nick has had a paper accepted that points out the orthogonality thesis is compatible with a lot of philosophical positions that would seem to contradict it.
I'm hoping to complement this with a paper laying out the positive arguments in favour of the thesis. So I'm asking you for your strongest arguments for (or against) the orthogonality thesis. Think of trying to convince a conservative philosopher who's caught a bad case of moral realism - what would you say to them?
Many thanks! Karma and acknowledgements will shower on the best suggestions, and many puppies will be happy.