Can you give me some references for the idea that "you don't need to have solved the AGI problem to have solved friendliness"? I'm not saying it's not true, I just want to improve this article.
Let's taboo "solved" for a minute.
Say you have a detailed, rigorous theory of Friendliness, but you don't have it implemented in code as part of an AGI. You are racing with your competitor to code a self-improving super-AGI. Isn't it still quicker to implement something that doesn't incorporate Friendliness?
To me, it seems like, even if the theory was settled, Friendliness would be an additional feature you would have to code into an AI that would take extra time and effort.
What I'm getting at is that, throughout the history of computing, the version of a system with desirable property X, even if the theoretical benefits of X are well known by the academy, has tended to be implemented and deployed commercially after the version without X. For example, it would have been better for the general public and web developers if web browsers obeyed W3C specifications and didn't have any extra proprietary tags - but in practice, commercial pressures meant that companies made grossly non-compliant browsers for years until eventually they started moving towards compliance.
The "Friendly browser" theory was solved, but compliant and non-compliant browsers still weren't on basically equal footing.
(Now, you might say that CEV will be way more mathematical and rigorous than browser specifications - but the only important point for my argument is that it will take more effort to implement than the alternative).
Now you could say that browser compliance is a fairly trvial matter, and corporations will be more cautious about deploying AGI. But the potential gain from deploying a super-AI first would surely be much greater than the benefit of supporting the blink tag or whatever - so the incentive to rationalise away the perceived dangers will be much greater.
If you have a rigorous, detailed theory of Friendliness, you presumably also know that creating an Unfriendly AI is suicide and won't do it. If one competitor in the race doesn't have the Friendliness theory or the understanding of why it's important, that's a serious problem, but I don't see any programmer who understands Friendliness deliberately leaving it out.
Also, what little I know about browser design suggests that, say, supporting the blink tag is an extra chunk of code that gets added on later, possibly with a few deeper changes to existing code....
Here's why I'm not going to give money to the SIAI any time soon.
Let's suppose that Friendly AI is possible. In other words, it's possible that a small subset of humans can make a superhuman AI which uses something like Coherent Extrapolated Volition to increase the happiness of humans in general (without resorting to skeevy hacks like releasing an orgasm virus).
Now, the extrapolated volition of all humans is probably a tricky thing to determine. I don't want to get sidetracked into writing about my relationship history, but sometimes I feel like it's hard to extrapolate the volition of one human.
If it's possible to make a Friendly superhuman AI that optimises CEV, then it's surely way easier to make an unFriendly superhuman AI that optimises a much simpler variable, like the share price of IBM.
Long before a Friendly AI is developed, some research team is going to be in a position to deploy an unFriendly AI that tries to maximise the personal wealth of the researchers, or the share price of the corporation that employs them, or pursues some other goal that the rest of humanity might not like.
And who's going to stop that happening? If the executives of Corporation X are in a position to unleash an AI with a monomaniacal dedication to maximising the Corp's shareholder value, it's probably illegal for them not to do just that.
If you genuinely believe that superhuman AI is possible, it seems to me that, as well as sponsoring efforts to design Friendly AI, you need to (a) lobby against AI research by any groups who aren't 100% committed to Friendly AI (pay off reactionary politicians so AI regulation becomes a campaign issue, etc.) (b) assassinate any researchers who look like they're on track to deploying an unFriendly AI, then destroy their labs and backups.
But SIAI seems to be fixated on design at the expense of the other, equally important priorities. I'm not saying I expect SIAI to pursue illegal goals openly, but there is such a thing as a false-flag operation.
While Michelle Bachmann isn't talking about how AI research is a threat to the US constitution, and Ben Goertzel remains free and alive, I can't take the SIAI seriously.