Security is solving the problem after the fact, and I think that is totally the wrong approach here, we should be asking if something can be designed into the AI that prevents people from wanting to take the AI over or prevents takeovers from being disastrous (three suggestions for that are included in this comment).
Perhaps the best approach to security is to solve the problems humans are having that cause them to commit crimes. Of course this appears to be a chicken-or-egg proposition "Well, the AI can't solve the problems until it's securely built, and it won't be securely built until it's solved the problems." but we could look at this problem in several other ways. Each of these are with the assumption that it is safest to deem the security problem impossible to solve after the fact:
Make a lesser AI that is just powerful enough to address the root reasons humans cause security issues - the fact that it is less powerful may also make it a lot less dangerous in the event that it's taken over.
Give everyone access to AI. If everyone already has access, then there will be no big jackpot to steal. If everyone else has access to AI, too, then taking one over wouldn't buy you any power. This is a paradigm shift from what you guys are suggesting, and would require that much of this is gone about in a different way. At first it looks like this would exacerbate security problems - won't people use them for bad purposes? Of course. But if EVERYONE has access, that won't be any scarier than a situation in which a gunman wants to take over, and everyone else has a gun. You can't take over a room group of equally armed people. Imagine taking out a gun in a room full of people with guns. You'll end up getting shot - you won't be taking over. One AI can't take over a world full of AI. Distributing the power everywhere would prevent many security problems ranging from deterring criminals from committing crimes (and limiting the damage criminals can do) to checking and balancing big powers to prevent tyranny to preventing the corruption of good people that may otherwise happen if they're given too much power to preventing a situation similar to Zimbardo's prison experiment where powerful AI builders feel a need to defend themselves from hackers, criminals and tyrants and therefore begin using the power they have in an oppressive way. {reason four}
Focus on enlightenment. The human race has far too much power and far too little wisdom. Increasing the power of toddlers won't solve toddler's issues but only increase the damage the toddlers do to one another because of their issues. And so, increasing the power of an unenlightened human race will only amplify their pain. If we all practice non-attachment, we will be strong enough to heed "those who sacrifice freedom for security deserve neither" and hopefully cease the addiction to violating freedom, either by taking advantage of others or sacrificing our own, in order to gain more security. Might it make more sense to focus on enlightenment than on throwing power at an unenlightened species in an attempt to solve their problems? Maybe we should be asking "How can technology help us become more enlightened?"
"Intelligent people solve problems, geniuses prevent them." - Einstein
I think AI builders need to do like the geniuses. Prevent people from WANTING to take over the AI or prevent a takeover from being a big deal. There is nothing you can do guarantee it won't be taken over, not with the level of certainty that we'd need to have for a problem that could be this devastating to the entire world. The only way this much power can exist, and be safe from humans, is to address the human element itself.
-- Nick Szabo
Nick Szabo and I have very similar backrounds and interests. We both majored in computer science at the University of Washington. We're both very interested in economics and security. We came up with similar ideas about digital money. So why don't I advocate working on security problems while ignoring AGI, goals and Friendliness?
In fact, I once did think that working on security was the best way to push the future towards a positive Singularity and away from a negative one. I started working on my Crypto++ Library shortly after reading Vernor Vinge's A Fire Upon the Deep. I believe it was the first general purpose open source cryptography library, and it's still one of the most popular. (Studying cryptography led me to become involved in the Cypherpunks community with its emphasis on privacy and freedom from government intrusion, but a major reason for me to become interested in cryptography in the first place was a desire to help increase security against future entities similar to the Blight described in Vinge's novel.)
I've since changed my mind, for two reasons.
1. The economics of security seems very unfavorable to the defense, in every field except cryptography.
Studying cryptography gave me hope that improving security could make a difference. But in every other security field, both physical and virtual, little progress is apparent, certainly not enough that humans might hope to defend their property rights against smarter intelligences. Achieving "security against malware as strong as we can achieve for symmetric key cryptography" seems quite hopeless in particular. Nick links above to a 2004 technical report titled "Polaris: Virus Safe Computing for Windows XP", which is strange considering that it's now 2012 and malware have little trouble with the latest operating systems and their defenses. Also striking to me has been the fact that even dedicated security software like OpenSSH and OpenSSL have had design and coding flaws that introduced security holes to the systems that run them.
One way to think about Friendly AI is that it's an offensive approach to the problem of security (i.e., take over the world), instead of a defensive one.
2. Solving the problem of security at a sufficient level of generality requires understanding goals, and is essentially equivalent to solving Friendliness.
What does it mean to have "secure property rights", anyway? If I build an impregnable fortress around me, but an Unfriendly AI causes me to give up my goals in favor of its own by crafting a philosophical argument that is extremely convincing to me but wrong (or more generally, subverts my motivational system in some way), have I retained my "property rights"? What if it does the same to one of my robot servants, so that it subtly starts serving the UFAI's interests while thinking it's still serving mine? How does one define whether a human or an AI has been "subverted" or is "secure", without reference to its "goals"? It became apparent to me that fully solving security is not very different from solving Friendliness.
I would be very interested to know what Nick (and others taking a similar position) thinks after reading the above, or if they've already had similar thoughts but still came to their current conclusions.