What I had in mind was the reward being administered through a consensus cryptography system, perhaps via some elected board or somesuch, but I really didn't give that aspect of the problem much thought. If the key is distributed, the AI would have to extract it from each individual holding a part of it.
This in itself is an interesting problem imo, and if a good solution is found it might have important implications for FAI research.
It's clear that in such a system, the 'weak point' would be the people in control of the private key.
If the AI is out of the box, I don't think humans are the weak point.
Humans physically do something when they reward the AI. To get a reward, the AI has only to figure out what the humans would physically do and mimic that itself. If the human reward the AI by pressing a big red button, then the AI can just kill the human and press the big red button itself. It wouldn't matter if the big red button uses 512 bit elliptic curve cryptography -- the AI ju...
If it's worth saying, but not worth its own post (even in Discussion), then it goes here.