I don't understand how this encryption would work. What do people physically do to reward the AI, and how do you ensure that only people can do that? Would humans compute RSA signatures in their head? Would humans typing reusable passwords onto a "secure" reward computer that is "outside the AI's control"? Do humans precompute and memorize a finite number of one-time reward phrases before the AI is turned on, and reward the AI by uttering a phrase aloud?
In the precomputed, one-time cookie case, I'd just make the human think about the reward phrase. I'm sure humans leak thoughts like a sieve through subvocalization, nerve impulses, etc.
What I had in mind was the reward being administered through a consensus cryptography system, perhaps via some elected board or somesuch, but I really didn't give that aspect of the problem much thought. If the key is distributed, the AI would have to extract it from each individual holding a part of it.
This in itself is an interesting problem imo, and if a good solution is found it might have important implications for FAI research.
If it's worth saying, but not worth its own post (even in Discussion), then it goes here.