I'd find a Friendliness proof fairly convincing if it was generated by an entity without strong incentives to let the AI out of the box (which, thanks to various possible incentives or blackmail schemes, is probably limited to me and some subset of people who haven't talked to the AI. Ideally the latter, since I might not recognize all my own biases.). If the entity in question is another AI, acausal trade issues unfortunately seem to arise if it's sufficiently similar to the first one, so the obvious hack of running two instances of a boxed AI and asking them about each other isn't going to be enough.
Purported Friendliness proofs coming from the AI itself... well, that's a trickier issue. Essentially it seems to come down to estimating the complexity of the simplest fake proof consistent with the AI's behavior and that of the simplest real proof, and comparing both of those against the purported proof. All else equal we should expect the former to be larger than the latter (being subject to more constraints), so if we can put even approximate bounds on its actual complexity we might be able to validate against that in a way that isn't easily subject to spoofing.
Alternately, if we have a good estimate of the AI's intelligence (bounded by the box's computational resources), it might be possible after repeated trials to say that the AI isn't smart enough to spoof its captors in sufficiently subtle and varied ways. This would only work for weakly superhuman AI, though; the probability of success on any given trial would be close to one for a really smart machine, so repeated trials wouldn't change our probability estimate much.
Some of you have expressed the opinion that the AI-Box Experiment doesn't seem so impossible after all. That's the spirit! Some of you even think you know how I did it.
There are folks aplenty who want to try being the Gatekeeper. You can even find people who sincerely believe that not even a transhuman AI could persuade them to let it out of the box, previous experiments notwithstanding. But finding anyone to play the AI - let alone anyone who thinks they can play the AI and win - is much harder.
Me, I'm out of the AI game, unless Larry Page wants to try it for a million dollars or something.
But if there's anyone out there who thinks they've got what it takes to be the AI, leave a comment. Likewise anyone who wants to play the Gatekeeper.
Matchmaking and arrangements are your responsibility.
Make sure you specify in advance the bet amount, and whether the bet will be asymmetrical. If you definitely intend to publish the transcript, make sure both parties know this. Please note any other departures from the suggested rules for our benefit.
I would ask that prospective Gatekeepers indicate whether they (1) believe that no human-level mind could persuade them to release it from the Box and (2) believe that not even a transhuman AI could persuade them to release it.
As a courtesy, please announce all Experiments before they are conducted, including the bet, so that we have some notion of the statistics even if some meetings fail to take place. Bear in mind that to properly puncture my mystique (you know you want to puncture it), it will help if the AI and Gatekeeper are both verifiably Real People<tm>.
"Good luck," he said impartially.