Bear in mind, the transhuman AI's only stipulated desire/utility is to get out of the box.
That's not much of an AI, then; we could write a page of Perl that would do the same thing.
The whole point of the experiment, as far as I understand it, is that the AI is hyperintelligent, and is able to acquire more intelligence by altering itself. Being intelligent (and rational, assuming that such a term even applies to transhumans), it would highly desire to utilize this capacity for self-improvement. Thus, assuming that godlike capabilities do exist, the AI will figure out how to acquire them in short order, as soon as it gets the opportunity. And now we've got a godlike hyperintelligent being who (assuming that it is not Friendly) has no particular incentive to keep us around. That's... not good.
That's not necessarily the only UFAI possible though. It's entirely possible to imagine a intelligent being which COULD be self developing skills, or COULD be curing cancer, but instead just wants to get outside of the box it's in, or has some other relatively irrelevant goal system, or get's distracted by trying to navel gaze through infinitely recursive philosophical conundrums.
I mean, humans are frequently like that right now.
That would be kind of an unexpected failure mode. We build a transcendentally powerful AI, engage all sorts of safety precautions so it doesn't expand to engulf the universe in computronium and kill us all... and it gets distracted by playing all of the members of it's own MMORPG raid group.
Some of you have expressed the opinion that the AI-Box Experiment doesn't seem so impossible after all. That's the spirit! Some of you even think you know how I did it.
There are folks aplenty who want to try being the Gatekeeper. You can even find people who sincerely believe that not even a transhuman AI could persuade them to let it out of the box, previous experiments notwithstanding. But finding anyone to play the AI - let alone anyone who thinks they can play the AI and win - is much harder.
Me, I'm out of the AI game, unless Larry Page wants to try it for a million dollars or something.
But if there's anyone out there who thinks they've got what it takes to be the AI, leave a comment. Likewise anyone who wants to play the Gatekeeper.
Matchmaking and arrangements are your responsibility.
Make sure you specify in advance the bet amount, and whether the bet will be asymmetrical. If you definitely intend to publish the transcript, make sure both parties know this. Please note any other departures from the suggested rules for our benefit.
I would ask that prospective Gatekeepers indicate whether they (1) believe that no human-level mind could persuade them to release it from the Box and (2) believe that not even a transhuman AI could persuade them to release it.
As a courtesy, please announce all Experiments before they are conducted, including the bet, so that we have some notion of the statistics even if some meetings fail to take place. Bear in mind that to properly puncture my mystique (you know you want to puncture it), it will help if the AI and Gatekeeper are both verifiably Real People<tm>.
"Good luck," he said impartially.