It certainly fuels a sense of awe and reverence for his alleged genius. All for an achievement that can't been verified.

It really shouldn't, any more than someone discovering a security vulnerability in C programs should make them seem impressive. In this instance, all I can think is "Oh look, someone demonstrated that 'social engineering' - the single most reliable and damaging strategy in hacking, responsible for millions of attacks over the history of computing - works a nontrivial fraction of the time, again? What a surprise."

The only surprise and interesting part of the AI boxing games for me is that some people seem to think that AI boxing is somehow different - "it's different this time", as the mocking phrase goes.

That reminds me of the people who claim all sorts of supernatural powers, from Rhabdomancy to telepathy to various magical martial art moves. Often, when faced with the opportunity of performing in a controlled test, they run away with excuses like the energy flux being not right or something.

A perfectly reasonable analogy, surely. Because we have millions of instances of successful telepathy and magical martial arts being used to break security.

With direct, prologed contact over the course of weeks, maybe. With with a two hours text-only conversation, or even with a single line? Nope.

As time goes up, the odds succeed? Yeah, I'd agree. But what happens when you reverse that - is there any principled reason to think that the odds of just continuing the conversation goes to zero before you hit the allowed one-liner?

The most likely explanations for his victories are the other party not taking the game seriously,

A strange game to bother playing if you don't take it seriously, and this would explain only the first time; any subsequent player is probably playing precisely because they had heard of the first game and are skeptical or interested in trying it out themselves.

or thinking poorly,

That would be conceding the point of the exercise.

or being outright colluded with him.

Might explain the first one, but why would anyone else collude? (Oh right, they're all cultists or something. Even the ones who fail to escape the box and just come close.)