Random thought: I've long known that police can often extract false confessions of crimes, but I only just now made the connection to the AI box experiment. In both cases you have someone being convinced to say or do something manifestly against their best interest. In fact, if anything I think the false confessions might be an even stronger result, just because of the larger incentives involved. People can literally be persuaded to choose to go to prison, just by some decidedly non-superhuman police officers. Granted, it's all done in person, so stronger psychological pressure can be applied. But still: a false confession! Of murder! Resulting in jail!

I think I have to revise downwards my estimate of how secure humans are.