In the latest William Gibson novel Agency, a bunch of human characters are constantly told exactly what to do by a superhuman AI in order to have everything turn out for the best. They accept the notion that this character, who they only talk to over their phones, is an AI, when it seems like it could just as easily be a fictional construct created by a person or organization. There's one part of the book where a character is convinced off-page that the AI is a superhuman AI by talking to it for a bit, although we're not privy to the conversation.
What I'm wondering is, how could a superhuman AI best prove that it was a superhuman AI through chat? Is there a more or less definitive way to do it? Is there a fast way to do it?
I remembered after writing this that there are also scenes where the AI provides people with schematics for advanced technology in return for labor; in these situations, it isn't trying to prove that it's an advanced AI though, and even if it were trying to use this as a proof there's no way to prove that the technologies it provides weren't actually developed by humans, even if they were advanced.
Some of these things are falsifiable, for example people have already made programs that can detect letters from acoustics, and trivia questions would similarly be pretty easy, but the idea of asking a question that requires complicated processing to find the answer and requiring a superhumanly fast response seems like it might be the best. The trick would be that it would have to be a question that couldn't have been answered in advance (question can't be predictable, so that a person or organization could have prepared answers).
So it sounds like you'd need to be able to generate a question that 1. requires human-level or above reasoning, including generalization 2. is too complex for a human to quickly answer 3. cannot be predicted in form or content to an extent that would allow its answer to be cached in advance