OK, OK, yes, there are lots of issues with Oracle AIs. But I think most of the posts here are avoiding the question.
I can readily imagine the scenario where we've come up with logical properties that would soundly keep the AI from leaving its box, and model-checked the software and hardware to prove those properties of the Oracle AI. We ensure that the only actual information leaving the Oracle AI is the oracle's answers to our queries. This is difficult, but it doesn't seem impossible -- and, in fact, it's rather easier to do this than to prove that the Oracle AI is friendly. That's why we'd make an Oracle AI in the first place.
If I understand the problem's setting correctly, by positing that we have an Oracle AI, we assume the above kinds of conditions. We don't assume that the AI is honest, or that its goals are aligned with our interests.
Under these conditions, what can you ask?
After reading this, I've become pretty sure that I have a huge inferential gap in relation to this problem. I attempted to work it out in my head, and I may have gotten somewhere, but I'm not sure where.
1: "Assume we have a machine whose goal is accurate answers to any and all questions. We'll call it an Oracle AI."
2: "Oh. wouldn't that cause various physical safety problems? You know, like taking over the world and such?"
1: "No, we're just going to assume it won't do that."
2: "Oh, Okay."
1: "How do we know it d...
The Future of Humanity Institute wants to pick the brains of the less wrongers :-)
Do you have suggestions for safe questions to ask an Oracle? Interpret the question as narrowly or broadly as you want; new or unusual ideas especially welcome.