For a UFAI:
Find me the shortest program that can
- Find a proof of whether P=NP, given only mathematical knowledge available on Wikipedia.
- Create translations of these English novels, science textbooks, and postmodern literary journal articles into Japanese, with each translation receiving a METEOR score greater than some human expert's.
- Win at least 9 out of 10 games of Starcraft II against a single "Insane"-level AI opponent.
This is actually an undecidable question. If you say find me the shortest program that does 'x' for sufficiently complex x, there will be shorter programs that output nothing forever, but which cannot be proved to halt, due to the halting problem. This can be fixed by imposing resource constraints on the program or saying "make it as short as you can", if the AI understands such things. Presumably, if you input this request as stated, the AI would tell you it could not solve it and nothing more, so other posters should keep this problem in mind.
Isaac Asimov once described a future in which all technical thought was automated, and the role of humans was reduced to finding appropriate questions to pose to thinking machines. I wouldn't suggest planning for this eventuality, but it struck me as an interesting situation. What would we do, if we could get the answer to any question we could formulate precisely? (In the story, questions didn't need to be formulated precisely, but nevermind.) For concreteness, suppose that we have a box as smart as a million Einsteins, cooperating effectively for a century every time we ask a question, but which is capable only of solving precisely specified problems.
You can't say "analyze the result of this experiment." You can say, "find me the setting for these 10 parameters which best explains this data" or "write me a short program which predicts this data." You can't say "find me a program that plays Go well." You can say, "find me a program that beats this particular Go ai, even with a 9 stone handicap." Etc. More formally, lets say you can specify any scoring program and ask the box to find an input that scores as well as it can.
What would you do, if you got exactly one question? I don't think humanity is posed to get any earth-shattering insights. I don't think we could find a theory of everything, or a friendly AI, or any sort of AI at all, or a solution to any real problem facing us, using just one question. But maybe that is just a failure of my creativity.
What would you plan to do, if you had unlimited access? An AGI or brain emulation arguably implicitly converts our vague real world objectives into a precise form. Are there other ways to bridge the gap between what humans can formally describe and what humans want? Can you bootstrap your way there starting from current understanding? What is any reasonable first step?