Obvious problem 1: the video output or descriptions can contain basilisks, including ones that cause problem 2.
Obvious problem 2: Someone could ask and then verify it to make full UFAI without realizing it.
Obvious problem 3: UFAI could arise inside the simulation used to produce the hypotheticals, and either hack it's way out directly or cause problem 1 followed by problem 2.
And most oblivious problem of all: Withe being able to repeatedly do the highly dangerous and active step of modifying it's own source code, it'll never get smart enough to be useful on 95% of queries.
Fair point. in that case, given an unknown partially complete AI, if the first action you take is "Let me just start reading the contents of these files without running it to see what it even does." then someone could say "A UFAI put a basilisk in the source code and used it to kill all of humanity, you lose."
That isn't even entirely without precedent, using this as an example: http://boingboing.net/2012/07/10/dropped-infected-usb-in-the-co.html Sometimes malicious code really is literally left physically lying around, waiting for someone to pop it into a computer out of curiosity.
Supposing you have been recruited to be the main developer on an AI project. The previous developer died in a car crash and left behind an unfinished AI. It consists of:
A. A thoroughly documented scripting language specification that appears to be capable of representing any real-life program as a network diagram so long as you can provide the following:
A.1. A node within the network whose value you want to maximize or minimize.
A.2. Conversion modules that transform data about the real-world phenomena your network represents into a form that the program can read.
B. Source code from which a program can be compiled that will read scripts in the above language. The program outputs a set of values for each node that will optimize the output (you can optionally specify which nodes can and cannot be directly altered, and the granularity with which they can be altered).
It gives remarkably accurate answers for well-formulated questions. Where there is a theoretical limit to the accuracy of an answer to a particular type of question, its answer usually comes close to that limit, plus or minus some tiny rounding error.
Given that, what is the minimum set of additional features you believe would absolutely have to be implemented before this program can be enlisted to save the world and make everyone live happily forever? Try to be as specific as possible.