Comment on MIRI's Approach

Fri, 31 Jul 2015 02:00:36 GMT

One problem is what Bostrom would call "the treacherous turn." When the AGI is dependent on us, satisfying us is a very good idea for it - if it's unsatisfactory it will be deleted. Behaving nicely is so good an idea that many different goal systems will independently choose this strategy. And so the fact that an AGI appears nice is only weak statistical evidence that it would be nice if it wasn't dependent on us, and further trials are not independent and so don't accumulate well. This type of problem appears when the AGI develops good enough long-term planning, and has information about its creators.

Another problem is the problem of expanding action spaces. Consider an AGI that wants to gather lots of stamps (example shamelessly stolen from Computerphile video). When the AGI is childlike, its effective action space only looks like spending money to purchase stamps. As it becomes as smart as a human its actions expand - now it might perform a job to make money to buy stamps, or try to steal money to buy stamps, or purchase a printing press to make its own stamps, or all the sorts of things you might do if you really wanted stamps. Then, as it becomes superintelligent, the stamp-gathering robot will proceed to take over the world and try to terraform the entire earth into stamps. This is a problem for using experimental evidence because as the set of actions expands, so do the possible preferences over actions. Which means that there are many possible sets of preferences that might lead to altruistic behavior among weak AIs; there's some un-eliminable error when trying to predict "many-options" behavior just from "few-options" behavior.

Discuss