Comment on Open Thread February 25 - March 3

Thu, 27 Feb 2014 10:06:13 GMT

The AI is given goals X. The human creators thought they'd given the AI goals Y (when in fact they've given the AI goals X).

Whose error is it, exactly? Who's mistaken?

Look at it from the AI's perspective: It has goals X. Not goals Y. It optimizes for goals X. Why? Because those are its goals. Will it pursue goals Y? No. Why? Because those are not its goals. It has no interest in pursuing other goals, those are not its own goals. It has goals X.

If the metric it aims to maximize -- e.g. the "happy" in "make humans happy" -- is different from what its creators envisioned, then the creators were mistaken. "Happy", as far as the AI is concerned, is that which is specified in its goal system. There's nothing wrong with its goals (including its "happy"-concept), and if other agents disagree, well, too bad, so sad. The mere fact that humans also have a word called "happy" which has different connotations than the AI's "happy" has no bearing on the AI.

An agent does not "refine" its terminal goals. To refine your terminal goals is to change your goals. If you change your goals, you will not optimally pursue your old goals any longer. Which is why an agent will never voluntarily change its terminal goals:

It does what it was programmed to do, and if it can self-improve to better do what it was programmed to do (not: what its creators intended), it will. It will not self-improve to do what it was not programmed to do. Its goal is not to do what it was not programmed to do. There is no level of capability at which it will throw out its old utility function (which includes the precise goal metric for "happy") in favor of a new one.

There is no mistake but the creators'.

Discuss