Why we should fear the Paperclipper [Link]

XiXiDu

The Scenario

A programmer has constructed an artificial intelligence based on an architecture similar to Marcus Hutter's AIXI model (see below for a few details). This AI will maximize the reward given by a utility function the programmer has given it. Just as a test, he connects it to a 3D printer and sets the utility function to give reward proportional to the number of manufactured paper-clips.

At first nothing seems to happen: the AI zooms through various possibilities. It notices that smarter systems generally can make more paper-clips, so making itself smarter will likely increase the number of paper-clips that will eventually be made. It does so. It considers how it can make paper-clips using the 3D printer, estimating the number of possible paper-clips. It notes that if it could get more raw materials it could make more paper-clips. It hence figures out a plan to manufacture devices that will make it much smarter, prevent interference with its plan, and will turn all of Earth (and later the universe) into paper-clips. It does so.

Only paper-clips remain.

Link: aleph.se/andart/archives/2011/02/why_we_should_fear_the_paperclipper.html

The Scenario

A programmer has constructed an artificial intelligence based on an architecture similar to Marcus Hutter's AIXI model (see below for a few details). This AI will maximize the reward given by a utility function the programmer has given it. Just as a test, he connects it to a 3D printer and sets the utility function to give reward proportional to the number of manufactured paper-clips.

At first nothing seems to happen: the AI zooms through various possibilities. It notices that smarter systems generally can make more paper-clips, so making itself smarter will likely increase the number of paper-clips that will eventually be made. It does so. It considers how it can make paper-clips using the 3D printer, estimating the number of possible paper-clips. It notes that if it could get more raw materials it could make more paper-clips. It hence figures out a plan to manufacture devices that will make it much smarter, prevent interference with its plan, and will turn all of Earth (and later the universe) into paper-clips. It does so.

Only paper-clips remain.

Link: aleph.se/andart/archives/2011/02/why_we_should_fear_the_paperclipper.html

Agree with timtyler. AIXI doesn't care about paperclips, it cares about the input string of bits. Given enough smarts, it can just disconnect the input wire from the 3D printer and replace it with a tiny machine that always says "you win".

If anyone here knows how to make AIXI or any other type of AI care about the "actual" number of paperclips produced (the article uses this assumption a lot), rather than some pattern of input into the algorithm, then please enlighten me.

From what I understand about AIXI specifically, I agree, but it seems that the most horrible thing that can be built from the corpse of this idea would involve an AI architecture capable of evaluating its utility function on hypothetical states of the world it predicts, so it can compute and choose actions that maximise expected utility.

2nshepperd15y

Of course you can't maximise the expected value of U by maximising the expected value of F (which happens to at this point in time empirically correlate to U) instead. If the AI takes the action at t=0 which maximises the expected integral of Reward from t=0 to {far-future}, it can be accurately described as caring about (and only about) Reward. If you want to make it care about Utility you have to program it to take the action at t=0 which maximises the expected integral of Utility from t=0 to {far-future}. Maximising "Utility" or [the output of this box which implements Utility, I swear!] is not the same thing. Replace the quotation with the referent, etc. Kind of like optimizing for [happiness, freedom, love, complex fun...] instead of "morality" (because you know what a babyeater would reply if you ask what "right" is). Of course, I don't know how you'd actually program AIXI to do that, but it's what would have to be done...

1Perplexed15y

I can sketch a solution which I think would work: Instead of a simple agent which maximizes a utility function, you need a more complex agent which maximizes utility subject to a constraint. The constraint, in this case, is that the maximizer is not allowed to fiddle with the relation between 'reality' and the input stream. (The AI is forbidden to wear rose-colored glasses.) But the maximizer is permitted to fiddle with the relation between the output stream and 'reality'. (The AI is encouraged to design and build waldos.) So how do you get constraints into the general framework of optimization? Well, the theory of Lagrange multipliers is one well known technique. Another, which I think might work well, is to build the AI as containing multiple simple utility maximizing subagents, who have negotiated a Nash bargain that forces adherence to a fairness constraint. Many details to be worked out, but I think that this is the right general approach. Of course we know that a multi-agent coalition can never be as perfectly 'rational' as a simple single agent with a unified utility function. But I don't think that this lack of 'rationality' should frighten us. 'Rationality' as defined in economic decision theory is just a word attached to a set of axioms. If those axioms don't win, pick some different axioms.