Nope. An AI foreman has been programmed before I tell him to handle paperclip production.
From the text:
If I owned a paperclip factory, and casually programmed my superintelligent AI to improve efficiency while I'm away
If you program it first, then a lot depends on the subtleties. If you tell it to wait a minute and record everything you say, then interpret that and set it to its utility function, you're effectively putting the finishing touches on programming. If you program it to assign utility to fulfilling commands you give it, you've already doomed the world before you even said anything. It will use all the resources at its disposal to make sure you say things that have already been done as rapidly as possible.
At the moment AIs are not built at all -- in any way or in no way.
Hence the
or more precisely, since you can't build an AI exactly the same as a human, building an AI that way has serious danger of having it evolve very inhuman goals.
and casually programmed my superintelligent AI
The programming I'm talking about is not this (which is "telling"). The programming I'm talking about is the one which converts some hardware and a bunch of bits into a superintelligent AI.
...since you can't build an AI exactly the same as a human, building an AI that way...
Huh? In any case, AIs self-develop and evolve. You might start with an AI that has an agreeable set of goals. There is no guarantee (I think, other people seem to disagree) that these goals will be the same after some time.
A stub on a point that's come up recently.
If I owned a paperclip factory, and casually told my foreman to improve efficiency while I'm away, and he planned a takeover of the country, aiming to devote its entire economy to paperclip manufacturing (apart from the armament factories he needed to invade neighbouring countries and steal their iron mines)... then I'd conclude that my foreman was an idiot (or being wilfully idiotic). He obviously had no idea what I meant. And if he misunderstood me so egregiously, he's certainly not a threat: he's unlikely to reason his way out of a paper bag, let alone to any position of power.
If I owned a paperclip factory, and casually programmed my superintelligent AI to improve efficiency while I'm away, and it planned a takeover of the country... then I can't conclude that the AI is an idiot. It is following its programming. Unlike a human that behaved the same way, it probably knows exactly what I meant to program in. It just doesn't care: it follows its programming, not its knowledge about what its programming is "meant" to be (unless we've successfully programmed in "do what I mean", which is basically the whole of the challenge). We can't therefore conclude that it's incompetent, unable to understand human reasoning, or likely to fail.
We can't reason by analogy with humans. When AIs behave like idiot savants with respect to their motivations, we can't deduce that they're idiots.