The September Open Thread, Part 2 has got nearly 800 posts, so let's have a little breathing room.
This thread is for the discussion of Less Wrong topics that have not appeared in recent posts. If a discussion gets unwieldy, celebrate by turning it into a top-level post.
It's interesting to look at people's arguments against paperclip maximizers. There seem to be two related categories that make up most of the objections:
People who can't imagine that a sufficiently intelligent being could be that different from us. One guy tried to claim that morality is universal so of course an artificial intelligence would share our values. Another said that a superintelligence would inevitably realize that its existence was pointless, as if there could possibly be some point other than maximizing the number of paperclips. Another claimed that morality is an "emergent phenomenon", but didn't explain what that actually meant, or how humanlike morality would emerge from a being whose only goal is paperclip maximization.
People who think of it as a dumb machine, more akin to a drill press than an alien god. Just put an off button on it! Or require that there be a human operator with an axe ready to cut the power lines to the computer.
What these objections both have in common is that they assume that the world consists of humanlike intelligences or dumb machines. It's unintuitive to imagine something that is both intelligent and profoundly alien.
I'd split the difference-- I don't think it's that hard to imagine an AI which has about as much loyalty to Ais as people have to people.
Really alien minds are naturally much harder to imagine. Clippy seems more like a damaged human than a thoroughly alien mind.
This may be a matter of assuming that minds would naturally have a complex mix of entangled goals, the way humans do. Even an FAI has two goals (Friendliness and increasing its intelligence) which may come into conflict.
Faint memory: an Alexis Gilliland cartoon of an automated bomber redirecting its target from a robot factory to a maternity ward.