(I reject the notion that one can factorize intelligence from goals, so that one could take a superintelligence and fuse it with a goal to optimize for paperclips.
Why would you believe that? Evolution was more than capable of building an intelligence that optimized for whatever goals it needed, notable reproduction and personal survival. Granted its version was imperfect, since humans have enough conflicting goals that we can sometimes make moves that are objectively bad for the perpetuation of our gametes, not to mention the obvious failure cases like asexuals. That said, evolution has fat fingers. We can do better, and any AI's we build will be able to do even better.
I promise you that if the production of paperclips was a survival trait in the ancestral environment, above all else, we would all be paperclip maximizers. We would consider paperclips profound and important, and we would be loathe to remove the desire to make paperclips-- any more than we would be inclined now to pare out our own sex drive and self-preservation instinct.
EDIT: I do think the scenario of simply immediately turning everything into paperclips is naive. A superintelligence would have an enormous incentive to devote its resources to research and development for optimizing its goals as rapidly as possible, and would probably spend a lot of time simply thinking before actually embarking on a large-scale manufacture of paperclips. That's still not good for us, though, because even in that case, we're clearly a lot more useful to it as solid-state paperclip R&D labs than as human beings.
This post is shameless self-promotion, but I'm told that's probably okay in the Discussion section. For context, as some of you are aware, I'm aiming to model C. elegans based on systematic high-throughput experiments - that is, to upload a worm. I'm still working on course requirements and lab training at Harvard's Biophysics Ph.D. program, but this remains the plan for my thesis.
Last semester I gave this lecture to Marvin Minsky's AI class, because Marvin professes disdain for everything neuroscience, and I wanted to give his students—and him—a fair perspective of how basic neuroscience might be changing for the better, and seems a particularly exciting field to be in right about now. The lecture is about 22 minutes long, followed by over an hour of questions and answers, which cover a lot of the memespace that surrounds this concept. Afterward, several students reported to me that their understanding of neuroscience was transformed.
I only just now got to encoding and uploading this recording; I believe that many of the topics covered could be of interest to the LW community (especially those with a background in AI and an interest in brains), perhaps worthy of discussion, and I hope you agree.