The section you quote allows for the possibility that an AI could be given a "compulsion" to optimize for paperclips, which it would eventually shrug off, whereas I am confident that an AI could be given a utility function that would make it actually optimize for paperclips.
Okay; but the examples you gave seem to me to be more similar to compulsions than to utility functions. A person can care a lot about cars, and cars can become a major part of human society, but they're not the point of human society- if they stop serving their purposes they'll go the way of the horse and buggy. I'm not sure I can express the meaning I'm trying to convey cleanly using that terminology, so maybe I ought to restart.
My model of davidad's view is that part of general intelligence, as opposed to narrow intelligence, is varied and complex goals...
This post is shameless self-promotion, but I'm told that's probably okay in the Discussion section. For context, as some of you are aware, I'm aiming to model C. elegans based on systematic high-throughput experiments - that is, to upload a worm. I'm still working on course requirements and lab training at Harvard's Biophysics Ph.D. program, but this remains the plan for my thesis.
Last semester I gave this lecture to Marvin Minsky's AI class, because Marvin professes disdain for everything neuroscience, and I wanted to give his students—and him—a fair perspective of how basic neuroscience might be changing for the better, and seems a particularly exciting field to be in right about now. The lecture is about 22 minutes long, followed by over an hour of questions and answers, which cover a lot of the memespace that surrounds this concept. Afterward, several students reported to me that their understanding of neuroscience was transformed.
I only just now got to encoding and uploading this recording; I believe that many of the topics covered could be of interest to the LW community (especially those with a background in AI and an interest in brains), perhaps worthy of discussion, and I hope you agree.