Okay; but the examples you gave seem to me to be more similar to compulsions than to utility functions. A person can care a lot about cars, and cars can become a major part of human society, but they're not the point of human society- if they stop serving their purposes they'll go the way of the horse and buggy. I'm not sure I can express the meaning I'm trying to convey cleanly using that terminology, so maybe I ought to restart.
My model of davidad's view is that part of general intelligence, as opposed to narrow intelligence, is varied and complex goals. We could make a narrow AI which only cared about the number of paperclips in the universe, but in order to make an intelligence that's general we need to make it also care about the future, planning, existential risk, and so on.
And so you might get a vibrant interstellar civilization of synthetic intelligences- that happens to worship paperclips, and uses them for currency and religious purposes- rather than a dead world with nothing but peculiarly bent metal.
but the examples you gave seem to me to be more similar to compulsions than to utility functions
I would have liked to use examples of plugging in clearly terminal values to a general goal achieving system. But the only current or historical general goal achieving systems are humans, and it is notoriously difficult to figure out what humans' terminal values are.
...My model of davidad's view is that part of general intelligence, as opposed to narrow intelligence, is varied and complex goals. We could make a narrow AI which only cared about the number of pa
This post is shameless self-promotion, but I'm told that's probably okay in the Discussion section. For context, as some of you are aware, I'm aiming to model C. elegans based on systematic high-throughput experiments - that is, to upload a worm. I'm still working on course requirements and lab training at Harvard's Biophysics Ph.D. program, but this remains the plan for my thesis.
Last semester I gave this lecture to Marvin Minsky's AI class, because Marvin professes disdain for everything neuroscience, and I wanted to give his students—and him—a fair perspective of how basic neuroscience might be changing for the better, and seems a particularly exciting field to be in right about now. The lecture is about 22 minutes long, followed by over an hour of questions and answers, which cover a lot of the memespace that surrounds this concept. Afterward, several students reported to me that their understanding of neuroscience was transformed.
I only just now got to encoding and uploading this recording; I believe that many of the topics covered could be of interest to the LW community (especially those with a background in AI and an interest in brains), perhaps worthy of discussion, and I hope you agree.