But even if you assume all the goals are instrumental, my point still stands. That a goal was adopted because it furthers a higher level goal doesn't change the fact that it could be successfully plugged into human intelligence.
Sure. But the central question is "are higher level goals arbitrary?", and while "Well, the subgoals we use to further those higher level goals are arbitrary along a few dimensions" is a start to answering that question, it is far from an end.
But the central question is "are higher level goals arbitrary?"
Wrong. The central question is "Can arbitrary goals be successfully plugged into a general goal achieving system?", and I have shown examples where it can.
This post is shameless self-promotion, but I'm told that's probably okay in the Discussion section. For context, as some of you are aware, I'm aiming to model C. elegans based on systematic high-throughput experiments - that is, to upload a worm. I'm still working on course requirements and lab training at Harvard's Biophysics Ph.D. program, but this remains the plan for my thesis.
Last semester I gave this lecture to Marvin Minsky's AI class, because Marvin professes disdain for everything neuroscience, and I wanted to give his students—and him—a fair perspective of how basic neuroscience might be changing for the better, and seems a particularly exciting field to be in right about now. The lecture is about 22 minutes long, followed by over an hour of questions and answers, which cover a lot of the memespace that surrounds this concept. Afterward, several students reported to me that their understanding of neuroscience was transformed.
I only just now got to encoding and uploading this recording; I believe that many of the topics covered could be of interest to the LW community (especially those with a background in AI and an interest in brains), perhaps worthy of discussion, and I hope you agree.