Comment on Superintelligence 9: The orthogonality of intelligence and goals

Tue, 11 Nov 2014 17:01:26 GMT

I suspect that it hides more assumptions about the nature of intelligence than we can necessarily make at this time.

At the present moment, we are the only general intelligences around, and we don't seem to have terminal goals as such. As biological bodies, we are constrained by evolutionary processes, and there are many ways in which human behavior actually is reducible to offspring maximization (social status games, etc.). But it doesn't appear to be a 'utility function', so much as a series of strong tendencies in the face of specific stimuli. Using novel approaches like superstimuli, it's just as easy to make an impulse's reproductive utility drop sharply. So we have habits constrained by evolutionary forces, but not algorithmic utility in the paper clipper sense.

There is no such thing as a general intelligence with a 'goal' (as Bostrom defines it). There may be at some point, but it's not real yet. And we do have non-general intelligences with goals, that's an easy weekend coding project. But before we declare that a GI could accept any goal regardless of its strength, we should at least check to make sure that a GI can have a goal at all.

Discuss