Comment on Superintelligence 9: The orthogonality of intelligence and goals

Tue, 11 Nov 2014 05:59:33 GMT

It seems to me that at least the set of possible goals is correlated with intelligence -- the higher it is, the larger the set. This is easier to see looking down rather than up: humans are more intelligent than, say, cows, and humans can have goals which a cow cannot even conceive of. In the same way a superintelligence is likely to have goals which we cannot fathom.

From certain points of view, we are "simple agents". I have doubts that goals of a superintelligence are predictable by us.

Discuss