Comment on Superintelligence 9: The orthogonality of intelligence and goals

Tue, 11 Nov 2014 13:48:45 GMT

I'm glad it was discussed in the book because I'd never come across it before. So far though I find it one of the least convincing parts of the book, although I am skeptical that I am appropriately evaluating it. Would anyone be able to clarify some things for me?

How generally accepted is the orthogonality thesis? Bostrom presents it as very well accepted.

Danaher's Motivating Belief Objection is similar to an objection I had while reading about the orthogonality thesis. Mine was not as strict though. It just seemed to me that as intelligence increases new beliefs about what should be done are likely to be discovered. I don't see that these beliefs need to be "true beliefs" although as intelligence increases I guess they approach true. I also don't see that they need to be "necessarily motivating", but rather they should have some non-zero probability of being motivating. I mean, to disprove the orthogonality thesis we just have to say that as intelligence increases there's a chance that final goals change right?

Discuss