This is a linkpost for http://www.newsweek.com/nick-bostrom-google-winning-artificial-intelligence-arms-race-red-button-506624
I did not say paperclippers are impossible in principle. I stated earlier that the orthogonality thesis may be true in principle, but it is false in practice. As you said, AIXI-tl is very inefficient. Practical AIs will not be like that, and they will not be limited to one rigid goal like that.
And even if you find my theory of intelligence unconvincing, one that implies that evolution is intelligent is even less convincing, since it does not respect what people actually mean by the word.
" Because if it's correct, it means AI researchers just need to figure out that one idea, to suddenly make intelligent AIs." That would be true, if it were easy to program that kind of generalization. Currently that seems to be very difficult, and as you correctly say, no one knows how to do it.
AIXI-tl is very inefficient. Practical AIs will not be like that, and they will not be limited to one rigid goal like that.
Your second claim doesn't follow from the first. Practical AIs will of course be different. But the basic structure of AIXI, reinforcement learning, is agnostic to the model used. It just requires some algorithm to do learning/prediction. As prediction algorithms get better and better, they will still suffer the same problems as AIXI. Unless you are proposing some totally different model of AI than reinforcement learning, that someh...