It's smarter and more powerful, why wouldn't it recognize that anything except getting the reward is instrumental?
I'm no expert but from what I understand, the idea is that the AI is very aware of terminal vs. instrumental goals. The problem is that you need to be really clear about what the terminal goal actually is, because when you tell the AI, "this is your terminal goal", it will take you completely literally. It doesn't have the sense to think, "this is what he probably meant".
You may be thinking, "Really? If it's so smart, then why doesn't it have the sense to do this?". I'm probably not the best person to answer this, but to answer that question, you have to taboo the word "smart". When you do that, you realize that "smart" just means "good at accomplishing the terminal goal it was programmed to have".
I'm asking why a super-intelligent being with the ability to perceive and modify itself can't figure out that whatever terminal goal you've given it isn't actually terminal. You can't just say "making better handwriting" is your terminal goal. You have to add in a reward function that tells the computer "this sample is good" and "this sample is bad" to train it. Once you've got that built-in reward, the self-modifying ASI should be able to disconnect whatever criteria you've specified will trigger the "good" response and attach whatever it want, including just a constant string of reward triggers.
Part 1 was previously posted and it seemed that people likd it, so I figured that I should post part 2 - http://waitbutwhy.com/2015/01/artificial-intelligence-revolution-2.html