whatever terminal goal you've given it isn't actually terminal.
This is a contradiction in terms.
If you have given it a terminal goal, that goal is now a terminal goal for the AI.
You may not have intended it to be a terminal goal for the AI, but the AI cares about that less than it does about its terminal goal. Because it's a terminal goal.
If the AI could realize that its terminal goal wasn't actually a terminal goal, all it'd mean would be that you failed to make it a terminal goal for the AI.
And yeah, reinforcement based AIs have flexible goals. That doesn't mean they have flexible terminal goals, but that they have a single terminal goal, that being "maximize reward". A reinforcement AI changing its terminal goal would be like a reinforcement AI learning to seek out the absence of reward.
whatever terminal goal you've given it isn't actually terminal.
This is a contradiction in terms.
I should have said something more like "whatever seemingly terminal goal you've given it isn't actually terminal."
Part 1 was previously posted and it seemed that people likd it, so I figured that I should post part 2 - http://waitbutwhy.com/2015/01/artificial-intelligence-revolution-2.html