I'm sorry, I must have misunderstood your initial proposal. I thought you were specifying an additional component - after it has achieved its maximum utility, the additional component steps in and shuts down the entity.
Rather, you were saying: If the AI achieves the goal, it will want nothing further, and therefore automatically act as if it were shut down. Presumably if we take this as given, the negative consequences would have to be while accomplishing the "fairly-easy" goal.
I am merely trying to create amusing or interesting science fiction "poetic justice" scenarios, similar to Dresden Codak's "caveman science fiction". I am not trying to create serious arguments, and I don't want to try to be serious on this subject.
Rather, you were saying: If the AI achieves the goal, it will want nothing further, and therefore automatically act as if it were shut down.
If you don't provide an explicit shutdown goal (as Dorikka did have in mind), then you get into a situation where all remaining potential utility gains come from skeptical scenarios where the upper bound hasn't actually been achieved, so the AI devotes all available resources to making ever more sure that there are no Cartesian demons deceiving it. (Also, depending on its implicit ontology, maybe to making sure time travelers can't undo its success, or other things like that.)
At the recent London meet-up someone (I'm afraid I can't remember who) suggested that one might be able to solve the Friendly AI problem by building an AI whose concerns are limited to some small geographical area, and which doesn't give two hoots about what happens outside that area. Cipergoth pointed out that this would probably result in the AI converting the rest of the universe into a factory to make its small area more awesome. In the process, he mentioned that you can make a "fun game" out of figuring out ways in which proposed utility functions for Friendly AIs can go horribly wrong. I propose that we play.
Here's the game: reply to this post with proposed utility functions, stated as formally or, at least, as accurately as you can manage; follow-up comments explain why a super-human intelligence built with that particular utility function would do things that turn out to be hideously undesirable.
There are three reasons I suggest playing this game. In descending order of importance, they are: