This seems like the well-worn discussion on whether rational agents should be expected to change their preferences. Here's Omohundro on the topic:

"Their utility function will be precious to these systems. It encapsulates their values and any changes to it would be disastrous to them. If a malicious external agent were able to make modifications, their future selves would forevermore act in ways contrary to their current values. This could be a fate worse than death! Imagine a book loving agent whose utility function was changed by an arsonist to cause the agent to enjoy burning books. Its future self not only wouldn’t work to collect and preserve books, but would actively go about destroying them. This kind of outcome has such a negative utility that systems will go to great lengths to protect their utility functions."

• http://selfawaresystems.files.wordpress.com/2008/01/ai_drives_final.pdf

He goes on to discuss the issue in detail and lists some exceptional cases.