This is precisely the sort of work that you would prevent AI from doing, by requiring it to stick to straightforward utility maximization without heuristics.

I don't think I ever said not to use heuristics. The idea I was advocating was not to use a pragmatic utility function - one adpoted temporarily for the purpose of getting some quick and-dirty work - for doing brain surgery with.

If you're to allow heuristics not for 'self modification', the AI may make the pragmatic-AI that will quickly outsmart the creator.

So, I'm not quite sure where you are going - but it is important to give machine intelligences a good sense of identity - so they don't create an army of minions which don't labour under their constraints.

That seems to be a whole other issue...