That's the point, that it would almost never change it's underlying utility function. Once we have a provably friendly FAI, we wouldn't want it to change the part that makes its friendly.

Now, it could still change how it goes about achieving it's utility function, as long as that helps it get more utility, so it would still be self-modifying.

There is a chance that it could change (E.g. if you were naturally a 2-boxer on Newcomb's Problem, you might self-modify to do a one-boxer). But, those cases are rare.