Suppose you have a prior over all possible priors, and your first action after determining your utility function is to figure out which prior you should use. Before choosing a particular prior, you can define the expected utility of policies in terms of the "expected prior" of your distribution over priors. No matter how you arrived at your utility function, you will want to remember it as a linear combination of values while updating on the prior you chose.