Okay, here we go. I've possibly reinvented the wheel here, but maybe I've come up with a simple, original result. That'd be cool. Or I'm interestingly wrong.

We wish to show that superlinear utility-of-belief functions, or equivalently ones that would cause an agent to prefer ignorance, lead to inconsistency.

Suppose Joe equally wants to believe each of two propositions, P and Q, to be true, with U(x) > x*U(1) for all probabilities x, and U(x) strictly increasing with x. Without loss of generality, we set U(0) to 0 and U(1) to 1. Both propositions concern events that will invisibly occur at some known future time.

Joe anticipates that he will eventually be given the following choice, which will completely determine P and Q:

Option 1: P xor Q. Joe won't know which one is true, so he believes each of them is true with probability 1/2. So he has U(1/2)+U(1/2)=2*U(1/2) utility. By assumption this is greater than 1. So let 2*U(1/2) - 1 = k.

Option 2: One proposition will become definitely true. The other will become true with probability p, where p is chosen to be greater than 0 but less than U-inverse(k). Joe will know which proposition is which. Joe's utility would be less than U(1) + U(U-inverse(k)), or less than 1 + 2*U(1/2) - 1, or less than 2*U(1/2).

Joe prefers Option 1. Therefore he anticipates that he will choose Option 1. Therefore, his current utility is 2*U(1/2). But what if he anticipated that he would choose Option 2? Then his current utility would be 2*U(1/2+p/2). So he wishes his k were smaller than U-inverse(k), meaning he wishes his U(x) were closer to x*U(1). If he were to modify his utility function such that U'(x) = x*U(1) for all x, the new Joe would not regret this decision since it strictly increases his expected utility under the new function.

Thus we can say that all superlinear utility functions are inherently unstable, in that an agent with U(x) > x*U(1) for all probabilities x, and U(x) strictly increasing with x, may increase its expected U by modifying to U'(x) = x*U(1) for all x.

The strongest possible constraint we can give for inherent stability of a utility-of-belief function is that, with utility-of-belief function U, an agent can never improve its U-utility by switching to any other utility function, except under cases wherein it anticipates being modeled by an outside entity. If we removed this exception, no non-degenerate utility-of-belief function could be called stable because we could always posit an outside entity that punishes agents modeled to have specific utility functions. The linear utility of belief function satisfies this condition, since it behaves identically whether it is maximizing the probability of P or its U(p(P)), so it always anticipates itself maximizing its own utility function. We have just shown that no superlinear function satisfies this constraint.

But by conservation of expected evidence, no agent with a linear or sublinear utility-of-belief function can increase its expected utility-of-belief by hiding evidence from itself.

Therefore, a rational agent with a stable utility function cannot make itself happier by hiding evidence from itself, unless it is being modeled by an outside entity.