This is a result from the first MIRIx Cambridge workshop (coauthored with Janos and Jim).
One potential problem with bounded utility functions is: what happens when the bound is nearly reached? A bounded utility maximizer will get progressively more and more risk averse as it gets closer to its bound. We decided to investigate what risks it might fear. We used a toy model with a bounded-utility chocolate maximizer, and considered what happens to its resource allocation in the limit as resources go to infinity.
We use "chocolate maximizer'' as conceptual shorthand meaning an agent that we model as though it has a single simple value with a positive long-run marginal resource cost, but only as a simplifying assumption. This is as opposed to a paperclip maximizer, where the inappropriate simplicity is implied to be part of the world, not just part of the model.
Conceptual uncertainty
We found that if a bounded utility function approaches its bound too fast, this has surprising pathological results when mixed with logical uncertainty. Consider a bounded-utility chocolate maximizer, with philosophical uncertainty about what chocolate is. It has a central concept of chocolate , and there are classes of mutated versions of the concept of chocolate
at varying distances from the central concept, such that the probability that the true chocolate is in class
is proportional to
(i.e. following a power law).
Suppose also that utility is bounded using a sigmoid function , where x is the amount of chocolate produced. In the limit as resources go to infinity, what fraction of those resources will be spent on the central class
? That depends which sigmoid function is used, and in particular, how quickly it approaches the utility bound.
Example 1: exponential sigmoid
Suppose we allocate resources to class
, with
for total resource r. Let
.
Then the optimal resource allocation is
Using Lagrange multipliers, we obtain for all i,
Then,
Thus, the resources will be evenly distributed among all the classes as r increases. This is bad, because the resource fraction for the central class goes to 0 as we increase the number of classes.
EDITED: Addendum on asymptotics
Since we have both r and n going to infinity, we can specify their relationship more precisely. We assume that n is the highest number of classes that are assigned nonnegative resources for a given value of r:
Thus,
so the highest class index that gets nonnegative resources satisfies
Example 2: arctan sigmoid
The optimal resource allocation is
Using Lagrange multipliers, we obtain for all i,
Then,
Thus, for the limit of the resource fraction for the central class
is finite and positive.
Conclusion
The arctan sigmoid results in a better limiting resource allocation than the exponential sigmoid, because it has heavier tails (for sufficiently large ). Thus, it matters which bounding sigmoid function you choose.
Certainly given a utility function and a model, the best thing to do is what it is. The point was to show that some utility functions (eg using the exponential-decay sigmoid) have counterintuitive properties that don't match what we'd actually want.
Every response to this post that takes the utility function for granted and remarks that the optimum is the optimum is missing the point: we don't know what kind of utility function is reasonable, and we're showing evidence that some of them give optima that aren't what we'd actually want if we were turning the world into chocolate/hedonium.
If it seems strange to you to consider representing what you want by a bounded utility function, a post about that will be forthcoming.
No, it doesn't seem strange to me to consider representing what I want by a bounded utility function. It seems strange to consider representing what I want by a utility function that converges exponentially fast towards its bound.
I'll repeat something I said in another comment:
... (read more)