@Peter As a human, I can't introspect and look at my utility function, so I don't really know if it's bounded or not. If I'm not absolutely certain that it's bounded, should I just assume it's unbounded, since there is much more at stake in this case?

This has been gnawing at my brain for a while. If the useful Universe is temporally unbounded, then utility arguably goes to aleph-null. Some MWI-type models and Ultimate-ensemble models arguably give you an uncountable number of copies of yourself, does that count as greater than than aleph-null or less than aleph-null (because we normalize to a measure [0, 1] that "looks" small)? What if someone claims "the Universe is spatially finite, but everyone has an inaccessible cardinal number of invisible copies of themselves?" Given my ignorance and confusion, maybe it makes sense to pick the X most credible utility measures, and give them each an "equal vote" in deciding what to do next at each stage, as a current interim measure. This horrendous muddled compromise is itself non-utilitarian and sub-optimal, but I personally don't have a better answer at the moment.

I used to think of my utility function as unbounded, and then after Eliezer's "Pascal's Mugging" post I thought of it as probably bounded. This decision changed the way I live my life... not at all. However, I can understand that if you want to instruct an AGI, you may not be able to allow yourself the luxury of such blissful agnosticism.

@Stephen An intuition in the opposite direction (which I think Rolf agrees with) is that once you reach giant tentacled squillions of units of fun, specifying when/where it happens takes just as much algorithmic complexity as making up a mind from scratch (or interpreting it from a rock).

Alas I'm not completely sure what you're talking about, the secret decoder ring says "fun = utility" but I think I require an additional cryptogram clue. Is this a UDASSA reference?