Peter de Blanc submitted a paper to arXiv.org in 2007 called "Convergence of Expected Utilities with Algorithmic Probability Distributions."  It claims to show that a computable utility function can have an expected value only if the utility function is bounded.

This is important because it implies that, if a utility function is unbounded, it is useless.  The purpose of a utility function is to compare possible actions k by choosing the k for which U(k) is maximal.  You can't do this if U(k) is undefined for any k, let alone for every k.

I don't know whether any agent we contemplate can have a truly unbounded utility function, since the universe is finite.  (The multiverse, supposing you believe in that, might not be finite; but as the utility function is meant to choose a single universe from the multiverse, I doubt that's relevant.)  But it is worth exploring, as computable functions are worth exploring despite not having infinitely long tapes for our Turing machines.  I previously objected that the decision process is not computable; but this is not important - we want to know whether the expected value exists, before asking how to compute (or approximate) it.

The math in the paper was too difficult for me to follow all the way through; so instead, I tried to construct a counterexample.  This counterexample does not work; the flaw is explained in one of comments below.  Can you find the flaw yourself?  This type of error is both subtle and common.  (The problem is not that the theorem actually proves that for any unbounded utility function, there is some set of possible worlds for which the expected value does not converge.)

The abstract says:

We consider an agent interacting with an unknown environment. The environment is a function which maps natural numbers to natural numbers; the agent's set of hypotheses about the environment contains all such functions which are computable and compatible with a finite set of known input-output pairs, and the agent assigns a positive probability to each such hypothesis. We do not require that this probability distribution be computable, but it must be bounded below by a positive computable function. The agent has a utility function on outputs from the environment. We show that if this utility function is bounded below in absolute value by an unbounded computable function, then the expected utility of any input is undefined. This implies that a computable utility function will have convergent expected utilities iff that function is bounded.

This is the formalism the paper uses as I understand it:

The world is described by a function h:N →N, where N represents the integers, and h is a function from a countable set of possible actions, into a countable set of possible observations.

H is the set of possible functions h.  p:H →[0 .. 1] is a probability distribution for H.  S_I is a subset of H representing all possible worlds (possible values for h) that are consistent with prior observations, meaning that for f in S_I, p(f) > 0, and for f not in S_I, p(f) = 0.

The agent considers a countable number of possible actions.  The expected utility of action k is the sum, over all possible worlds f in S_I, of p(f)U(f(k)).  Theorem 1 says that this sum does not converge, given some conditions which are supposed to have the meaning, "U is unbounded".

First, note that S_I must be countable.  This is because the sum over f in S_I of p(f) = 1, and as Planetmath.org says, a sum can be finite only if the number of items summed is countable.  (Also, they are all computable functions, and the number of computable functions is countable; and they are already indexed with Gödel numbers in the paper.)  I will renumber the set S_I.  Instead of indexing them with Gödel numbers, as the paper does, I will index them so that f_i (the ith member of S_I) is a function of i in a different way.

I choose a probability function p:S_I→[0 .. 1].  The sum of p(f_i) must be 1.  I choose the function p(f_i) = 3/4ⁱ.  This sums to 1, since the infinite series 1/4ⁱ (starting from i=1) sums to 1/3.  (To see this, let the sum from i=0 to infinity be S; see that S/4 = S - 1, 4S-S = 4. s(4-1)=4, S = 4/3.)

Since the theorem is supposed to hold for all cases, I choose the case where the set of possible worlds consistent with observation is S_I = { f_i | f_i(k) = (2(1-1/2^k))ⁱ }.  Note that for every i, for every k₂ > k₁, f_i(k₂) ≥ f_i(k₁), and the limit as k approaches infinity of f_i(k) is 2ⁱ.  So f_i(k) is bounded above by 2ⁱ.

I choose the unbounded utility function U(f(k)) = f(k).  Since f_i(k) is bounded above by 2ⁱ, p(f_i)U(f_i(k)) is bounded above by (3/4ⁱ)2ⁱ.  The sum from i=1 to infinity of (3/4ⁱ)2ⁱ is 3×(sum from i=1 to infinity of 2ⁱ/4ⁱ) = 3×(sum from i=1 to infinity of 1/2ⁱ) = 3.

So, for this case, the utility function is U(n) = n, which is unbounded; all other criteria are met; and the expected value is always bounded above by 3.

Can you find any reason why this is not a counterexample?

The only step I'm suspicious of is the step where I choose the set of possible worlds.  Does the theorem implicitly assume that the utility function may be evaluated in any countable set of possible worlds (meaning it is actually proving that, for any unbounded utility function, there is some set of possible worlds for which the expected value does not converge)?