The function U in PdB's paper doesn't take integers as arguments, it takes infinite sequences of "perceptions". Provided there are at least two possible perceptions at each time step, there are uncountably many such sequences. How are you proposing to change the model to make the domain of U countable, while still making it be a model of agents acting in time?
The domain of U must not only be countable; it must be finite. Remember, an agent needs to compute this sum once for every possible action! Both the number of possible worlds, and the number of possible actions, need to be finite; or the utility function needs to be solvable analytically (unlikely); or else the agent will take an infinite amount of time to make a decision.
So, the model of an agent acting in time, where we compute utility at each timestep in the future, does not seem to produce a computable decision process. Is that right?
I think you are mistaken about what those requirements are. What he needs is a probability measure on the space of environments; he doesn't insist that each individual possible environment have nonzero probability.
You're right. Let's try to figure out how many possible environments there are.
"Let p:H->R be our probability distribution on the hypothesis set. We require that there exist some computable function p-bar:N->Q such that for all psi_n in S_I, 0 < p-bar(n) <= p(psi_n)".
p(psi_n) must be non-zero for all psi_n in S_I. S is the set of mu-recursive functions on a single argument. A mu-recursive function is one that can be computed by Turing machines. S_I is the set of computable environments f such that for every input that has been tested, f(i) = h(i), where h is the true environment.
How large is S_I? It could be up to the size of S. Is the number of computable functions less than R (2^N)?
In any case, if you try to sum an infinite series with infinitely many terms of absolute value > 1, then the range of values in your partial sums depends on the order of summation. In particular, even if the U(k) were bounded, that wouldn't enable you to say things like "the expected utility is a divergent series but its value oscillates only between -13 and +56" unless there were One True Order of summation for the terms of the series. Which I can't see that there is or could be.
If I construct a countable series so as to have a 1-1 pairing between terms with value 1 and terms with value -1, does that do no good? e.g. U(n) = 1 if n is even, -1 if n is odd.
Is there a way to enter subscripts and superscripts in this thing?
No, the domain of U need not be finite. (1) The paper explicitly says that U need not be computable. (2) The model of computation implicit in the paper permits computable functions to have infinite domain (it's basically Turing machines). (3) The way the paper proposes that a computable U should be implemented involves only looking at a finite amount of the information in the argument of U. (Kinda.)
But yes: the paper's model does permit a non-computable decision process. (Neither does it require that U be uncomputable.)
There is a countable infinite of comp...
I've written before about the difficulty of distinguishing values from errors, from algorithms, and from context. Now I have to add to that list: How can we distinguish our utility function from the parameters we use to apply it?
In my recent discussion post, "Rationalists don't care about the future", I showed that exponential time-discounting, plus some assumptions about physics and knowledge, leads to not caring about the future. Many people responded by saying that, if I care about the future, this shows that my utility function does not use exponential time-discounting.
This response assumes that the shape of my time-discounting function is part of my utility function. In other words, the way you time-discount is one of your values.
By contrast, Eliezer wrote an earlier post saying that we should use human values, but without time-discounting. Eliezer is aware that humans appear to use time discounting. Therefore, this implicitly claims that the time-discounting function is not one of our values. It's a parameter for how we implement them.
(Some of the arguments Eliezer used were value-based arguments, suggesting that we can use our values to set the parameters that we use to implement our values... I suspect this recursive approach could introduce bogus solutions, like multiplying both sides of an equation by a variable, or worse; but that would take a longer post to address. I will note that some recursive equations do have unique solutions.)
The program of CEV assumes that a transhuman can use some extrapolated version of values currently used by some humans. If that transhuman has a life expectancy of a billion years, it will likely view time discounting differently. Eliezer's post against time discounting suggests, to me, a God-like view of the universe, in which we eliminate time discounting in the same way (and for the same reasons) that many people want to eliminate space-discounting (not caring about far-away people) in contemporary ethics. This is taking an ethical code that evolved agents have, which is constructed to promote the propagation of those agents' genes, and applying it without reference to any particular set of genes. This is also pretty much what folk-morality says a social moral code is. So the idea that you can apply the same utility function from a radically different context, is inherent in CEV, and is common to much public discourse on ethics which assumes that you can construct a social morality that is based on the morality we find in individual agents.
On the other hand, I have argued that assuming that social ethics and individual ethics are the same, is either merely sloppy thinking, or an evolved (or deliberately constructed) lie. People who believed this would probably subscribe to a social-contract theory of ethics. (This view also has problems, beyond the scope of this post.)
I have one heuristic that I think is pretty good for telling when something is not a value: If it's mathematically wrong, it's an error, not a value. So my inclination is to point out that exponential time-discounting is correct. All other forms of time-discounting lead to inconsistencies. You can time-discount exponentially; or you can not time-discount at all, as Eliezer suggested; or you can be in error.
But my purpose in this post is not to continue the arguments from that other post. It's to point out this additional challenge in isolating what values are. Is your time-discounting function a value, or a value parameter?