If "parameter" were "instrumental value", that would mean you could set the parameter according to your values. That would be very similar to saying it was part of your values.

The notion of "value" vs. "instrumental value" is probably bankrupt. It's very similar to the dichotomy of "grounded symbol vs. dependent symbol". I talked about this in the section "Value is a network problem" of my regrettably long post on values.

In this particular case, I'm also presenting the option that "parameter" is something you can place logical constraints on, regardless of values. Like math: You can't say that believing that 1+1=3 is one of your values.

Answer to Wei Dai: It's a parameter you need to set to implement your values; but I don't know how you set it. That's the problem I'm pointing out. Resolving whether this was an instrumental value or not would solve the problem.

See Eliezer's post against discounting - I've been trying to address the problem that's left even after you factor out uncertainty.

Nitpick: Feeling that way would prevent you from conquering the world.

As in, hunter-gatherers don't feel the way towards their cousins that citizens of the developed world are encouraged to feel towards their fellow citizens? Or that if I personally feel that way, I personally will not become the ruler of the world? Or something else?

I was not claiming that it said that. The paper did not mention time discounting but, if you understand the paper, it is clear that discounting cannot affect the result.

As Manfred said, it does not appear that the results of the paper are affected by time discounting.

Let's be a bit more explicit about this. The model in the paper is that an action (or perhaps a pair (action,context)) is represented by a single natural number; this is provided to the environment and it returns a single natural number; the agent feeds that number into its utility function, and out comes a utility. The agent has measured what the environment does in response to a finite set of actions; what it cares about is the expected utility (over computable environments whose behaviour is consistent with the agent's past observations, with some not-necessarily-computable but not too ill-behaved probability distribution on them) of the environment's response to its actions.

The paper says that the expected utilities don't exist, if the utility function is unbounded and computable (or merely bounded below in absolute value by an unbounded computable function).

(Remark: It seems to me that this isn't necessarily fatal; if the agent cannot exactly repeat previous actions, and if it happens that the expected utility difference between two of the agent's actions exists, then it can still decide between actions. However, (1) the "cannot repeat previous actions" condition seems a bit artificial and (2) I'd guess that the arguments in the paper can be adjusted to show that expected utility differences are also divergent. But I could be wrong, and it would be interesting to know.)

So, anyway. How does time discounting fit into this? It seems to me that this is meant to model the immediate response of the environment to the agent's action; time doesn't come into it at all. And the conclusion is that even then -- even without considering the possible infinite future -- the relevant expectations don't exist.

The pathology described in the paper doesn't seem to me to have anything to do with not discounting in time. Turning the consequences of an action into an infinite stream rather than a single result might make things worse, but it can't possibly make them better.

Actually, that's not quite fair. Here's one way it could make them better. One way to avoid the divergence described in the paper is to have a bounded utility function. That seems troublesome to many people. But it's not so unreasonable to say that the utility you attach to what happens in any bounded region of spacetime should be bounded. So maybe there's some mileage to taking the bounded-utility case of this model (where the expectations all exist happily), then representing an agent's actual deliberations as involving (say) some kind of limit as the spacetime region gets larger, and hoping that lim { larger spacetime region } E { utility } converges even though E { lim { larger spacetime region} utility } doesn't. Which might be the case; I haven't thought about it carefully enough to have much idea how plausible that is.

In that scenario, you might well get saved by exponential time-discounting. Or even something weaker like 1/t^2. (Probably not 1/t, though.) But it seems to me that filling in the details is a big job; and I don't think it can possibly be right to assert that time discounting makes the result of the paper go away, without doing that work.

I don't have my head around the proof in the paper, but I can answer your specific questions:

First, the paper needs to define what an unbounded function is.

U maps N to R and is unbounded, which means for all x in R there exists a y in N such that U(y) > x.

Next, in Lemma one that B(x) > f(x) infinitely often. But f(x) is defined as "a function that describes the environment". There is no ordering defined on f(x); no interpretation of what the number you get out of f(x) means.

His point is that B grows really fast, faster than any computable f. I don't see where it defines f(x) to be "a function that describes the environment", and that definition wouldn't make sense at this point anyway. Once he's done proving B grows faster than any computable function, he doesn't need that f any more.

Some retractions, clarified by Peter in email:

• Peter does place a limit on the probability distribution: It must never be zero. (I read a '<' as '<='.) This removes counterexamples 1 and 3. However, I am not sure it's possible to build a probability distribution satisfying his requirements (one that has cardinality 2^N, no zero terms, and sums to one). This link says it is not possible.

• The reason Peter's expected value calculation is not of the form p(x)U(x) is because he is summing over one possible action. p(h) is the probability that a particular hypothesis h is true; h(k) is the result of action k when h is true; U(h(k)) is the utility from that result.

• "To establish that this series does not converge, we will show that infinitely many of its terms have absolute value >= 1." This statement is valid.

However, my second counterexample still looks solid to me: U(n) = n if n is even, -n if n is odd. p(n) = 1/2^n. This doesn't work in Peter's framework because the domain of p is not countable.

So I give 3 caveats on Peter's theorem:

• It only applies if the number of possible worlds, and possible actions you are summing over, is uncountable. This seems overly restrictive. It really doesn't matter if expected utility converges or not, when the sum for the expected utility for a single action would be non-computable regardless of whether it converged.

• It is impossible to construct a probability distribution satisfying his requirements, so the theorem doesn't apply to any possible situations.

• It doesn't prove that the expected utility isn't bounded. The way the theorem works, it can't rule out a bound over an interval of 2, e.g., U(k) might be provably between -1.1 and 1.1. A reasoner can work fine with bounded utility functions.