Concise version:
If we have some maximum utility per unit space (reasonable, since there is maximum entropy, and therefore probably a maximum information, per unit space), and we do not break the speed of light, our maximum possible utility will expand polynomially. If we discount future utility exponentially, like the 10-year doubling time of the economy can suggest, the merely polynomial growth gets damped exponentially and we don't care about the far future.
Big problem:
Assumes exponential discounting. However this can also be seen as a reductio of exponential discounting - we don't want to ignore what happens 50 years from now, and we exhibit many behaviors typical of caring about the far future. There's also a sound genetic basis for caring about our descendants, which implies non-exponential discounting programmed into us.
Downvoted for (1) being an extraordinarily laborious way of saying "decaying exponential times modest-degree polynomial is rapidly decaying", (2) only doing the laborious calculations and not mentioning why the result was pretty obvious from the outset, (3) purporting to list ways around the problem (if it is one) and not so much as mentioning "don't discount exponentially", (4) conflating "rationalist" with "exponentially discounting expected-utility maximizer", and most of all (5) the horrible, horrible I-know-I'm-going-to-be-downvoted-for-this-and-you're-all-so-stupid sympathy-fishing.
[EDITED to fix a typo: I'd numbered my points 1,2,3,5,5. Oops.]
What information can be derived about utility functions from behavior?
(Here, "information about utility functions" may be understood in your policy-relevant sense, of "factors influencing the course of action that rational expected-utility maximization might surprisingly choose to force upon us after it was too late to decommit.")
Suppose you observe that some agents, when they are investing, take into account projected market rates of return when trading off gains and losses at different points in time. Here are two hypotheses about the utility functions of those agents.
Hypothesis 1: These agents happened to already have a utility function whose temporal discounting was to match what the market rate of return would be. This is to say: The utility function already assigned particular intrinsic values to hypothetical events in which assets were gained or lost at different times. The ratios between these intrinsic values were already equal to what the appropriate exponential of the integrated market rate of return would later turn out to be.
Hypothesis 2: These agents have a utility function in which assets gained or lost in the near term are valued because of an in...
Any temporal discounting other than temporal is provably inconsistent
The conditions of the proof are applicable only to reinforcement agents which, as a matter of architecture, are forced to integrate anticipated rewards using a fixed weighting function whose time axis is constantly reindexed to be relative to the present. If we could self-modify to relax that architectural constraint -- perhaps weighting according to some fixed less temporally indexical schedule, or valuing something other than weighted integrals of reward -- would you nonetheless hold that rational consistency would require us to continue to engage in exponential temporal discounting? Whether or not the architectural constraint had previously been a matter of choice? (And who would be the "us" who would thus be required by rational consistency, so that we could extract a normative discount rate from them? Different aspects of a person or civilization exhibit discount functions with different timescales, and our discount functions and architectural constraints can themselves partially be traced to decision-like evolutionary and ecological phenomena in the biosphere, whose "reasoning" we may ...
You keep constructing scenarios whose intent, as far as I can tell, is to let you argue that in those scenarios any currently imaginable non-human system would be incapable of choosing a correct or defensible course of action. By comparison, however, you must also be arguing that some human system in each of those scenarios would be capable of choosing a correct or defensible course of action. How?
And: Suppose you knew that someone was trying to understand the answer to this question, and create the field of "Artificial Ability to Choose Correct and Defensible Courses of Action The Way Humans Apparently Can". What kinds of descriptions do you think they might give of the engineering problem at the center of their field of study, of their criteria for distinguishing between good and bad ways of thinking about the problem, and of their level of commitment to any given way in which they've been trying to think about the problem? Do those descriptions differ from Eliezer's descriptions regarding "Friendly AI" or "CEV"?
You seem to be frustrated about some argument(s) and conclusions that you think should be obvious to other people. The above is an explanation of how some conclusions that seem obvious to you could seem not obvious to me. Is this explanation compatible with your initial model of my awareness of arguments' obviousnesses?
Rational expected-utility-maximizing agents get to care about whatever the hell they want. Downvoted.
Both Eliezer and Robin Hanson have argued strongly against time discounting of utility.
EDIT: I'm partially wrong, Hanson is in favour. Sorry and thanks for the correction.
Roko once argued to me that if we are to discount the future, we should use our true discounting function: a hyperbolic function. Because even if that's inherently irrational, it's still what we want. This would also not display the behaviour you discuss here.
Both Eliezer and Robin Hanson have argued strongly against time discounting of utility.
Not Robin Hanson AFAIK - see his: For Discount Rates. Here's YuEl's Against Discount Rates.
Downvoted, because your math is wrong.
(2¹ºº)^t = exp(t*ln(2¹ºº)), so the factor you call 'c' is not a constant multiplier for the integral; in fact, that combination of constants doesn't even show up. The (approximated) integral is actually b∫t²*exp(-at)dt, where a = 100*ln(2) and b = 120. Evaluating this from 0 to T produces the expression: (2b/a³)*(1 - exp(-aT)*(1 + aT + ½(aT)²)).
These factors of exp(-aT) show up when evaluating the integral to T<∞. (Obviously, when T → ∞, the integral converges to (2b/a³).) For a ~ O(5) or higher, then, the entire total utility is found in 10 years, within double precision. That corresponds to a ≈ 7*ln(2). I think this indicates that the model may not be a good approximation of reality. Also, for slower subjective time (a < ln(2) ≈ 0.693), the percentage of total utility found in 10 years drops. For a = 0.1*ln(2), it's only 3.33%.
Also, you defined either 'x' or your linear density function incorrectly. If you want x to be stars/ly^2, the density function should be ρ = x(1 - r/50000). If you do all of the calculations symbolically and don't plug in values until the end, the equation for total utility as a function of time (before discountin...
Among other things, this means we might not want to turn the Universe over to a rational expected-utility maximizer.
So this is just a really long way of saying that your utility function doesn't actually include temporal discounting.
I think this post (Evolution and irrationality) is interesting but don't know what to make of it due to a lack of general expertise:
...Sozou’s idea is that uncertainty as to the nature of any underlying hazards can explain time inconsistent preferences. Suppose there is a hazard that may prevent the pay-off from being realised. This would provide a basis (beyond impatience) for discounting a pay-off in the future. But suppose further that you do not know what the specific probability of that hazard being realised is (although you know the probability dis
Utility functions are calculated from your preferences, not vice-versa. (To a first approximation.)
Caring about the future is also problematic, because the utility of the distant future then overwhelms any considerations about the present.
Indeed! I am still waiting for this problem to be tackled. At what point is an expected utility maximizer (without time preferences) going to satisfy its utility function, or is the whole purpose of expected utility maximization to maximize expected utility rather than actual utility?
People here talk about the possibility of a positive Singularity as if it was some sort of payoff. I don't see that. If you think it ...
After spending some time thinking about the result from the correct math, here are my conclusions:
You claimed that the percentage of total utility attained in the first 10 years was independent of the level of time discounting. This is clearly not the case, as the percentage of total utility attained in the first T years with time discounting factor a is given by (1 - exp(-aT)*(1 + aT + ½(aT)²)). The expression -exp(-aT)*(1 + aT + ½(aT)²) (the difference between the previous expression and 1) goes to zero within double precision when the combined factor a...
This is a critical topic, but LessWrong hates it. Matthew 7:6 comes to mind.
For the record, I dispute your causal model of the audience's response.
In particular, I dispute your model of the audience's moral reasoning as to what is inevitably being approved of or disapproved of by expressions of approval or disapproval of your actions relating to the post.
I also dispute your model of the audience's factual and moral reasoning about the gravity of the problem you suggest. I dispute specifically your model of the audience's process of choosing to suppose that non-exponential weighting functions could be considered sufficiently indicative of potential solutions as to justify relative unconcern. (This is because I dispute your model of the utility function structures initially familiar to the audience. As part of this, I dispute your model of their descriptions of discounting functions, according to which it apparently would be impossible for them to intend to refer to a function which was to be applied on a prespecified absolute timescale, without being translated to start at an agent's present time. If that was not your model, then I dispute your confusing apparent claim that such ...
I feel like my concern for the well being of people I don't know does not change at all with time, but my concern for people I do know is discounted, and for myself, I discount more heavily. This seems to imply that we do not discount with increasing time but instead with decreasing association. As in, we care more about minds more similar to our own, or with whom we interact more, and our minds become more different farther in the future.
I second Manfred and gjm's comments.
One additional point regarding subjective time. You say:
Strange but true. (If subjective time is slower, the fact that t=20 matters more to us is balanced out by the fact that t=2 and t=.2 also matter more to us.)
But even if I temporally discount by my subjective sense of time, if I can halt subjective time (e.g. by going into digital or cryonic storage) then the thing to do on your analysis is to freeze up as long as possible while the colonization wave proceeds (via other agents, e.g. Von Neumann probes or the res...
Didn't like this post much either (sorry!). Yes, if you assume a substantial level of temporal discounting that makes the future matter less. If you don't like that, perhaps do not apply so much temporal discounting.
The dense maths hinders the reader here. I don't really approve of the dissing of expected utility maximizers at the end either.
Could someone please explain any possible justification for exponential discounting in this situation? I asked earlier, but got voted below the threshold. If this is a sign of disagreement, then I would like to understand why there is disagreement.
Robin Hanson's argument for exponential discounting derives from an exponential interest rate. Our current understanding of physics implies there won't be an exponential interest rate forever (in fact this is the point of the present article). So Robin Hanson's argument doesn't apply at all to the situation in th...
People pay an exponential amount of future utility for utility now because we die. We inappropriately discount the future because our current environment has a much longer life expectancy than the primitive one. One should discount according to actual risk, and I plan on self modifying to do this when the opportunity arises.
Could you perhaps give some plausible argument for exponential discounting, or some record of anyone who has seriously considered applying it universally? I appear to discount approximately exponentially in the near term, but really its a reflection of my uncertainty about the future. I value future humans about as much as present humans, I just doubt my ability to understand my influence on them (in most but not all cases).
Even if you accept exponential discounting, your physical arguments seem pretty weak. How confident are you that faster than light tra...
So, this calculation motivates non-expansion, but an agent with an identical utility function that is expansionist anyway attains greater utility and for a longer time... is that right?
Rewrite 2^(100t) as (2^100)^t = ln(2^100)e^t.
Plugging in for t=2 is giving me 2^(100t)=1.6*10^60 and ln(2^100)e^t = 512.17
Is this an error or did I read it wrong?
There are strong reasons for believing that time-discounting is exponential.
For all utility functions a human may have? What are these reasons?
If we use time discounting we should care about the future, because it's possible that time machines can be made, but are difficult. If so, we'd need a lot of people work it out. A time machine would be valuable normally, but under time discounting, it gets insane. I don't know what half-life you're using, but let's use 1000 years, just for simplicity. Lets say that we bring a single person back to the beginning of the universe, for one year. This would effectively create about 8.7*10^4,154,213 QALYs. Any chance of time travel would make this worth while.
I...
If we assume that our time-discounting function happens to be perfectly adjusted to match our rate of economic growth now, is it wise to assume that eventually the latter will change drastically but the former will remain fixed?
ADDED: Downvoting this is saying, "This is not a problem". And yet, most of those giving their reasons for downvoting have no arguments against the math.
A major problem with simple voting systems like that used on LW is that people impute meanings to voters more confidently than they should. I've seen this several times here.
If people give a reason for downvoting, they're probably not being deceptive and may even be right about their motives, but most who vote will not explain why in a comment and you're overstepping the bounds of what you ca...
If you compute the implications of a utility function and they do not actually agree with observed preferences, then that is an argument that the utility function you started with was wrong. In this case, you seem to have an argument that our utility function should not have time discounting that's stronger than polynomial.
Discussion with Jeff Medina made me realize that I can't even buy into the model needed to ask how to time discount. That model supposes you compute expected utility out into the infinite future. That means that, for every action k, you compute the sum, over every timestep t and every possible world w, of p(w(t))U(w(t, k))_t .
If any of these things have countably many objects - possible worlds, possible actions, or timesteps - then the decision process is uncomputable. It can't terminate after finitely many steps. This is a fatal flaw with the standard approach to computing utility forward to the infinite future.
Assuming people that have children are rational then the time discounting factor is not what you claim or (perhaps more likely) as people love and expect to love their children and grandchildren (and future descendents) then their (the descendents) expected utility is not time discounted even while ones own is. I likewise imagine that some people will treat long lived versions of themselves in a similar fashion such that they discount their own expected utility for the near term but do not discount their expected utility in the same amount for the version...
This is a critical topic, but LessWrong hates it. Matthew 7:6 comes to mind.
For the record, I dispute your causal model of the audience's response.
In particular, I dispute your model of the audience's moral reasoning as to what is inevitably being approved of or disapproved of by expressions of approval or disapproval of your actions relating to the post.
I also dispute your model of the audience's factual and moral reasoning about the gravity of the problem you suggest. I dispute specifically your model of the audience's process of choosing to suppose that non-exponential weighting functions could be considered sufficiently indicative of potential solutions as to justify relative unconcern. (This is because I dispute your model of the utility function structures initially familiar to the audience. As part of this, I dispute your model of their descriptions of discounting functions, according to which it apparently would be impossible for them to intend to refer to a function which was to be applied on a prespecified absolute timescale, without being translated to start at an agent's present time. If that was not your model, then I dispute your confusing apparent claim that such functions, if non-exponential, must be dynamically inconsistent.)
I am concerned that the errors in your model of the audience, if left unchallenged, will only serve to reinforce in you the apparent resentful, passive-aggressive self-righteousness which would have largely been itself the cause of the misinterpretations which led to those errors originally. This self-reinforcing effect might create needless mutual epistemic alienation.
Related to Exterminating life is rational.
ADDED: Standard assumptions about utility maximization and time-discounting imply that we shouldn't care about the future. I will lay out the problem in the hopes that someone can find a convincing way around it. This is the sort of problem we should think about carefully, rather than grasping for the nearest apparent solution. (In particular, the solutions "If you think you care about the future, then you care about the future", and, "So don't use exponential time-discounting," are easily-grasped, but vacuous; see bullet points at end.)
The math is a tedious proof that exponential time discounting trumps geometric expansion into space. If you already understand that, you can skip ahead to the end. I have fixed the point raised by Dreaded_Anomaly. It doesn't change my conclusion.
Suppose that we have Planck technology such that we can utilize all our local resources optimally to maximize our utility, nearly instantaneously.
Suppose that we colonize the universe at light speed, starting from the center of our galaxy (we aren't in the center of our galaxy; but it makes the computations easier, and our assumptions more conservative, since starting from the center is more favorable to worrying about the future, as it lets us grab lots of utility quickly near our starting point).
Suppose our galaxy is a disc, so we can consider it two-dimensional. (The number of star systems expanded into per unit time is well-modeled in 2D, because the galaxy's thickness is small compared to its diameter.)
The Milky Way is approx. 100,000 light-years in diameter, with perhaps 100 billion stars. These stars are denser at its center. Suppose density changes linearly (which Wikipedia says is roughly true), from x stars/sq. light year at its center, to 0 at 50K light-years out, so that the density at radius r light-years is x(50,000-r). We then have that the integral over r = 0 to 50K of 2πrx(50000-r)dr = 100 billion, 2πx(50000∫rdr - ∫r2dr) = 100 billion, x = 100 billion / 2π(50000∫rdr - ∫r2dr) = 100 billion / π[(50000r2 - 2r3/3) from r=0 to 50K = π(50000(50000)2 - 2(50000)3/3) = 500003π(1 - 2/3)] = 100 billion / 130900 billion = .0007639.
We expand from the center at light speed, so our radius at time t (in years) is t light-years. The additional area enclosed in time dt is 2πtdt, which contains 2πtx(50000-t)dt stars.
Suppose that we are optimized from the start, so that expected utility at time t is proportional to number of stars consumed at time t. Suppose, in a fit of wild optimism, that our resource usage is always sustainable. (A better model would be that we completely burn out resources as we go, so utility at time t is simply proportional to the ring of colonization at time t. This would result in worrying a lot less about the future.) Total utility at time t is 2πx∫t(50000-t)dt from 0 to t = 2πx(50000t2/2 - t3/3) ≈120t2 - .0016t3.
Our time discounting for utility is related to that we find empirically today, encoded in our rate of return on investment, which roughly doubles every ten years. Suppose that, with our Planck technology, subjective time is Y Planck-tech years = 1 Earth year, so our time discounting says that utility x at time t is worth utility x/2.1Y at time t+1. Thus, the utility that we, at time 0, assign to time t, with time discounting, is (120t2 - .0016t3) / 2.1Yt. The total utility we assign to all time from now to infinity is the integral, from t=0 to infinity, of (120t2 - .0016t3) / 2.1Yt.
Look at that exponential, and you see where this is going.
Let's be optimistic again, and drop the .0016t3, even though including it would make us worry less about the future. <CORRECTION DUE TO Dreaded_Anomaly> Rewrite 2.1Yt as (2.1Y)t = eat, a = .1Yln2. Integrate by parts to see ∫t2e-atdt = -e-at(t2/a + 2t/a2 + 2/a3). Then ∫120t2/2.1Ytdt = 120∫t2e-atdt = -120e-at(t2/a + 2t/a2 + 2/a3) from t=0 to infinity.</CORRECTION DUE TO Dreaded_Anomaly>
For Y = 1 (no change in subjective time), t=0 to infinity, this is about 6006. For comparison, the integral from t=0 to 10 years is about 5805. Everything after the first 10 years accounts for 3.3% of total utility over all time, as viewed by us in the present. For Y = 100, the first 10 years account for all but 1.95 x 10-27 of the total utility.
What all this math shows is that, even making all our assumptions so as to unreasonably favor getting future utility quickly and having larger amounts of utility as time goes on, time discounting plus the speed of light plus the Planck limit means the future does not matter to utility maximizers. The exponential loss due to time-discounting always wins out over the geometric gains due to expansion through space. (Any space. Even supposing we lived in a higher-dimensional space would probably not change the results significantly.)
Here are some ways of making the future matter:
The first two don't work:
So it seems that, if we maximize expected total utility with time discounting, we need not even consider expansion beyond our planet. Even the inevitable extinction of all life in the Universe from being restricted to one planet scarcely matters in any rational utility calculation.
Among other things, this means we might not want to turn the Universe over to a rational expected-utility maximizer.
I know that many of you will reflexively vote this down because you don't like it. Don't do that. Do the math.
ADDED: This post makes it sound like not caring about the future is a bad thing. Caring about the future is also problematic, because the utility of the distant future then overwhelms any considerations about the present. For example, while a FAI that doesn't care about the future might neglect expansion into space, it won't kill 90% of the people on earth because they pose a threat during this precarious transition period.
ADDED: Downvoting this is saying, "This is not a problem". And yet, most of those giving their reasons for downvoting have no arguments against the math.