= 27d567bc2d48d9f40e9ccc20ea370b15)
CarlShulman comments on Newcomb's Problem and Regret of Rationality - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (588)
Wait a second, the following bounded utility function can explain the quoted preferences:
Benja Fallenstein gave an alternative formulation that does imply an unbounded utility function:
But these preferences are pretty counter-intuitive to me. If U(live n years) is unbounded, then the above must hold for any nonzero p, q, and with "googolplex" replaced by any finite number. For example, let p = 1/3^^^3, q = .8, n = 3^^^3, and replace "googolplex" with "0". Would you really be willing to give up .8 probability of 3^^^3 years of life for a 1/3^^^3 chance at a longer (but still finite) one? And that's true no matter how many up-arrows we add to these numbers?
"Would you really be willing to give up .8 probability of 3^^^3 years of life for a 1/3^^^3 chance at a longer (but still finite) one?"
I'd like to hear this too.
Okay. There's two intuitive obstacles, my heuristic as a human that my mind is too weak to handle tiny probabilities and that I should try to live my life on the mainline, and the fact that 3^^^3 already extrapolates a mind larger than the sum of every future experience my present self can empathize with.
But I strongly suspect that answering "No" would enable someone to demonstrate circular / inconsistent preferences on my part, and so I very strongly suspect that my reflective equilibrium would answer "Yes". Even in the realm of the computable, there are simple computable functions that grow a heck of a lot faster than up-arrow notation.
Eliezer, would you be willing to bet all of your assets and future earnings against $1 of my money, that we can do an infinite amount of computation before the universe ends or becomes incapable of supporting life?
Your answer ought to be yes, if your preferences are what you state. If it turns out that we can do an infinite amount of computation before the universe ends, then this bet increases your money by $1, which allows you to increase your chance of having an infinite lifetime by some small but non-zero probability. If it turns out that our universe can't do an infinite amount of computation, you lose a lot, but the loss of expected utility is still tiny compared to what you gain.
So, is it a bet?
Also, why do you suspect that answering "No" would enable someone to demonstrate circular / inconsistent preferences on your part?
No for two reasons - first, I don't trust human reason including my own when trying to live one's life inside tiny probabilities of huge payoffs; second, I ordinarily consider myself an average utilitarian and I'm not sure this is how my average utilitarianism plays out. It's one matter if you're working within a single universe in which all-but-infinitesimal of the value is to be found within those lives that are infinite, but I'm not sure I would compare two differently-sized possible Realities the same way. I am not sure I am willing to say that a finite life weighs nothing in my utility function if an infinite life seems possible - though if both were known to coexist in the same universe, I might have to bite that bullet. (At the opposite extreme, a Bostromian parliament might assign both cases representative weight proportional to probability and let them negotiate the wise action.)
Also I have severe doubts about infinite ethics, but that's easily fixed using a really large finite number instead (pay everything if time < googolplex, keep $1 if time > TREE(100), return $1 later if time between those two bounds).
Keep growing the lifespan by huge computational factors, keep slicing near-infinitesimally tiny increments off the probability. (Is there an analogous inconsistency to which I expose myself by answering "No" to the bet above, from trying to treat alternative universes differently than side-by-side spatial reasons?)
In that case, it's not that your utility function is unbounded in years lived, but rather your utility for each year lived is a decreasing function of the lifetime of the universe (or perhaps total lifetime of everyone in the universe).
I'll have to think if that makes sense.
It's possible that I'm reasoning as if my utility function is over "fractions of total achievable value" within any given universe. I am not sure if there are any problems with this, even if it's true.
After thinking about it, that doesn't make sense either. Suppose Omega comes to you and says that among the universes that you live in, there is a small fraction that will end in 5 years. He offers to kill you now in those universes, in exchange for granting you a googleplex years of additional life in a similar fraction of universes with time > TREE(100) and where you would have died in less than googleplex years without his help (and where others manage to live to TREE(100) years old if that makes any difference). Would you refuse?
No. But here, by specification, you're making all the universes real and hence part of a larger Reality, rather than probabilities of which only a single one is real.
If there were only one Reality, and there were small probabilities of it being due to end in 5 years, or in a googolplex years, and the two cases seemed of equal probability, and Omega offered to destroy reality now if it were only fated to last 5 years, in exchange for extending its life to TREE(100) if it were otherwise fated to last a googolplex years... well, this Reality is already known to have lasted a few billion years, and through, say, around 2 trillion life-years, so if it is due to last only another 5 years the remaining 30 billion life-years are not such a high fraction of its total value to be lost - we aren't likely to do so much more in just another 5 years, if that's our limit; it seems unlikely that we'd get FAI in that time. I'd probably still take the offer. But I wouldn't leap at it.
In that case, would you accept my original bet if I rephrase it as making all the universes part of a larger Reality? That is, if in the future we have reason to believe that Tegmark's Level 4 Multiverse is true, and find ourselves living in a universe with time < googolplex, then you'd give you all your assets and future earnings, in return for $1 of my money if we find ourselves living in a universe with time > TREE(100).
That does have quite a bit of intuitive appeal! However, when you look at a possible universe from the outside, there are no levers nor knobs you can turn, and all the value achieved by the time of heat death was already inherent in the configurations right after the big bang--
--so if you do not want "fraction of total achievable value" to be identically one for every possible universe, the definition of your utility function seems to get intertwined with how exactly you divvy up the world into "causal nodes" and "causal arrows", in a way that does not seem to happen if you define it in terms of properties of the outcome, like how many fulfilling lifes lived. (Of course, being more complicated doesn't imply being wrong, but it seems worth noting.)
And yes, I'm taking a timeful view for vividness of imagination, but I do not think the argument changes much if you don't do that; the point is that it seems like number-of-fulfilling-lifes utility can be computed given only the universal wavefunction as input, whereas for fraction-of-achievable-fulfilling-lifes, knowing the actual wavefunction isn't enough.
Could your proposal lead to conflicts between altruists who have the same values (e.g. number of fulfilling lifes), but different power to influence the world (and thus different total achievable value)?