The Lifespan Dilemma
One of our most controversial posts ever was "Torture vs. Dust Specks". Though I can't seem to find the reference, one of the more interesting uses of this dilemma was by a professor whose student said "I'm a utilitarian consequentialist", and the professor said "No you're not" and told them about SPECKS vs. TORTURE, and then the student - to the professor's surprise - chose TORTURE. (Yay student!)
In the spirit of always making these things worse, let me offer a dilemma that might have been more likely to unconvince the student - at least, as a consequentialist, I find the inevitable conclusion much harder to swallow.
I'll start by briefly introducing Parfit's Repugnant Conclusion, sort of a little brother to the main dilemma. Parfit starts with a world full of a million happy people - people with plenty of resources apiece. Next, Parfit says, let's introduce one more person who leads a life barely worth living - but since their life is worth living, adding this person must be a good thing. Now we redistribute the world's resources, making it fairer, which is also a good thing. Then we introduce another person, and another, until finally we've gone to a billion people whose lives are barely at subsistence level. And since (Parfit says) it's obviously better to have a million happy people than a billion people at subsistence level, we've gone in a circle and revealed inconsistent preferences.
My own analysis of the Repugnant Conclusion is that its apparent force comes from equivocating between senses of barely worth living. In order to voluntarily create a new person, what we need is a life that is worth celebrating or worth birthing, one that contains more good than ill and more happiness than sorrow - otherwise we should reject the step where we choose to birth that person. Once someone is alive, on the other hand, we're obliged to take care of them in a way that we wouldn't be obliged to create them in the first place - and they may choose not to commit suicide, even if their life contains more sorrow than happiness. If we would be saddened to hear the news that such a person existed, we shouldn't kill them, but we should not voluntarily create such a person in an otherwise happy world. So each time we voluntarily add another person to Parfit's world, we have a little celebration and say with honest joy "Whoopee!", not, "Damn, now it's too late to uncreate them."
And then the rest of the Repugnant Conclusion - that it's better to have a billion lives slightly worth celebrating, than a million lives very worth celebrating - is just "repugnant" because of standard scope insensitivity. The brain fails to multiply a billion small birth celebrations to end up with a larger total celebration of life than a million big celebrations. Alternatively, average utilitarians - I suspect I am one - may just reject the very first step, in which the average quality of life goes down.
But now we introduce the Repugnant Conclusion's big sister, the Lifespan Dilemma, which - at least in my own opinion - seems much worse.
To start with, suppose you have a 20% chance of dying in an hour, and an 80% chance of living for 1010,000,000,000 years -
Now I know what you're thinking, of course. You're thinking, "Well, 10^(10^10) years may sound like a long time, unimaginably vaster than the 10^15 years the universe has lasted so far, but it isn't much, really. I mean, most finite numbers are very much larger than that. The realms of math are infinite, the realms of novelty and knowledge are infinite, and Fun Theory argues that we'll never run out of fun. If I live for 1010,000,000,000 years and then die, then when I draw my last metaphorical breath - not that I'd still have anything like a human body after that amount of time, of course - I'll go out raging against the night, for a life so short compared to all the experiences I wish I could have had. You can't compare that to real immortality. As Greg Egan put it, immortality isn't living for a very long time and then dying. Immortality is just not dying, ever."
Well, I can't offer you real immortality - not in this dilemma, anyway. However, on behalf of my patron, Omega, who I believe is sometimes also known as Nyarlathotep, I'd like to make you a little offer.
If you pay me just one penny, I'll replace your 80% chance of living for 10^(10^10) years, with a 79.99992% chance of living 10^(10^(10^10)) years. That's 99.9999% of 80%, so I'm just shaving a tiny fraction 10-6 off your probability of survival, and in exchange, if you do survive, you'll survive - not ten times as long, my friend, but ten to the power of as long. And it goes without saying that you won't run out of memory (RAM) or other physical resources during that time. If you feel that the notion of "years" is ambiguous, let's just measure your lifespan in computing operations instead of years. Really there's not much of a difference when you're dealing with numbers like 10^(1010,000,000,000).
My friend - can I call you friend? - let me take a few moments to dwell on what a wonderful bargain I'm offering you. Exponentiation is a rare thing in gambles. Usually, you put $1,000 at risk for a chance at making $1,500, or some multiplicative factor like that. But when you exponentiate, you pay linearly and buy whole factors of 10 - buy them in wholesale quantities, my friend! We're talking here about 1010,000,000,000 factors of 10! If you could use $1,000 to buy a 99.9999% chance of making $10,000 - gaining a single factor of ten - why, that would be the greatest investment bargain in history, too good to be true, but the deal that Omega is offering you is far beyond that! If you started with $1, it takes a mere eight factors of ten to increase your wealth to $100,000,000. Three more factors of ten and you'd be the wealthiest person on Earth. Five more factors of ten beyond that and you'd own the Earth outright. How old is the universe? Ten factors-of-ten years. Just ten! How many quarks in the whole visible universe? Around eighty factors of ten, as far as anyone knows. And we're offering you here - why, not even ten billion factors of ten. Ten billion factors of ten is just what you started with! No, this is ten to the ten billionth power factors of ten.
Now, you may say that your utility isn't linear in lifespan, just like it isn't linear in money. But even if your utility is logarithmic in lifespan - a pessimistic assumption, surely; doesn't money decrease in value faster than life? - why, just the logarithm goes from 10,000,000,000 to 1010,000,000,000.
From a fun-theoretic standpoint, exponentiating seems like something that really should let you have Significantly More Fun. If you can afford to simulate a mind a quadrillion bits large, then you merely need 2^(1,000,000,000,000,000) times as much computing power - a quadrillion factors of 2 - to simulate all possible minds with a quadrillion binary degrees of freedom so defined. Exponentiation lets you completely explore the whole space of which you were previously a single point - and that's just if you use it for brute force. So going from a lifespan of 10^(10^10) to 10^(10^(10^10)) seems like it ought to be a significant improvement, from a fun-theoretic standpoint.
And Omega is offering you this special deal, not for a dollar, not for a dime, but one penny! That's right! Act now! Pay a penny and go from a 20% probability of dying in an hour and an 80% probability of living 1010,000,000,000 years, to a 20.00008% probability of dying in an hour and a 79.99992% probability of living 10^(1010,000,000,000) years! That's far more factors of ten in your lifespan than the number of quarks in the visible universe raised to the millionth power!
Is that a penny, friend? - thank you, thank you. But wait! There's another special offer, and you won't even have to pay a penny for this one - this one is free! That's right, I'm offering to exponentiate your lifespan again, to 10^(10^(1010,000,000,000)) years! Now, I'll have to multiply your probability of survival by 99.9999% again, but really, what's that compared to the nigh-incomprehensible increase in your expected lifespan?
Is that an avaricious light I see in your eyes? Then go for it! Take the deal! It's free!
(Some time later.)
My friend, I really don't understand your grumbles. At every step of the way, you seemed eager to take the deal. It's hardly my fault that you've ended up with... let's see... a probability of 1/101000 of living 10^^(2,302,360,800) years, and otherwise dying in an hour. Oh, the ^^? That's just a compact way of expressing tetration, or repeated exponentiation - it's really supposed to be Knuth up-arrows, ↑↑, but I prefer to just write ^^. So 10^^(2,302,360,800) means 10^(10^(10^...^10)) where the exponential tower of tens is 2,302,360,800 layers high.
But, tell you what - these deals are intended to be permanent, you know, but if you pay me another penny, I'll trade you your current gamble for an 80% probability of living 1010,000,000,000 years.
Why, thanks! I'm glad you've given me your two cents on the subject.
Hey, don't make that face! You've learned something about your own preferences, and that's the most valuable sort of information there is!
Anyway, I've just received telepathic word from Omega that I'm to offer you another bargain - hey! Don't run away until you've at least heard me out!
Okay, I know you're feeling sore. How's this to make up for it? Right now you've got an 80% probability of living 1010,000,000,000 years. But right now - for free - I'll replace that with an 80% probability (that's right, 80%) of living 10^^10 years, that's 10^10^10^10^10^10^10^1010,000,000,000 years.
See? I thought that'd wipe the frown from your face.
So right now you've got an 80% probability of living 10^^10 years. But if you give me a penny, I'll tetrate that sucker! That's right - your lifespan will go to 10^^(10^^10) years! That's an exponential tower (10^^10) tens high! You could write that as 10^^^3, by the way, if you're interested. Oh, and I'm afraid I'll have to multiply your survival probability by 99.99999999%.
What? What do you mean, no? The benefit here is vastly larger than the mere 10^^(2,302,360,800) years you bought previously, and you merely have to send your probability to 79.999999992% instead of 10-1000 to purchase it! Well, that and the penny, of course. If you turn down this offer, what does it say about that whole road you went down before? Think of how silly you'd look in retrospect! Come now, pettiness aside, this is the real world, wouldn't you rather have a 79.999999992% probability of living 10^^(10^^10) years than an 80% probability of living 10^^10 years? Those arrows suppress a lot of detail, as the saying goes! If you can't have Significantly More Fun with tetration, how can you possibly hope to have fun at all?
Hm? Why yes, that's right, I am going to offer to tetrate the lifespan and fraction the probability yet again... I was thinking of taking you down to a survival probability of 1/(10^^^20), or something like that... oh, don't make that face at me, if you want to refuse the whole garden path you've got to refuse some particular step along the way.
Wait! Come back! I have even faster-growing functions to show you! And I'll take even smaller slices off the probability each time! Come back!
...ahem.
While I feel that the Repugnant Conclusion has an obvious answer, and that SPECKS vs. TORTURE has an obvious answer, the Lifespan Dilemma actually confuses me - the more I demand answers of my mind, the stranger my intuitive responses get. How are yours?
Based on an argument by Wei Dai. Dai proposed a reductio of unbounded utility functions by (correctly) pointing out that an unbounded utility on lifespan implies willingness to trade an 80% probability of living some large number of years for a 1/(3^^^3) probability of living some sufficiently longer lifespan. I looked at this and realized that there existed an obvious garden path, which meant that denying the conclusion would create a preference reversal. Note also the relation to the St. Petersburg Paradox, although the Lifespan Dilemma requires only a finite number of steps to get us in trouble.
Loading…
Comments (187)
This tends to imply the Sadistic Conclusion: that it is better to create some lives that aren't worth living than it is to create a large number of lives that are barely worth living.
Average utilitarianism also tends to choke horribly under other circumstances. Consider a population whose average welfare is negative. If you then add a bunch of people whose welfare was slightly less negative than the average, you improve average welfare, but you've still just created a bunch of people who would prefer not to have existed. That can't be good.
There are several "impossibility" theorems that show it is impossible to come up with a way to order populations that satisfies all of a group of intuitively appealing conditions.
I read the fun theory sequence, although I don't remember it well. Perhaps someone can re-explain it to me or point me to the right paragraph.
How did you prove that we'll never run out of fun? You did prove that we'll never run out of challenges of the mathematical variety. But challenges are not the same as fun. Challenges are fun when they lead to higher quality of life. I must be wrong, but as far as I understood, you suggested that it would be fun to invent algorithms that sort number arrays faster or prove theorems that have no applications other than exercising your brain. But is this really fun for a version of me that has no worries about health, wealth, social status, etc.? And if FAI decides to impose these challenges artificially, there will be no hope of overcoming them.
No doubt, FAI will find a solution that keeps things fun for way longer than any of my schemes. But forever?
In other words, I don't see how you solved the problem of endless boredom.
(BTW, what chance of death should you accept if Omega offers you ETERNAL life?)
I wonder if the reason you think your answers are obvious is that you learned about scope insensitivity, see the obvious stupidity of that, and then jumped to the opposite conclusion, that life must be valued without any discounting whatsoever.
But perhaps there is a happy middle ground between the crazy kind of moral discounting that humans naively do, and no discounting. And even if that's not the case, if the right answer really does lie on the extreme of the space of possibilities instead of the much larger interior, I don't see how that conclusion could be truly obvious.
In general, your sense of obviousness might to turned up a bit too high. As evidence of that, there were times when I apparently convinced you of the "obvious" correctness or importance of some idea before I'd convinced myself.
Notice that, as was mentioned, torture vs. specks also works for average utilitarianism: in that case, the negative effect of torture is effectively divided by the huge number of people, making it negligible in comparison with a speck, in contrast with total utilitarianism that multiplies the effect of a speck by the number of people, making it huge in comparison with that of torture. So, it's not so much about the extent of discounting, as about asymmetric discounting, which would make the problem depend on who the tortured person is.
I misread Vladimir's comment as "torture vs. specks only works for average" ... when in fact he said "also works". So what I said was in fact already obvious to him. My apologies.
Avg utility for torture is (Nk-T)/N. Avg utility for dust specs is (Nk-bD)/N
where n is the number (3^^3^^3) who'd get dust specks, N>n is the total number of people, and per person: k is the mean utility, and T and D are the (negative) utilities of torture and a single dust speck, respectively.
For the total utility, just remove the "/(n+b)" part. There's no difference in which you should prefer under avg. vs total.
A small change - the differences between average and total utility occur in decisions on whether to create a person or not. Average utilitarians create people if their utility would be higher than average, while toatl utilitarians create people if their utility would be positive.
And if they would not, in existing, decrease anyone else's utility by enough to offset their own.
That's true. I was only addressing dust specks vs. torture, where people are neither created nor destroyed. Just saying that would have been sufficient; it's a generally sufficient condition for the preferred outcome to be the same under avg. vs total.
It's worth noting that this approach has problems of its own. The Stanford Encyclopedia of Philosophy on the Repugnant Conclusion:
(Unfortunately I haven't been able to find the referenced papers by Sikora and Anglin.).
I agree with this conclusion in principle, but I need to point out some caveats. Although the argument says a world with one person would have better average quality of life, it is implied that the world would be worse due to loneliness. A world with one person would have to make up for this in some other way. More importantly, going from our current world with lots of people to a world with one person would require killing lots of people, which is unacceptable.
I hadn't considered this before. The only decent rebuttal I can think of is to claim that negative utility lives (ones not worth living) are fundamentally different from positive utility lives (ones worth living). My first impulse is to maximize average positive utility but minimize total negative utility. Unfortunately, making this distinction raises questions about math between the two types of lives. I would say that minimizing negative utility lives trumps maximizing average positive utility, but I'm pretty sure that would make it hard to choose TORTURE instead of DUST SPECKS.
The flaws in both of these dilemmas seems rather obvious to me, but maybe I'm overlooking something.
The Repugnant Conclusion
First of all, I balk at the idea that adding something barely tolerable to a collection of much more wonderful examples is a net gain. If you had a bowl of cherries (and life has been said to be a bowl of cherries, so this seems appropriate) that were absolutely the most wonderful, fresh cherries you had ever tasted, and someone offered to add a recently-thawed frozen non-organic cherry which had been sitting in the back of the fridge for a week but nonetheless looked edible, would you take it?
"But how can you equate HYOOMAN LIIIIIVES with mere INANIMATE CHERRIES, you heartless rationalist you?" I hear someone cry (probably not one of us, but they're out there, and the argument needs to be answered).
Look, we're not talking about whether someone's life should be saved; we're talking about whether to create an additional life, starting from scratch. To suppose anything else is to make an assumption about facts not mentioned in the scenario. Why would anyone, under these circumstances, add even one life that was barely worth living, if everyone else is much better off?
I think what happens in most people's minds, when presented with conundrums like this, is that they subconsciously impose a context to give the question more meaning. In this case, the fact that we know (somehow) the quality-of-life of this one additional person implies that they already exist, somewhere -- and therefore that we are perhaps rescuing them. Who could turn that down? Indeed, who could turn down a billion refugees, rather than let them die, if we knew that we could then sustain everyone at a just-barely-positive level? Surely we would soon be able to put them to work and improve everyone's lot soon enough.
I could go on with the inquiries, but the point is this: the devil is in the details, and scenarios such as these leave us without the necessary context to make a rational decision.
I propose that this is a type of fallacy -- call it Reasoning Without Context.
Which brings me to today's main dish...
The Lifespan Dilemma
The essential fallacy here is the same: we lack sufficient context to make a rational decision. We have absolutely no experience with human lifespans exceeding even 1000 years, so how can we possibly guage the value of extending life by an almost incomprehensible multiple of that, and what are the side-effects and consequences of the technique being used?
Some further context which I would want to know before making this decision:
Math and logic deal in absolute certainties and facts; real life, which is the realm of rational decisionmaking, depends on context. You can't logically or mathematically analyze a problem with no real-world context and expect the answer to make rational sense.
I didn't vote your comment down, but I can guess why someone else did. Contradicting the premises is a common failure mode for humans attacking difficult problems. In some cases it is necessary (for example, if the premises are somehow self-contradictory), but even so people fail into that conclusion more often than they should.
Consider someone answering the Fox-Goose-Grain puzzle with "I would swim across" or "I would look for a second boat".
http://en.wikipedia.org/wiki/Fox,_goose_and_bag_of_beans_puzzle
Where did I contradict the premises?
Points 1 through 5. In general, you can understand any thought experiment someone proposes to be "trued". The doubting listener adds whatever additional hypotheses were not mentioned about Omega's about powers, trustworthiness, et cetera, until (according to their best insight into the original poster's thought process) the puzzle is as hard as the original poster apparently thought it was.
I just re-read it more carefully, and I don't see where it says that I can assume that Omega is telling the truth...
...but even if it did, my questions still stand, starting with how do I know that Omega is telling the truth? I cannot at present conceive* of any circumstances under which I would believe someone making the claims that Omega makes.
As I understand it, the point of the exercise is to show how our intuitive moral judgment leads us into inconsistencies or contradictions when dealing with complex mathematical situations (which is certainly true) -- so my point about context being important is still relevant. Give me sufficient moral context, and I'll give you a moral determination that is consistent -- but without that context, intuition is essentially dividing by zero to fill in the gaps.
It's a convention about Omega that Omega's reliability is altogether beyond reproach. This is, of course, completely implausible, but it serves as a useful device to make sure that the only issues at hand are the offers Omega makes, not whether they can be expected to pan out.
I think this one of the biggest problems with these examples. It is theoretically impossible that (assuming your current life history has finite Kolmogorov complexity) you could hoard enough evidence to trust someone completely.
To me it seems like a fundamental (and mathematically quantifiable!) about these hypothetical situations: if a rational agent (one that uses Occam's razor to model the reality) encounters a really complicated god-like being that does all kind of impossible looking things, then the agent would rather conclude that his brain is not working properly (or maybe that he is a Boltzmann brain) which would still be a simpler explanation than the assuming the reality of Omega.
Okay... this does render moot any conclusions one might draw from this exercise about the fallibility of human moral intuition.
Or was that not the point?
If the question is supposed to be considered in pure mathematical terms, then I don't understand why I should care one way or the other; it's like asking me if I like the number 3 better than the number 7.
The point is that Omega's statements (about Omega itself, about the universe, etc.) are all to be taken at face value as premises in the thought experiments that feature Omega. From these premises, you attempt to derive conclusions. Entertaining variations on the thought experiment where any of the premises are in doubt is cheating (unless you can prove that they contradict one another, thereby invalidating the entire experiment).
Omega is a tool to find your true rejection, if you in fact reject something.
So what I'm supposed to do is make whatever assumptions are necessary to render the questions free of any side-effects, and then consider the question...
So, let me take a stab at answering the question, given my revised understanding.
If you pay me just one penny, I'll replace your 80% chance of living for 10^(10^10) years, with a 79.99992% chance of living 10^(10^(10^10)) years. ...with further shaving-off of survival odds in exchange for life-extension by truly Vast orders of magnitude.
First off, I can't bring myself to care about the difference; both are incomprehensibly long amounts of time.
Also, my natural tendency is to avoid "deal sweeteners", presumably because in the real world this would be the "switch" part of the "bait-and-switch" -- but Omega is 100% trustworthy, so I don't need to worry -- which means I need to specifically override my natural "decision hysteresis" and consider this as an initial choice to be made.
Is it cheating to let the "real world" intrude in the form of the following thought?:
If, by the time 10^^3 years have elapsed, I or my civilization have not developed some more controllable means of might-as-well-be-immortality, then I'm probably not going to care too much how long I live past the end of my civilization, much less the end of the universe.
...or am I simply supposed to think of "years of life" as a commodity, like money? (The ensuing monetary analogies would seem to imply this...) Too much of anything, though -- money or time -- becomes meaningless when multiplied further.:
Time: Do I assume my friends get to come with me, and that together we will find some way to survive the inevitable maximization of entropy?
Money: After I've bought the earth, and the rights to the rest of the solar system and any other planets we're able to find with the infinite improbability drive developed by the laboratories I paid for, what do we do with the other $0.99999 x 10^^whatever? (And how do I spend the first part of that money without causing a global economic crisis that will make this one look like a slow day at the taco stand? Oh, wait, though, I'm probably supposed to assume I earned it legitimately by contributing that much value to the global economy... how??? Mind boggles, scenario fails.)
In other words... Omega can have the penny, because it's totally not about the penny, but I don't see any point in starting down the road of shaving off probability-points in exchange for orders of magnitude, no matter how large.
In fact, I'd be more inclined to go the other way, if that were an option -- reducing the likelihood of death in exchange for a shorter life. (I'm not quite clear on whether this could be reverse-extrapolated from the examples given.) I suspect a thousand years would be enough; give me that, and I can get the rest for myself. (Or am I supposed to assume that I will never be able to extend my life beyond the years Omega gives me? If so, we're getting way too mystical and into premises that seem like they would force me to revise my understanding of reality in some significant way.)
So I guess my primary answer to Eliezer's question is that I don't even start down the garden path because I'm more inclined to walk the other way.
Am I still missing anything?
Please stop allowing your practical considerations get in the way of the pure, beautiful counterfactual!
Seriously though, either you allow yourself to suspend practicalities and consider pure decision theory, or you don't. This is a pure maths problem, you can't equate it to 'John has 4 apples.' John has 3^^^3 apples here, causing your mind to break. Forget the apples and years, consider utility!
this was my initial reaction as well, ask if I can go the other way until we're at, say, 1000 years. but if you truly take the problem at face value (we're negotiating with omega, the whole point of omega is that he neatly lops off alternatives for the purposes of the thought experiment) and are negotiating for your total lifespan +- 0 then yes, I think you'd be forced to come up with a rule.
This problem seems to have an obvious "shut up and multiply" answer (take the deal), but our normal intuitions scream out against it. We can easily imagine some negligible chance of living through the next hour, but we just can't imagine trusting some dude enough to take that chance, or (properly) a period longer than some large epoch time.
Since our inability to properly grok these elements of the problem is the fulcrum on which our difficulty balances it seems more reasonable than usual to question Omega & her claims.
(This problem seems as easy to me as specks vs torture: in both cases you need to shut up and multiply, and in both cases you need to quiet your screaming intuitions - they were trained against different patterns.)
Failing to question them is another. In the political world, the power to define the problem trumps the power to solve it.
Within the terms of this problem, one is supposed to take Omega's claims as axiomatically true. p=1, not 1-epsilon for even an unimaginably small epsilon. This is unlike Newcomb's problem, where an ordinary, imaginable sort of confidence is all that is required.
Thinking outside that box, however, there's a genuine issue around the question of what it would take to rationally accept Omega's propositions involving such ginormous numbers. I notice that Christian Szegedy has been voted up for saying that in more technical language.
These are answers worth giving, especially by someone who can also solve the problem on its own terms.
Side note: ya know, it would be really nice if there was some way for a negative vote to be accompanied by some explanation of what the voter didn't like. My comment here got one negative vote, and I have no idea at all why -- so I am unable to take any corrective action either with regard to this comment or any future comments I may make.
(I suppose the voter could have replied to the comment to explain what the problem was, but then they would have surrendered their anonymity..)
That assumes those people down voting are doing so with some well thought out intention.
I would refuse the next step in the garden somewhere between reaching a 75% to 80% chance of not dying in an hour. Going from a 1/5 chance to a 1/4 chance of soon dying is huge in my mind. I'd likely stop at around 79%.
Can someone point me to a discussion as to why bounded utility functions are bad?
I must say, this is my intuition as well.
Why 78.5000000000000% and not 78.499999999999% (assuming this is what you meant by "around 79%")? For ORDERS OF MAGNITUDE more life, that sure seems an arbitrary limit.
Your axiology is arbitrary. Everyone has arbitrary preferences, and arbitrary principles that generate preferences. You are arbitrary - you can either live with that or self-modify into something much less arbitrary like a fitness maximizer, and lose your humanity.
If you were to ask me, at two different random points in time, what odds I would take to live 10^10^10^10 years or die in an hour, and what odds I would take to live 10^10^10^10^10^10 years or die in an hour, you would likely get the same answer. I can identity that one number is bigger than the other, but the difference means about as little to me as the difference between a billion dollars and a billion and one dollars.
At some point, it simply doesn't matter how much you increase the payoff, I won't take the new bet no matter how little you increase the odds against me. Where that point lies is arbitrary in the same sense as any other point where the utility of two different events times their respective probabilities balance out.
I think this is equivalent to my comment below about patching the utility function, but more pithily expressed. The difficulty lies in trying to reconcile human intuition, which deals well with numbers up to 7 or so, with actual math. If we could intuitively feel the difference between 10^10^10 and 10^10^10^10, in the same way we feel the difference between 5 and 6, we might well accept Omega's offers all the way down, and might even be justified in doing so. But in fact we don't, so we'll only go down the garden path until the point where the difference between the current probability and the original 80% becomes intuitively noticeable; and then either stop, or demand the money back. The paradox is that the problem has two sets of numbers: One too astronomically large to care about, one that starts out un-feelable but eventually hits the "Hey, I care about that" boundary.
I think the reconciliation, short of modifying oneself to feel the astronomically large numbers, is to just accept the flaws in the brain and stop the garden path at an arbitrary point. If Omega complains that I'm not being rational, well, what do I care? I've already extracted a heaping big pile of utilons that are quite real according to my actual utility function.
I disagree that it's a flaw. Discounting the future, even asymptotically, is a preference statement, not a logical shortcoming. Consider this situation:
Omega offers you two bets, and you must choose one. Bet #1 says you have a 50% chance of dying immediately, and a 50% chance of living 10 average lifespans. Bet #2 says you have a 100% chance of living a single average lifespan.
Having lived a reasonable part of an average lifespan, I can grok these numbers quite well. Still, I would choose Bet #2. Given the opportunity, I wouldn't modify myself to prefer Bet #1. Moreover, I hope any AI with the power and necessity to choose one of these bets for me, would choose Bet #2.
Yes, fair enough; I should have said "accept the way the brain currently works" rather than using loaded language - apparently I'm not quite following my own prescription. :)
At least part of the problem might be that you believe now, fairly high confidence, as an infinite set atheist or for other reasons, that there's a finite amount of fun available but you don't have any idea what the distribution is. If that's the case, then a behavior pattern that always tries to get more life as a path to more fun eventually ends up always giving away life while not getting any more potential fun.
Another possibility is that you care somewhat about the fraction of all the fun you experience, not just about the total amount. If utilities are relative this might be inevitable, though this has serious problems too.
I think that the answer to this conundrum is to be found in Joshua Greene's dissertation. On page 202 he says:
"The mistake philosophers tend to make is in accepting rationalism proper, the view that our moral intuitions (assumed to be roughly correct) must be ultimately justified by some sort of rational theory that we’ve yet to discover ... a piece of moral theory with justificatory force and not a piece of psychological description concerning patterns in people’s emotional responses."
When Eliezer presents himself with this dilemma, the neural/hormonal processes in his mind that govern reward and decisionmaking fire "Yes!" on each of a series of decisions that end up, in aggregate, losing him $0.02 for no gain.
Perhaps this is surprising because he implicitly models his "moral intuition" as sampling true statements from some formal theory of Eliezer morality, which he must then reconstruct axiomatically.
But the neural/hormonal decisionmaking/reward processes in the mind are just little bits of biology that squirt hormones around and give us happy or sad feelings according to their own perfectly lawful operation. It is just that if you interpret those happy or sad feelings as implying some utility function, you end up deriving a contradiction. The fact that Eliezer knows about ordinal numbers and Knuth notation just makes it easier for his formidable analytic subsystem to more easily see those implicit contradictions.
What is the solution?
Then just arbitrarily refuse some step. Your brain encodes your own motivations in a way that doesn't make global sense, you will have to draw a lot of arbitrary lines.
If you are not Roko, you should change your username to avoid confusion.
Omega seems to run into some very fundamental credibility problem:
Let us assume the premise of the OP, that lifetime can be equated with discrete computational operations. Furthermore, also assume that the universe (space-time/mulltiverse/whatever) can be modeled result of a computation of n operations (let us say for simplicity n=10^100, we could also assume 10^^100 or any finite number, we will just need a bit more iterations of offers then).
... after some accepted offers ... :
OMEGA: ... I'll replace your p% chance of living for 10^n years, with a 0.9999999p% chance of living 10^(10^n) years...
AGENT: Sounds nice, but I already know, what I would do first with 10^n years.
OMEGA: ???
AGENT: I will simulate my previous universe up to the current point. Inclusive this conversation.
OMEGA: What for?
AGENT: Maybe I am a nostalgic type. But even if I would not be, Given so much computational resources, the probability that I would not do it accidentally would be quite negligible.
OMEGA: Yes, but you could do even more simulations if you would take my next offer. AGENT: Good point, but how can I tell that this conversation is not already taking place in that simulation? Whatever you would tell me (even if you are outside my universe, your intervention in my world would only add a few bits of extra information to those n bits), would not even change the information content of my universe by a single order of magnitude, so it would definitely end up among my countless emulations. I don't see any conceivable way of you being able to convince me, that I am not the one emulated by a version of myself that already gained the resources in question, especially that I am determined to run these simulations just for FUN.
This situation gnaws at my intuitions somewhat less than the dust specks.
You're offering me the ability to emulate every possible universe of the complexity of the one we observe, and then some. You're offering to make me a God. I'm listening. What are these faster-growing functions you've got and what are your terms?
Did you miss that before Omega gave you the offer, you already had "the ability to emulate every possible universe of the complexity of the one we observe, and then some"?
By accepting Omega's offers, you're almost certainly giving up those 10^10,000,000,000 years of life, in exchange for some tiny probability of a longer but still finite lifespan.
No, but I evidently forgot it by the time I tacked on the 'offering to make me a God' drivel. Nevertheless, as with (I believe it was) Pys-Kosh my intuition tells me to take the deals and fast, before Omega changes his mind.
OMEGA: If you pay me just one penny, I'll replace your 80% chance of living for 10^(10^10) years, with a 79.99992% chance of living 10^(10^(10^10)) years
HUMAN: That sounds like an awful lot of time. Would you mind to write it as a decimal number
OMEGA: Here it is... Of course don't expect to read this number in less than 10 ^ 9999999990 years.
HUMAN: Nevermind... So It's such a mind boggling amount of time. If I would get bored or otherwise distressed, loose my lust for life. Am I allowed to kill myself?
OMEGA: Not really. If I'd allow that and assuming that the probability of killing yourself would be 0.000000001 in 10^10 years, then it would be almost sure that you kill yourself by the end of 10^(10^(10^10)) years
HUMAN: This sounds depressing. So my decision has the potential to confining me to grillions of years of suffering, if I'd lost my lust for life.
OMEGA: OK, I see your point, I also offer you some additional drugs to make you happy whenever you would have any distress. I also promise you to modify your brain that you will never even wish to kill yourself during these few eons.
HUMAN: Sounds great,, but I also enjoy your company very much, can I hope you to entertain me from time to time with bets like this?
OMEGA: Yes, BUT: you will never be able to enter a bet again that has a nonzero chance of survival. Otherwise you will enter too many bets over your lifetime so that it would become inevitable that you die long before the end of the lifetime.
HUMAN: Hmmm, interesting... I think I could still live with that. In this case I take your offer.
OMEGA: But wait! There's another special offer, and you won't even have to pay a penny for this one - this one is free! That's right, I'm offering to exponentiate your lifespan again, to 10^(10^(10^10,000,000,000)) years! Now, I'll have to multiply your probability of survival by 99.9999% again, but really, what's that compared to the nigh-incomprehensible increase in your expected lifespan?
HUMAN: Interesting. However that sounds like a bit irrealistic to me. How long do we know each other?
OMEGA: Why do you ask?
HUMAN: I already determined to agree to your original offer. Loosing those eons even at 0.0001% chance sounds quite risky to me. I just want to check whether I can trust you enough to expect that risk to pay off?
OMEGA: What do you mean?
HUMAN: I need some Bayesian evidence that you are really capable of keeping your promise.
OMEGA: What do you expect me to do?
HUMAN: I am in a generous mood. I think 0.000001% of that 10^(10^(10^10))) years of supporting me, answering my questions, providing company would be enough to convince me that you mean your offer seriously enough such that it is worth that 0.0001% risk of loosing the remaining 99.999999% of those eons.
OMEGA: Nice try tricking me. If I granted your wish, you'd get 10^(10^999999990) years with a potential prolongation to 10^(10^(10^10)) years.
HUMAN: Come on, I just try to be rational, how could I trust you without enough Bayesian evidence? I'd risk gazillions of eons, for a dubious payoff. That requires extraordinary amount of evidence.
OMEGA: This stinks. I don't do anything like that. Either you trust me 100% or not. Your choice.
HUMAN: No problem. Even with 100% certainty in your capabilities, I am not 100% sure about mine. You see: I've got only those puny neurons and only 10^11 of them and you wish me to make an educated choice on 10^(10^(10000000000)) years? We have already found several caveats: possibility suicide, unhappiness, bayesian evidence etc. that had to be resolved. How can I be sure that that's all? If there is some other catch that I could not think of and then I'll have to live with my decision for 10^(10^(1000000000)) years? Certainly the probability of overlooking something that would spoil those years is high enough not to risk taking the offer. Anyways what is the probability that you come up with an even more convincing offer during a period of 10^1000000000 years?
Actually, by the terms of the first bet, you are PREVENTED from taking the second. You're not allowed to suicide or take any action with nonzero chance of death.
Drugs? You don't need drugs. You just need FUN! Hey, there's a reason why I wrote that, you know.
"Drug" was just a catchy phrase for omega's guarantee to cure you out from any psychological issues the could cause you any prolonged distress.
You could insist that it is entirely impossible that you'd need it.
Would not it be a bit overconfident to make any statements on what is possible to some insanely complex and alien future self of you over a period of time which is measured by a number (in years) that takes billion to the power of billions of your current lifetime just to read?
Assuming independence, which is unreasonable.
With even very slowly growing estimates p(suicide in t years) = log ( log ... ( log (log t))) would give the human enough incentives refuse the offer at some point (after accepting some) without an extra guarantee of not dieing earlier due to suicide.
Therefore, at that point omega will have to make this offer if he wants to convince the human.
The limit as t->infinity of p(suicide in t years) is probably considerably less than 1; I think that averts your concern.
This is highly subjective...
and not the point anyways. The point is that there are too many unclear points and one can come up with a lot of questions that were not specified in the OP. For example: it is not even clear whether you die with 100% certainty once your agreed upon lifetime expires or is there still a chance that some other offer comes by? etc. Your estimted probability of suicide, omega's guarantee on that, guarantees on the quality of life, bayesian evidence on Omega, etc. These are all factors that could influence the decision,...
And once one realizes that these were all there, hidden, doubts would arise that whether a human mind should at all attempt to make such high stake decisions based on so little evidence for so much ahead in time.
I wonder if this might be repairable by patching the utility function? Suppose you say "my utility function in years of lifespan is logarithmic in this region, then log(log(n)) in this region, then (log(log(log(n)))..." and so on. Perhaps this isn't very bright, in some sense; but it might reflect the way human minds actually deal with big numbers and let you avoid the paradox. (Edit) More generally, you might say "My utility function is inverse whatever function you use to make big numbers." If Omega starts chatting about the Busy Beaver, fine, your utility function in that region is inverse-Busy-Beaver. Again, this is probably not very smart, but I suspect it's psychologically realistic; this seems to be something like the way human minds actually deal with such big numbers. In some sense that's why we invented logarithms in the first place!
On the subject of SPECKS and TORTURE, a perspective occurred to me the other day that I don't think I've seen raised, although in retrospect it's obvious so I may have missed a comment or even a whole discussion. Suppose, before making your decision, you poll the 3^^^^3 prospective victims of SPECKS? It seems to me that they will almost all say that yes, they are willing to tolerate a SPECK to save someone else from TORTURE. (Obviously in such a number you're going to find any amount of trillions who won't, but no worries, they'll be at most a few percent of the whole.) Ought you not to take their preference into account?
This (being psychologically realistic, not being my actual utility function) seems very plausible.
This form of the question considers the other people's speckings to be held fixed. (What if each is willing to suffer 25 years of torture to spare the other guy 50?)
I didn't say that their preference should be the only criterion, just that it's something to think about. As a practical matter, I do think that not many humans are going to volunteer for 25 years of torture whatever the payoff, except perhaps parents stepping in for their children.
I don't think holding other speckings constant is a bug. If you ask the 3^^^^3 people "should I choose TORTURE or SPECKS", you are basically just delegating the decision to the standard human discounting mechanisms, and likely going to get back SPECKS. That's a quite separate question from "Are you, personally, willing to suffer SPECKS to avoid TORTURE". But perhaps it can be modified a bit, like so: "Are you, personally, willing to suffer SPECKS, given that there will be no TORTURE if, and only if, at least 90% of the population answers yes?"
In the payout is computational resources with unlimited storage, then patching utility doesn't work well. If utility is sublinear in experienced time, then forking yourself increases utility.
This makes it difficult to avoid taking Omega up on the offer every time. For clarity, suppose Omega makes the offer to a group of 1.25M forked copies of you. If you turn it down, then on the average 10^6 of you live for 10^(10^10) years. If you all accept and fork a copy, then on the average 2.(10^6 - 1) of you live for 10^(10^(10^10))/2 years each. Clearly this is better; there are more of you living for longer.
The only thing that changes on the shift to 1 initial copy of you is that the (std. dev. of utilons)/(mean utilons) increases by a factor of 10^6. Unless you place a special cost on risk, this doesn't matter. If you do place such a cost on risk, then you fail to take profitable bets.
ETA: The only reason to not take the offer immediately is if you think some other Omega-esque agent is going to arrive with an even better offer, and you'd better be very sure of that before you risk loosing so much.
I am not certain that utility is linear in the number of copies of me. (Is that a fair rephrasing of your objection?) It seems to me that I should only anticipate one experience of however long a duration; however many copies there are, each one is experiencing exactly one subjective time-stream. Whatever satisfaction I take in knowing that there are other mes out there surely cannot be as large as the satisfaction in my own subjective experience. So it looks to me as though my anticipated utility should grow very sublinearly in the number of copies of me, perhaps even reaching flatness at some point, even though the total utility in the universe is probably linear in copies. What do I care about the total utility in the universe? Well, as an altruist I do care somewhat. But not to the point where it can realistically compete with what I, personally, can expect to experience!
Fair-rephrasing.
On the other hand, your patching of the utility function requires it to be bounded above as subjective time tends to infinity, or I can find a function that grows quickly enough to get you to accept 1/3^^^^3 chances. If altruistic utility from the existence of others also is bounded above by some number of subjective-you equivalents, then you are asserting that total utility is bounded above.
On a related point you do need to care equally about the utility of other copies of you; otherwise you'll maximise utility if you gain 1 utilon at an overall cost of 1+epsilon to all other copies of you. You'd defect in PD played against yourself...
Ok, I'll bite the bullet and bound my utility function, mainly perhaps because I don't recall why that's such a problem. In a finite universe, there are finitely many ways to rearrange the quarks; short of turning yourself into Orgasmium, then, there's only so many things you can discover, rearrange, build, or have fun with. And note that this even includes such cerebral pleasures as discovering new theorems in ever more abstruse mathematics, because such theorems are represented in your brain as some particular arrangement of quarks, and so there is an upper bound to how many theorems can be expressed by the matter in the Universe.
I don't understand your PD-defection dilemma; why shouldn't I defect in a PD played against a copy of myself? (Apart, that is, from all the reasons not to defect in an arbitrary PD - reputation, altruism, signalling, and so on.) What changes if you replace "a random human" with "a copy of me?" Perhaps the answer can be found in our apparently different intuitions about just how equal copies are; you say "A PD against yourself", I say "A PD against a copy of yourself". These are not quite the same thing.
Perhaps you might say, "Ah, but if you reason like that, probably your copy will reason likewise, having the same brain; thus you will both defect, decreasing the total utility." Fair enough, but it cuts both ways: If I can predict my copy by looking at my own actions, then I can decide to cooperate and be confident that he will do likewise! In effect I get to set both players' actions, but I have to choose between CC or DD, and I'd be pretty stupid to choose to defect.
Summary: Either the copy is sufficiently like me that whatever motivates me to cooperate will also motivate him; splendid, we both cooperate. Or else our experiences have caused us to diverge sufficiently that I cannot predict his actions by introspection; then there's no difference between him and some random human.
Suppose you know that you will be copied in the future and the copies will have to play PD against each other. Does the current you prefer that they cooperate against each other? I find it hard to believe the answer could be "no". So assuming that it's "yes" and you could do self-modification, wouldn't you modify yourself so that your future copies will cooperate against each other, no matter how far they've diverged?
Yes, but why should my current preferences be binding on my future selves? They presumably know more than I do. I would hate to be bound by the preferences of my 9-year-old self with regards to, say, cooties. Or, to put it differently: I have a preference in the matter, but I'm not convinced it is strong enough to require binding self-modification.
I also have this problem with your scenario: Your "no matter how far" presupposes that I can put a limit on divergence: To wit, the copies cannot diverge far enough to work around my modification. This assumption may be unwarranted. It seems to amount to saying that I am able to decide "I will never defect against myself" and make it stick; but in this formulation it doesn't look anywhere near so convincing as talking of 'self-modification'. I don't think speaking of self-modification is useful here; you should rather talk of making decisions, which is a process where we have actual experience.
That's irrelevant, because their change in preference is not caused by additional knowledge, but due to a quirk in how humans make decisions. We never had mind copying in our EEA, so we make decisions mostly by valuing our own anticipated future experiences. In other words, we're more like the first picture in http://lesswrong.com/lw/116/the_domain_of_your_utility_function/ (Presumably because that works well enough when mind copying isn't possible, and is computationally cheaper, or just an easier solution for evolution to find.)
I don't understand this. As long as we're talking about mind copying, why shouldn't I talk about self-modification?
ETA: Perhaps you mean that I should consider the inconvenient possible world where mind copying is possible, but self-modification isn't? In that case, yes, you may not be able to make "I will never defect against myself" stick.
But even in that world, you will be in competition against other minds, some of whom may be able to make it stick, and they will have a competitive advantage against you, since their copies will be able to better cooperate with each other. I don't know if that argument moves you at all.
No, what I mean is that you should taboo "self-modification" and see what happens to your argument. If I decide, today, that I will go on a diet, is that not a weaker form of self-modifying? It is an attempt to bind future selves to a decision made today. Granted, it is a weak binding, as we all know; but to say "self-modification" is just to dismiss that difficulty, assuming that in the future we can overcome akrasia. Well, the future will contain many wonderful things, but I'm not convinced a cure for weak-willedness is among them! So "self-modification" becomes, when tabooed, "decide, with additional future inventions making me able to overcome the natural temptation to re-decide"; and I think this is a much more useful formulation. The reason is that we all have some experience with deciding to do something, and can perhaps form some impression of how much help we're going to need from the future inventions; while we have zero experience with this magical "self-modification", and can form no intuition of how powerful it is.
We do have some experience with self-modification, in the form of self-modifying programs. That is, programs that either write directly into their own executable memory, or (since modern CPUs tend to prohibit this) write out an executable file and then call exec on it.
But anyway, I think I get your point.
That bullet has hidden issues for reflective consistency. You're asserting that any future you would not have wished to take Omega up on the offer again.
This seems unlikely: If you're self-modifying or continually improving, then it's likely that new things will become accessible and "fun" to do, if only in terms of new deep problems to solve. It seems very likely that your conception of the bounds of utility shift up as you become more capable. The bounds that you think are on utility probably will alter given 10^^10 years to think.
You shouldn't defect because you will regret it; in retrospect you'd choose to self-modify to be an agent that cooperates with copies of you. Actually, you'd choose to self-modify to cooperate with anything that implements such a no-later-regrets decision algorithm.
I am not certain, but I think you are confusing the pre- and post-copying selves. The pre-copy self wants to maximise utility over all the copies, because it doesn't know which one it will wake up as. Post-copying selves have additional knowledge; they know which one they are, and want to maximise their own personal utility. There doesn't seem to be any inconsistency in having preferences that change over time when additional information is added. Consider designing a feudal society which you'll then live in: If you don't know whether you're an aristocrat or a peasant, you'll give the peasants as many privileges as the economy can support, on the grounds that you're a lot more likely to wake up as a peasant. But if you then find yourself an aristocrat, you'll do your level best to raise the taxes and impose the droit d'seigneur! This is not inconsistency, it is just ordinary ignorance about the future.
It's worth pointing out that I'll never experience the total utility over all my copies. However many copies are made, my anticipation ought to be waking up as one copy and experiencing one utility. Maximising the total is my best bet only so long as I don't know which one I am.
I don't understand how I am making this assertion; could you please clarify?
Did you mean, maximizing the average? Because your decisions could also affect how many copies get created.
I was considering the number of copies as fixed, which makes the two maximisations equivalent; if it is not fixed, then sure, substitute 'average' for 'total'.
Sorry; it's apparent that what I wrote confused two issues.
The assertion is necessary if you are reflectively consistent and you don't take Omega up on offer n. If a future copy of you is likely to regret a decision not to take Omega up again, then the decision was the very definition of reflectively inconsistent.
Now we try to derive a utility function from a DT. The problem for bounded utility is that bounds on conceivable and achievable utility will only increase with time. Hence a future you will likely regret any decision you make on the basis that utility is bounded above, because your future bound on achievable utility almost certainly exceeds your current bound on conceivable utility. Hence asserting that utility is bounded above is probably reflectively inconsistent. (The "almost certainly" is, to my mind, justified by EY's posts on Fun Space)
Your example suggests that you don't consider reflective consistency to be a good idea; the peasants would promptly regret the decision not to self-modify to move from a CDT (as the aristocrat is using) to a TDT/UDT/other DT which prevents defection.
A finite amount of mass contains a finite amount of information; this is physics, not to be overcome by Fun Theory. I may be mistaken about the amount of mass the Universe contains, in which case my upper bound on utility would be wrong; but unless you are asserting that there is infinite mass, or else that there are an infinite number of ways to arrange a finite number of quarks in a bounded space, there must exist some upper bound. My understanding of Fun Theory is that it is intended to be deployed against people who consider 1000-year lifespans and say "But wouldn't you get bored?", rather than an assertion that there is actually infinite Fun to be had. But when dealing with Omega, your thought experiment had better take the physical limits into account!
As for the self-modification, I gave my thoughts on this in my exchange with Wei_Dai; briefly, try doing Rationalist Taboo on "self-modify" and see what happens to your argument.
So your scenario is that I stopped at some arbitrary point in the garden path; my future self has now reached the end of his vastly extended lifespan; and he wishes he'd taken Omega up on just one more offer? Ok, that's a regret, right enough. But I invite you to consider the other scenario where I did accept Omega's next offer, the randomness did not go my way, and I have an hour left to live, and regret not stopping one offer earlier. These scenarios have to be given some sort of weighting in my decision; the one that treats the numbers as plain arithmetic isn't necessarily any better than the one that accept immediacy bias. They are both points in decision-algorithm space. The inconsistency that turns you into a money pump lies in trying to apply both.
The fact that Omega is offering unbounded lifespans implies that the universe isn't going to crunch or rip in any finite time. Excluding them leaves you with a universe where the Hubble radius tends to infinity, which thus makes negentropy (information) unbounded above.
Self-modification is just an optimisation process over the design space for agents and run by some agent, with the constraint that only one agent can be instantiated at any time.
And regardless of what n is, only a 10^-6 portion of the (n-1)-survivors regret taking decision n. If you're in the block that's killed off by decision 1, then decisions 2,3,4,... are all irrelevant to you. Clearly attempting to apply both and applying neither consistently leads to money pumping.
Doesn't many-worlds solve this neatly? Thinking of it as 99.9999999% of the mes sacrificing ourselves so that the other 0.00000001% can live a ridiculously long time makes sense to me. The problem comes when you favor this-you over all the other instances of yourself.
Or maybe there's a reason I stay away from this kind of thing. <shrug>
There's an easier solution to the posed problem if you assume MWI. (Has anyone else suggested this solution? It seems too obvious to me.)
Suppose you are offered & accept a deal where 99 out of 100 yous die, and the survivor gets 1000x his lifetime's worth of computational resources. All the survivor has to do is agree to simulate the 99 losers (and obviously run himself) for a cost of 100 units, yielding a net profit of 900 units.
(Substitute units as necessary for each ever more extreme deal Omega offers.)
No version of yourself loses - each lives - and one gains enormously. So isn't accepting Omega's offers, as long as each one is a net profit as described, a Pareto-improving situation? Knowing this is true at each step, why would one then act like Eliezer and pay a penny to welsh on the entire thing?
I was thinking of this the other day...
Suppose that a scientist approached you and wanted to pay you $1000 to play the role of Schrödinger's cat in an open-mike-night stage performance he's putting together. Take as given that the trigger for the vial of poison will result in a many-worlds timeline split;(1) the poison is painless and instantaneous;(2) and there is nobody left in the world who would be hurt by your death (no close friends or family). You can continue performing, for $1000 a night, for as long as you want.
Personally I can't think of a reason not to do this.
(1) I'm 83% confidant that I said something stupid about Many Worlds there.
(2) No drowning or pain for your other self like in The Prestige.
That's actually an extremely strong precondition. People in modern society play positive-sum games all the time; in most interactions where people exchange one good or service for another (such as in selling their time or buying a material object for money), that leaves both participants better off.
A productive member of society killing themselves - even if they have no friends and are unlikely to make any - leaves the average surviving member of that society worse off. Many unproductive members of society (politicians come to mind) could probably become productive if they really wanted to; throwing your life away in some branches is still a waste.
None of this applies if you're a perfect egoist, of course.
The opportunity cost of dying is the utility you could be netting by remaining alive. Unless you only value the rest of your life at less than £1000, you should go for life (presuming the decay is at 50:50, adjust as required)
The result applies to MWs too, I think- taking the bet results in opportunity cost for all the future yous who die/never exist, reducing your average utility across all future worlds.
It is possible that this sort of gamble on quantum immortality will maximise utility, but it is unlikely for such a small quantity of money,
I'd argue that it's reasonable to place a $0 utility on my existence in other Everett branches; while theoretically I know they exist, theoretically there is something beyond the light-barrier at the edge of the visible universe. It's existence is irrelevant, however, since I will never be able to interact with it.
Perhaps a different way of phrasing this - say I had a duplicating machine. I step into Booth B, and then an exact duplicate is created in booths A and C, while the booth B body is vapourized. For reasons of technobabble, the booth can only recreate people, not gold bullion, or tasty filet mignons. I then program the machine to 'dissolve' the booth C version into three vats of the base chemicals which the human body is made up of, through an instantaneous and harmless process. I then sell these chemicals for $50 on ebay. (Anybody with enough geek-points will know that the Star Trek teleporters work on this principle).
Keep in mind that the universe wouldn't have differentiated into two distinct universes, one where I'm alive and one where I'm dead, if I hadn't performed the experiment (technically it would still have differentiated, but the two results would be anthropically identical). Does my existence in another Everett branch have moral significance? Suffering is one thing, but existence? I'm not sure that it does.
I think this depends on the answers to problems in anthropics and consciousness (the subjects that no one understands). The aptness of your thought experiment depends on Everett branching being like creating a duplicate of yourself, rather than dividing your measure or "degree-of-consciousness" in half. Now, since I only have the semipopular (i.e., still fake) version of QM, there's a substantial probability that everything I believe is nonsense, but I was given to understand that Everett branching divides up your measure, rather than duplicating you: decoherence is a thermodynamic process occuring in the universal wavefunction; it's not really about new parallel universes being created. Somewhat disturbingly, if I'm understanding it correctly, this seems to suggest that people in the past have more measure than we do, simply by virtue of being in the past ...
But again, I could just be talking nonsense.
One Everett branch in the past has more measure than one Everett branch now. But the total measures over all Everett branches containing humans differ only by the probability of an existential disaster in the intervening time. The measure is merely spread across more diversity now, which doesn't seem all that disturbing to me.
Hopefully this conversation doesn't separate into decoherence - though we may well have already jumped the shark. :)
First of all, I want to clarify something: do you agree that duplicating myself with a magical cloning booth for the $50 of mineral extracts is sensible, while disagreeing with the same tactic using Everett branches?
Secondly, could you explain how measure in the mathematical sense relates to moral value in unknowable realites (I confess, I remember only half of my calculus).
Thirdly, following up on the second, I was under the "semipopular (i.e., still fake) version of QM" idea that differing Everett branches were as unreal as something outside of my light cone. (This is a great link regarding relativity - sorry I don't know how to html: http://www.theculture.org/rich/sharpblue/ )
For the record, I'm not entirely certain that differeing Everett branches of myself have 0 value; I wouldn't want them to suffer but if one of the two of us stopped existing, the only concern I could justify to myself would be concern over my long-suffering mother. I can't prove that they have zero value, but I can't think of why they wouldn't.
Well, I know that different things are going to happen to different future versions of me across the many worlds. I don't want to say that I only care about some versions of me, because I anticipate being all of them. I would seem to need some sort of weighing scheme. You've said you don't want your analogues to suffer, but you don't mind them ceasing to exist, but I don't think you can do that consistently. The real world is continuous and messy: there's no single bright line between life and death, between person and not-a-person. If you're okay with half of your selves across the many worlds suddenly dying, are you okay with them gradually dropping into a coma? &c.
"Well, I know that different things are going to happen to different future versions of me across the many worlds."
From what I understand, the many-worlds occur due to subatomic processes; while we're certain to find billions of examples along the evolutionary chain that went A or B due to random-decaying-netronium-thing (most if not all of which will alter the present day), contemporary history will likely remain unchanged; for there to be multiple future-histories where the Nazis won (not Godwin's law!), there'd have to be trillions of possible realities, each of which is differentiated by a reaction here on earth; and even if these trillions do exist, then it still won't matter for the small subset in which I exist.
The googleplex of selves which exist down all of these lines will be nearly identical; the largest difference will will be that one set had a microwave 'ping' a split-second earlier than the other.
I don't know that two googleplexes of these are inherently better than a single googleplex.
As for coma - is it immediate, spontaneous coma, with no probability of ressurection? If so, then it's basically equivalent to painless death.
It just seems kind of oddly discontinuous to care about what happens to your analogues except death. I mention comas only in an attempt to construct a least convenient possible world with which to challenge your quantum immortalist position. I mean---are you okay with your scientist-stage-magician wiping out 99.999% of your analogues, as long as one copy of you exists somewhere? But decoherence is continuous: what does it even mean, to speak of exactly one copy of you? Cf. Nick Bostrom's "Quantity of Experience" (PDF).
Evidence to support your idea- whenever I make a choice, in another branch, 'I' made a the other decision, so if I cared equally about all future versions of myself, the I'd have no reason to choose one option over another.
If correct, this shows I don't care equally about currently parallel worlds, but not that I don't care equally about future sub-branches from this one.
Whenever I make a choice, there are branches that made another choice. But not all branches are equal. The closer my decision algorithm is to deterministic (on a macroscopic scale), the more asymmetric the distribution of measure among decision outcomes. (And the cases where my decision isn't close to deterministic are precisely the ones where I could just as easily have chosen the other way -- where I don't have any reason to pick one choice.)
Thus the thought experiment doesn't show that I don't care about all my branches, current and future, simply proportional to their measure.
Not all probabilities are quantum probabilities.
True, I was assuming a quantum probability.
These thought experiments all seem to require vastly more resources than the physical universe contains. Does that mean they don't matter?
As with Torture vs. Specks, the point of this is to expose your decision procedure in a context where you don't have to compare remotely commensurable utilities. Learning about the behavior of your preferences at such an extreme can help illuminate the right thing to do in more plausible contexts. (Thinking through Torture vs. Dust Specks helped mold my thinking on public policy, where it's very tempting to weigh the salience of a large benefit to a few people against a small cost to everyone.)
EDIT: It's the same heuristic that mathematicians often use when we're pondering a conjecture— we try it in extreme or limiting cases to see if it breaks.
It seems to me that you can rephrase them in terms of the resources the universe actually does contain, without changing the problem. Take SPECKS: Suppose that instead of the 3^^^^3 potential SPECKing victims, we instead make as many humans as possible given the size of the universe, and take that as the victim population. Should we expect this to change the decision?
Yes, I think it will change the decision. You need a very large number of minuscule steps to go from specs to torture, and at each stage you need to decimate the number of people affected to justify inflicting the extra suffering on the few. It's probably fair to assume the universe can't support more than say 2^250 people, which doesn't seem nearly enough.
You can increase the severity of the specking accordingly, though. Call it PINPRICKS, maybe?
What if we're wrong about the size of the universe?
But we aren't wrong about the observable universe, does it really matter to us what happens outside our interaction range?
I've read too many articles here, I saw where you were going before I finished this sentence...
I still don't buy the 3^^^3 dust specks dilemma; I think it's because a dust speck in the eye doesn't actually register on the "bad" scale for me. Why not switch it out for 3^^^3 people getting hangnails?
You mean... you didn't get it from just reading the title?
The point is to imagine the event that is the least bad, but still bad. If dust specks doesn't do it for you, imagine something else. What event you choose is not supposed to be the crucial part of the dilemma.
What if torture is your low end? Think masochists.
Well, masochists like experiencing some things generally regarded as unpleasant, in moderation. Also, AFAIK, it matters who causes them the experience. For instance, I'd be surprised to find out that a masochist routinely enters biker bars and severely annoys the biggest person around for the pleasure of getting a few broken bones.
In fact, I don't think I've ever heard of repeated broken bones, severe burns, live amputations, or water-boarding as a practiced masochistic fetish. Practiced things tend to be very mild versions of these (excepting accidents).
I don't have Eliezer's original article in hand, but I doubt “torture” meant 80 years of, say, a couple of daily spankings and wearing a dog collar.
(I'm not talking about sadistic people, but about masochistic ones; nor am I talking about fantasies, but what people actually choose to do.)
I couldn't resist posting a rebuttal to Torture vs. Dust Specks. Short version: the two types of suffering are not scalars that have the same units, so comparing them is not merely a math problem.
bwahaha. Though my initial thought is "take the deal. This seems actually easier than choosing TORTURE. If you can actually offer up those possibilities at those probabilities, well... yeah."
Unless there's some fun theoretic stuff that suggests that when one starts getting to the really big numbers, fun space seriously shrinks to the point that, even if it's not bounded, grows way way way way way slower than logarithmic... And even then, just offering a better deal would be enough to overcome that.
Again, I'm not certain, but my initial thought is "take the deal, and quickly, before the seller changes their mind"
But this is just an initial consideration. I don't think this problem is particularly nastier than SPECKS vs TORTURE, given that we can reliably juggle the teeny probabilities and establish that the seller really could offer what's being offered.
I had much the same observations.
Does that actually mean anything? Is there any number you can say this about where it's both true and worth saying?
It's true of any number, which is why it's funny.
It's only true if you're counting positive integers. If you allow rational numbers, for any X greater than zero, there are as many rational numbers between zero and X as there are rational numbers greater than X.
And either way, it still means very little.
That's the point where my limited mathematical skills sputter in disbelief. It seems to me that however many rational numbers there are between, say, zero and one, there are exactly as many between one and two, and having completely accounted for the space between zero and one thus, you can move on to numbers two and up (of which there are a great many).
The trick is that there are an infinite number of rational numbers between zero and one. When dealing with infinite sets, one way to count their members is to put them into one-to-one correspondence with some standard set, like the set of natural numbers or the set of real numbers. These two sets (i.e., the naturals and the reals) have different sizes: it turns out that the set of natural numbers cannot be put into one-to-one correspondence with the real numbers. No matter how one tries to do it, there will be a real number that has been left out. In this sense, there are "more" real numbers than natural numbers, even though both sets are infinite.
Thus, a useful classification for infinite sets is as "countable" (can be put into one-to-one correspondence with the naturals) or "uncountable" (too big to be put into one-to-one correspondence with the naturals). The rational numbers are countable, so any infinite subset of rationals is also countable. When CronoDAS says that there are as many rationals between zero and X as there are greater than X, he means that both such sets are countable.
That doesn't quite work when comparing infinite sets. It might seem surprising, but indeed, there are exactly as many rational numbers between zero and one as there are between zero and two.
The short version of the explanation:
Two infinite sets are the same size if you can construct a one-to-one correspondence between them. In other words, if you can come up with a list of pairs (x,y) of members of sets X and Y such that every member of set X corresponds to exactly one member of set Y, and vice versa, then sets X and Y are the same size. For example, the set of positive integers and the set of positive even integers are the same size, because you can list them like this:
(1,2),
(2,4),
(3,6),
(4,8),
and so on. Each positive integer appears exactly once on the left side of the list, and each positive even integer appears exactly once on the right side of the list. You can use the same function I used here, f(x)=2x, to map the rational numbers between zero and one to the rational numbers between zero and two.
(As it turns out, you can map the positive integers to the rational numbers, but you can't map them to the real numbers...)
You are not the first person to try to explain this to me, but it doesn't seem "surprising", it seems like everybody is cooperating at pulling my leg. Since I'm aware that such a conspiracy would be impractical and that I am genuinely terrible at math, I don't think that's actually happening, but the fact remains that I just do not get this (and, at this point, no longer seriously entertain the hope of learning to do so). It is only slightly less obvious to me that there are more numbers between 0 and 2 than 0 and 1, than it is that one and one are two.
To put it a little differently, while I can understand the proofs that show how you may line up all the rationals in a sensible order and thereby assign an integer to each, it's not obvious to me that that is the way you should count them, given that I can easily think of other ways to count them where the integers will be used up first. Nothing seems to recommend the one strategy over the other except the consensus of people who don't seem to share my intuitions anyway.
Imagine A is the set of all positive integers and B is the set of all positive even integers. You would say B is smaller than A. Now multiply every number in A by two. Did you just make A become smaller without removing any elements from it?
...Okay, that's weird! Clearly that shouldn't work. Thanks for the counterexample.
Well, that was the short explanation. The long one makes a little more sense. (By the way, the technical term for the number of members in a set is the cardinality of a set.)
Let's try this from a different angle.
If you have two sets, X and Y, and you can map X to a subset of Y and still have some members of Y left over, then X can't be a bigger set than Y is. In other words, a set can't "fit inside" a set that's smaller than itself. For example, {1,2,3} can fit inside {a,b,c,d}, because you can map 1 to a, 2 to b, 3 to c, and still have "d" left over. This means that {1,2,3} can't be bigger than {a,b,c,d}. It shouldn't matter how you do the mapping, because we only care about whether or not the whole thing fits. Am I making sense here?
Now, because you can map the positive even integers to a subset of the positive integers (for example, by mapping each positive integer to itself) and still have positive integers left over (all the odd ones), the set of positive even integers fits inside the set of positive integers, and so it can't be bigger than the set of positive integers.
On the other hand, the positive integers can also fit inside the positive even integers. Just map every positive integer n to the positive even integer 2*(n+1). You get the list (1,4), (2,6), (3,8), and so on. You've used up every positive integer, but you still have a positive even integer - 2 - left over. So, because the positive integers fit inside the positive even integers, so they're not bigger, either.
If the positive even integers aren't bigger than the positive integers, and the positive integers aren't bigger than the positive even integers, then the only way that could happen is if they are both exactly the same size. (Which, indeed, they are.)
So in fact, we count them both ways, get both answers, and conclude that since each answer says that it is not the case that the one set is bigger than the other, they must be the same size?
Congratulations! I think I have, if not a perfect understanding of this, at least more of one than I had yesterday! Thanks :)
You're welcome. I like to think that I'm good at explaining this kind of thing. ;) To give credit where credit is due, it was the long comment thread with DanArmak that helped me see what the source of your confusion was. And, indeed, all the ways of counting them matter. Mathematicians really, really hate it when you can do the same thing two different ways and get two different answers.
I learned about all this from a very interesting book I once read, which has a section on Georg Cantor, who was the one who thought up these ways of comparing the sizes of different infinite sets in the first place.
I think that part of the difficulty (and part of the reason that certain people call themselves infinite set atheists) stems from the fact that we have two very basic intuitions about the quantity of finite sets, and it is impossible to define quantity for infinite sets in a way that maintains both intuitions.
Namely, you can have a notion of quantity for which
(A) sets that can be set in some 1-to-1 correspondence will have the same quantity,
OR a notion of quantity for which
(B) a set that strictly contains another set will have a strictly larger quantity.
As it turns out, given the importance of functions and correspondences in basic mathematical questions, the formulation (cardinality) that preserves (A) is very natural for doing math that extends and coheres with other finite intuitions, while only a few logicians seem to toy around with (B).
So it may help to realize that for mainstream mathematics and its applications, there is no way to rescue (B); you'll just need to get used to the idea that an infinite set and a proper subset can have the same cardinality, and the notion that what matters is the equivalence relation of there existing some 1-to-1 correspondence between sets.
(B) is roughly measure theory, innit?
Yes, for some value of "roughly".
(A value of "roughly" that encompasses sets of measure zero is what I had in mind.)
My problem doesn't arise only when comparing sets such that one strictly contains another. I can "prove" to myself that there are more rational numbers between any two integers than there are natural numbers, because I can account for every last natural number with a rational between the two integers and have some rationals left over. I can also read other people "proving" that the rationals (between two integers or altogether, it hardly matters) are "countably infinite" and therefore not more numerous than the integers, because they can be lined up. I get that the second way of arranging them exists. It's just not at all clear why it's a better way of arranging things, or why the answer it generates about the relative sizes of the sets in question is a better answer.
If you come up with a different self-consistent definition of how to compare sizes of sets ("e.g. alicorn-bigger"), that would be fine. Both definitions can live happily together in the happy world of mathematics. Note that "self-consistent definition" is harder than it sounds.
There are cases where mainstream mathematical tradition was faced with competing definitions. Currently, the gamma function is the usual extension of the factorial function to the reals, but at one time, there were alternative definitions competing to be standardized.
http://www.luschny.de/math/factorial/hadamard/HadamardsGammaFunction.html
Another example: The calculus was motivated by thought experiments involving infinitesimals, but some "paradoxes" were discovered, and infinitistic reasoning was thought to be the culprit. By replacing all of the arguments with epsilon-delta analogs, the main stream of mathematics was able to derive the same results while avoiding infinitistic reasoning. Eventually, Abraham Robinson developed non-standard analysis, showing an alternative, and arguably more intuitive, way to avoid the paradoxes.
http://en.wikipedia.org/wiki/Non-standard_analysis
Thanks for that super-interesting link about factorial-interpolating functions!
The trouble is that with a little cleverness, you can account for all of the rationals by using some of the natural numbers (once each) and still have infinitely many natural numbers left over. (Left as an exercise to the reader.) That's why your intuitive notion isn't going to be self-consistent.
Sounds like you want measure instead of cardinality. Unfortunately, any subset of the rationals has measure 0, and I'm not pulling your leg either.
I don't even understand the article on measure...
The main takeaway should be that counting, or one-to-one mapping, isn't a complete approach to comparing the "sizes" of infinite sets of numbers. For example, there are obviously as many prime numbers as there are naturals, because the number N may correspond to the Nth prime and vice versa; also see this Wikipedia article. For the same reason there are as many points between 0 and 1 as there are between 0 and 2, so to compare those two intervals we need something more than counting/cardinality. This "something more" is the concept of measure, which takes into account not only how many numbers a set contains, but also where and how they're laid out on the line. Unfortunately I don't know any non-mathematical shortcut to a rigorous understanding of measure; maybe others can help.
You are guaranteed to lose me if you say things like this, especially if you put in "obviously". It's obvious to me (if false, in some freaky math way) that there are more natural numbers than prime numbers. The opposite of this statement is therefore not obvious to me.
The common-sense concept of "as many" or "as much" does not have a unique counterpart in mathematics: there are several formalizations for different purposes. In one widely used formalization (cardinality) there are as many primes as there are naturals, and this is indeed obvious for that formalization. If we take some other way of assigning sizes to number sets, like natural density, our two sets won't be equal any longer. And tomorrow you could invent some new formula that would give a third, completely different answer :-) It's ultimately pointless to argue which idea is "more intuitive"; the real test is what works in applications and what yields new interesting theorems.
It is obvious only if you've had the oddities of infinite sets hammered into you. Here's why our intuitions are wrong (the common ones I hear):
"Clearly there are more natural numbers than prime numbers. Prime numbers are a strict subset of natural numbers!" --> the strict subset thing works when everything is finite. But why? Because you can count out all the smaller set, then you have more left over in the larger set, so it's bigger. For infinite sets, though, you can't "count out all the smaller set" or equivalent.
"Okay, but if I choose an integer uniformly at random, there's a 50% chance it's a natural number and a < 50% chance it's a prime number. 50 > <50, so there are more natural numbers." --> You can't choose an integer uniformly at random.
"Really?" --> Yes, really. There are an infinite number of them, so with what probability is 42 selected? Not 0, 'cause then it won't be selected. Not >0, 'cause then the probabilities don't add to 1.
"Fine, if I start counting all the natural numbers and prime numbers (1: 1,0. 2: 2,1. 3: 3,2. 4: 4,2.) I'll find that the number of naturals is always greater than the number of primes." --> You've privileged an order, why? Instead let's start at 2, then 3, then 5, then 7, etc. Now they're equal.
"Something's still fishy." --> Yes, all of these are fine properties to think about. They happen to be equivalent for finite sets and not for infinite sets. We choose cousin_it's correspondence thing to be "size" for infinite sets, because it turns out to make the most sense. But the other properties could be interesting too.
Cardinality compares two sets using one-to-one mappings. If such a mapping exists, the two sets are equal in cardinality.
In this sense, there are as many primes as there are natural numbers. Proof: arrange the primes as an infinite series of increasing numbers. Map each prime in the series to its index in the series, which is a natural number.
This definition is mathematically simple. On the other hand, the intuitive concept of "size" where the size of the real line segment [0,1] is smaller than that of [0,2] and there are fewer primes than naturals, is much more complex to define mathematically. It is handled by measure theory, but one of the intuitive problems with measure theory is that some subsets simply can't be measured.
If I understand correctly, there really are no actual infinities in the universe, at least not inside a finite volume (and therefore not in interaction due to speed of light limits). And as far as I can make out (someone please correct me if I'm wrong), there aren't infinitely many Everett branches arising from a quantum fork in the sense that we can't physically measure the difference between sufficiently similar outcomes, and there are finitely many measurement results we can see. So the mathematical handling of infinities shouldn't ever directly map to actual events in a non-intuitive sense.
If you think [0,1] has fewer elements than [0,10], then how come each number x in [0,10] can find a unique partner x/10 in [0,1]?
It might seem unusual that the set [0,10] can be partnered with a proper subset of itself. But in fact, this property is sufficient to define the concept of an "infinite set" in standard axiomatic set theory.
I have a different but related dilemma.
Omega presents you with the following two choices:
1) You will live for at least 100 years from now, in your 20 year old body, perfect physical condition etc and you may live on later as long as you manage.
2) You will definitely die in this universe within 10 years, but you get a box with 10^^^10 bytes of memory/instructions capacitance. The computer can be programmed in any programming language you'd like (also with libraries to deal with huge numbers, etc.). Although the computer has a limit on the number of operations, it will take zero time (in your universe) to run and display the result of any computation not exceeding the limits.
Even better: it also includes a function f(s:MindBogglinglyHugeInt). If you call it, it will create a simulation of this universe with a version of you inside that is modified to guaranteed (modified?) to live there as long as you want him (within the resource limitations of 10^^^10 operations). On the screen, a chat window appears and you can talk real time to him (i.e. you) as much as you want starting at s seconds in future.
Would you choose A) or B)?
The problem goes away if you allow a finite present value for immortality. In other words, there should be a probability level P(T) s.t. I am indifferent between living T periods with probability 1, and living infinitely with probability P(T). If immortality is infinitely valued, then you run into all sorts of ugly reducto ad absurdum arguments along the lines of the one outlined in your post.
In economics, we often represent expected utility as a discounted stream of future flow utilities. i.e.
V = Sum (B^t)(U_t)
In order for V to converge, we need B to be less than zero, and U_t to grow at rate less than 1/B for all t > T for some T. If a person with such a utility function were offered the deal described in the post above, they would at some point stop accepting. If you offered a better deal, they would accept for a while, but then stop again. If you continued this process, you would converge to the immortality case, and survival probability would converge to P.
Of course, this particular form is merely illustrative. Any utility function that assigns a finite value to immortality will lead to the same result.
What I'm doing wrong? I think that one obviously should be happy with 1<(insert ridiculous amount of zeros)> years for 1 - 1:10^1000 chance of dying within an hour. In a simplistic way of thinking. I could take into account things like "What's going to happen to the rest of all sentient beings", "what's up with humanity after that", and even more importantly, If this offer were to be available for every sentient being, I should assign huge negative utility for chance of all life being terminated due to ridiculously low chance of anyone winning those extra years.
Also I could assume to justify pure time discounting due to the fact that there is no premise that defines how long I'm going to live otherwise, after declining the offer. If Omega can make that offer, why couldn't post-humans make a better one after 1000 years of research? And if they can, shouldn't I be trying to maximize my chance of living merely 1000 years longer? If there was a premise that set my lifespan on case of declining the offer("You shall live only 100 years"), I'd be in favor of ridiculously low chance of ridiculously many years, if considerations from my earlier points justify that(My strategy for choosing wouldn't endanger all sentient life within the universe etc). I think the counter-intuitiveness rises from the fact that especially now it's fairly difficult to make good estimations of "natural" lifespan expectations in absence of deals from Omega.
Eliezer said:
that seems like quite a big sacrifice to make in order to resolve Parfit's repugnant conclusion; you have abandoned consequentialism in a really big way.
You can get off parfit's conclusion by just rejecting aggregative consequentialism.
Think of the goal being stated in terms of world-histories rather than world-states. It makes more sense this way. Then, you can say that your preference for world-histories where a person is created (leading to the state of the world X) is different than for world-histories where a person is killed (starting from a different state, but leading to the same state X).
Sure, you can be a histories-preferer, and also a consequentialist. In fact you have preferences over histories anyway, really.
Hmm... Then, in what sense can you mean the top-level comment while keeping this in mind?
I meant it in a hypothetical way. I don't actually like state-consequentialism - trivially, human experiences are only meaningful as a section of the history of the universe.
... is, I am gratified to see, the same as mine.
When TORTURE v DUST SPECKS was discussed before, some people made suggestions along the following lines: perhaps when you do something to N people the resulting utility change only increases as fast as (something like) the smallest program it takes to output a number as big as N. (No one put it quite like that, which is perhaps just as well since I'm not sure it can be made to make sense. But, e.g., Tom McCabe proposed that if you inflict a dust speck on 3^^^3 people, the number of non-identical people suffering the dust speck will be far smaller than that, and that that greatly reduces the resulting disutility. Wei Dai made a proposal to do with discounting utilities by some sort of measure, related to algorithmic complexity. Etc.) Anyway, I mention all this because it may be more believable within a single person's life than when aggregated over many people: while it sure seems that 10^N years of life is a whole lot better than N years when N is large, maybe for really large N that stops being true. Note that this doesn't require bounded utilities.
If so, then it seems (to me, handwavily) like the point at which you start refusing to be led down the garden path might actually be quite early. For my part, I don't think I'd take more than two steps down that path.
As long as your U(live n years) is unbounded, then my reductio holds. With the discounting scheme you're proposing, Omega will need to offer you uncomputable amounts of lifespan to induce you to accept his offers, but you'll still accept them and end up with a 1/3^^^3 chance of a finite lifespan.
How is he going to describe to me what these uncomputable amounts of lifespan are, and how will he convince me that they're big enough to justify reducing the probability of getting them?
By using non-constructive notation, like BusyBeaver(10^n). Surely you can be convinced that the smallest program it takes to output a number as big as BusyBeaver(10^n) is of size 10^n, and therefore accept a 10-fold reduction in probability to increase n by 1?
Also, if you can't be convinced, then your utility function is effectively bounded.
Somewhere I missed something, is there something wrong with bounded utilities? Every usable solution to these manipulations of infinity get dismissed because they are bounded, if they work what is the problem?
If your utility function is in fact bounded, then there's nothing wrong with that. But the utility function isn't up for grabs. If I care about something without bound, then I can't solve the dilemma by switching to a bounded utility function; that would simply make me optimize for some metric other than the one I wanted.
What does "the utility function isn't up for grabs" mean? I think Eliezer originated that phrase, but he apparently also believes that we can be and should be persuaded by (some) moral arguments. Aren't these two positions contradictory?
(It seems like a valid or at least coherent, and potentially persuasive, argument that unbounded utility functions lead to absurd decisions.)
A notion can be constant and yet we can learn about it.
For example: "The set of all prime numbers" is clearly unchanged by our reasoning, and yet we learn about it (whether it is finite, for example). Kripke used (for a different purpose) the morning star and the evening star. The concepts are discovered to be the same concept (from scientific evidence).
The argument that unbounded utility functions lead to absurdity is also persuasive.
That seems to be a reasonable interpretation, but if we do interpret "the utility function isn't up for grabs" that way, as a factual claim that each person has a utility function that can be discovered but not changed by moral arguments and reasoning, then I think it's far from clear that the claim is true.
There could be other interpretations that may or may not be more plausible, and I'm curious what Eliezer's own intended meaning is, as well as what pengvado meant by it.
There is a sense in which anything that makes choices does have a utility function - the utility function revealed by their choices. In this sense, for example, that akrasia doesn't exist. People prefer to procrastinate, as revealed by their choice to procrastinate.
People frequently slip back and forth between this sense of "utility function" (a rather strange description of their behavior, whatever that is) and the utilitarian philosophers' notions of "utility", which have something to do with happiness/pleasure/fun. To the extent that people pursue happiness, pleasure, and fun, the two senses overlap. However, in my experience, people frequently make themselves miserable or make choices according to lawful rules (of morality, say) - without internal experiences of pleasure in following those rules.
I agree that if you can derive from my preferences a conclusion which is judged absurd by my current preferences, that's grounds to change my preferences. Though unless it's a preference reversal, such a derivation usually rests on both the preferences and the decision algorithm. In this case, as long as you're evaluating expected utility, a 1/bignum probability of +biggernum utilons is just a good deal; it's not even a bullet to be bitten. Afaict the nontrivial question is how to apply the thought experiment to the real world, where I don't have perfect knowledge or well calibrated probabilities, and want my mistakes to not be catastrophic. And the answer to that might be a decision algorithm that doesn't look exactly like expected utility maximization, but whose analogue of the utility function is still unbounded. Not that I have any more precise suggestions.
What if you aren't balancing tiny probabilities, and Omega just gives you 80% chance of 10^^3 years and asks you if you want to pay a penny to switch to 80% chance of 10^^4 ? Assuming both of those are so far into the diminishing returns end of your bounded utility function that you see a negligible (< 20% of a penny) difference between them, that seems to me like an absurd conclusion in the other direction. Just giving up an unbounded reward is a mistake too.
Does the paradox go away if we set U(death) = -∞ utilons (making any increase in the chance of dying in the next hour impossible to overcome)? Does that introduce worse problems?
but U(death in bignum years) would also be - infinity utilions then, right?
This problem was explicitly constructed as "living a long time and then dying vs living a short time and then dying."
However, this doesn't describe people's actual utility functions- people crossing the road shows they're willing to take a small risk of death for other rewards.
I think this needs a bit of refinement, but it might work. Humans have a pretty strong immediacy bias; a greater than 0.1% chance of dying in the next hour really gets our attention. Infinity is way too strong; people do stand their ground on battlefields and such. But certainly you can assign a vast negative utility to that outcome as a practical description of how humans actually think, rather than as an ideal utility function describing how we ought to think.
Since we're talking about expected utility, I'd rather you answered this old question of mine...
The problem is that using a bounded utility function to prevent this sort of thing will lead to setting arbitrary bounds to how small you want the probability of continuing to live to go, or arbitrary bounds on how long you want to live, just like the arbitrary bounds that people tried to set up in the Dust Specks and Torture case.
On the other hand, an unbounded utility function, as I have said many times previously, leads to accepting a 1/(3^^^3) probability of some good result, as long as it is good enough, and so it results in accepting the Mugging and the Wager and so on.
The original thread is here. A Google search for "wei_dai lesswrong lifetime" found it.
ETA: The solution I proposed is down in the thread here.
If I wanted to be depressing, I'd say that, right now, my utility is roughly constant with respect to future lifespan...