Imagine 2 valleys. One valley leads to billions and billions of years of extreme joy, while the other valley leads to billions and billions of years of extreme suffering. A superintelligence comes to you and tells you that there is a 98% chance that you will end up in the valley of joy for billions of years and a 2% chance that you will end up in the valley of suffering for billions of years. It also gives you a choice to choose non-existence. Would you take the gamble, or would you choose non-existence?
The argument presented in this post occurred to me several months ago and in the last several months, I have spent time thinking about the argument and discussed it with AI models and have not found a satisfactory answer to it given the real-world situation. The argument can be formulated for things other than advanced AI, but given the rapid progress in the AI field and that the argument was originally formulated in the context of AI, I will present it in that context.
Now apply the reasoning from the valleys from above to AGI/ASI. AGI could be here in about 15 months and ASI not long after that. Advanced AI(s) could prolong human life to billions and billions of years, take over the world and create a world in its image - whatever that might be. People have various estimates of how likely it is that the AGI/ASI will go wrong, but one thing that many of them keep saying is that the worst case scenario is that it will kill us all. That is not the worst case scenario. The worst case scenario is that it will cause extreme suffering or torture us for billions and trillions of years.
Let's assume better than 2% odds, let's say they are 0.5%, would you be willing to take the gamble with heaven or hell even if the odds are 0.5% for hell? And if not, at what point would you be willing to take the gamble instead of choosing non-existence?
If some of you might say that you would be willing to take the gamble at 0.5% for a living hell, in this case, would you be willing to spend 1 hour in a real torture chamber now for every 199 hours that you are in a positive mental state and not there? Advanced ASI could not only create suffering based on the current levels of pain experience that humans can have, which are already horrible, but increase pain experience of humans to unimaginable levels (for whatever reason - misalignment, indifference, evilness, sadism, counterintuitive ethical theories, whatever it might be).
If you do not assume a punitive afterlife for choosing non-existence and if choosing non-existence is an option, at which point, what odds do you need to have to take the gamble between almost a literal heaven or hell instead of choosing non-existence? I've asked myself this question and the answer is, when it comes to extreme suffering of billions and trillions of years, the odds would have to be very, very close to zero. What are your thoughts on this? If you think that this argument is not a valid argument, can you show me where is the flaw?
To say that non-existence is a gamble too is kind of like saying that a person who does not gamble in a casino is gambling too - because they are missing on a chance to win millions of dollars - to me that is more a matter of definitions and if one wants to argue for that, sure, let's accept that every single thing in life is a gamble.
Your assertion that humans will be able to integrate over large timespans might be true given the current human brain - but here we are talking about superintelligence, even with relatively primitive AIs we are already talking about new medication and cures, superintelligence that would want to cause widespread suffering or torture you and be able to build a Dyson sphere around the sun and a thousand of other advanced technologies will be able not just to figure out how to torture you persistently (so that your brain does not adapt to the new state of constant torture) but also to increase your pain levels by 1000x - not all animals feels the same pain, and there is no reason to think that current pain experience of humans cannot be increased by a huge amount.
I don't think that it is not rational to take the gamble when the odds are 1%, much less when the odds are 20% or 49% or 70%. Let's go with 1% because I am willing to give you favourable odds - so the post asks, would you be willing to be in a torture chamber now 1 hour for every 99 hours that you are in a really happy state? We can increase that to 20 hours (20%) or what have you. And here I am talking about real extreme torture, not you have a headache. So imagine the worst torture methods that currently exist, and it is not waterboarding - check worst torture methods of history and if you are objective, whatever odds you would be willing to accept, if you say 20% or 1%, would you be willing to be really tortured for that amount of time single every day?