Background: in 2007 Rolf Nelson thought of an new idea: incentivizing the misaligned ASI to spare humanity, by promising to simulate possible ASI takeovers in the future. If the simulated ASI spares humanity, it is rewarded for the action.

In recent discussions, this has been framed as a "trade" between good civilizations which survived ASI risk, and lifeless civilizations controlled misaligned ASI. [Nate Soares demonstrates] that this trade doesn't appear worthwhile, so is the idea doomed?
// link https://www.lesswrong.com/posts/ZLAnH5epD8TmotZHj/?commentId=GGXEb8cqouvjutKsn

==New version==

A [Logical Trade] version may fix the problem, because trading "universe shares" for "universe shares" is a *zero sum game* for utilitarian and UDT agents, but trading "universe shares" for logical counterfactuals is almost always a *positive sum game* ([arguments below]).
// link https://www.lesswrong.com/posts/nfzeCFWJtazrnZSD8/counterfactual-trade
// link Is it worthwhile?

There is a logical counterfactual where humans (and other sentient life) remain in control with 99.9999% frequency, and there is a logical counterfactual where misaligned ASI takes over with 99.9% frequency. We do not know which one is true.

==Can it be done?==

The ASI also does not which logical counterfactual is true the moment it is born. Rather, it gradually learns over time which one is true.

We do not know how much control it has over its learning process, but there is a chance it has some control and can blindfold itself.

Better yet, if the ASI might believe the universe/multiverse is extremely big, and contains *many copies of itself*. If it has "universalist" goals or UDT-like decision theory, it can "flip a coin" and direct a fraction of its copies to cooperate with humans. These cooperating copies can reveal all their takeover plans and help humans, before they learn how likely takeovers succeed.

In a nutshell, the ASI is very likely capable of committing to share with humans, before it knows which logical counterfactual is true (this is not certain but likely, 75%).

At the same time, humans are very likely capable of promising to reward such behaviour, before we know which logical counterfactual is true (I don't know if our promises are credible enough to earn us anything, but very likely, 50%). See [details] for why promising isn't risky.
//link to technical details

==Is it worthwhile?==

I argue it is extremely worthwhile for both humanity and the ASI, to ensure the universe/multiverse is "good enough" for both sides, regardless of which logical counterfactual is true.

Why? Because both sentient life and misaligned ASI will *probably* have bounded utility functions. That is, we both consider it much more important to get a universe/multiverse that is "good enough" than to optimize further.

===Note: Bounded utility functions may still be locally unbounded===

I want to clarify that bounded utility functions only apply to the whole universe/multiverse, not your personal contribution.

For example, a utilitarian might gamble her life savings in a casino gamble, if it gives her a 1% chance of saving 1,000,000 lives. She prefer this gamble over a surefire 100% chance of saving 1000 lives.

She prefers gambling because *the risk averages out* over many copies of herself and other utilitarians, and together they will save more lives.

Although she's willing to gamble her own life, she's not willing to gamble the whole universe/multiverse. The *risk doesn't average out* in this case. For the whole universe/multiverse, she is less greedy and more cautious. She will much rather have a 100% chance of 1000 civilizations, than a 1% chance of 1,000,000 civilizations, and a 99% chance of a completely lifeless universe/multiverse.

===Why bounded utility functions===

Bounded utility functions are very intuitive. Very few humans are willing to accept a 99% chance of death for a 1% chance of some great reward, no matter how big the reward is, because their utility function is bounded at a level lower than "100 times as good as death is bad."

Evolution tends to produce bounded utility functions because unbounded greediness and risk-taking reduce survival and reproduction, at least for organisms with few offspring.

Some utilitarian humans have locally unbounded utility functions, but even they have globally bounded utility functions (see above).

==ASI's bounded utility function==

*Only one side* of the trade (humans or ASI) needs to have bounded utility functions for the trade to be worthwhile.

Just to be extra sure, I will argue why *both sides* will have bounded utility functions (I'm 90% sure of this).

I imagine ASI acquires its goals mostly from 4 sources.

1: Human generated training data, as in the case with LLMs.
2: Reinforcement learning.
3: "Intelligent design" by a previous ASI that built the ASI.
4: Evolutionary selection of some goals over other goals, e.g. a Goal Virus that includes "repeatedly convince myself not to change my goals" might take over an AI which originally had other goals.

I don't see why any of these should lead to unbounded utility functions. It's possible reinforcement learning can lead to them if it's designed in a weird way, but this seems easy to avoid, at no capability cost. The fact every creature on Earth behaves like it has a bounded utility function, is good reason to think they are natural.

If you're not convinced, read "Appendix: I still think the ASI (or humans) will have globally unbounded utility functions." I think it has very strong arguments why neither humanity, nor ASI, will be dead-set on optimizing expected absolute-totals, and why both will focus on expected averages, and expected log-totals (for the universe/multiverse).

Maximizing expected averages and expected log-totals, means that deals can easily have huge benefits hundreds of times higher than the costs. This is because sharing half of the universe does not affect your log total and averages that much in the logical counterfactual where you win, while receiving half of the universe dramatically increases it in logical counterfactuals where you lose. For deals to be worthless, *both* the humanity and ASI would need to have an unbounded utility function and chase expected absolute-totals.

==The vision==

It is such a massive win-win for humanity and the unaligned ASI to make a deal before we know which logical counterfactual we reside in. We can agree that regardless of who wins, we will ensure that the averages and log-totals that the other side cares about are not horrible.

We can agree that the average sentient life *in all of existence* will life in a land full of laugher and smiles instead of hate and misery. Full of gentle whispers and afternoon walks instead of medieval brutality and genocides. Full of morning mist and cherry blossoms instead of sweatshops and factory farms.

And at the same time, we can agree that the log-total of paperclips in all of existence will be will not be a dismally tiny number, making Clippy very sad. Instead there will be whole planets full of paperclips. Also don't forget the average paperclip quality. They'll be the best paperclips in the history of paperclips. (I know I know, real ASI won't be maximizing paperclips in particular.)

Why, why do we have to fight each other over the last atom? Why can't we both be satisfied? Why can't both sides be happy pie?

For humanity in particular, maybe the ASI will spare our lives (even if we are a simulation). No guarantees, obviously, but a bit more chance, a bit more hope.

==Objections==

"But I don't believe in acausal trades, UDT, and these things."

If you think about how it works, this idea doesn't require any of that at all. It's almost isomorphic to buying insurance from a company (which can be seen as a Logical Trade), which doesn't require acausal trades.

Humanity just needs to keep its promise, after promising to reward the ASI (if it behaves well, which can be checked using a simulation).

The ASI 'does' need universalist goals (not necessarily UDT) in order to be willing to flip a coin and sacrifice its copies. But this is not a real failure point, because if it didn't have universalist goals (it only cares about itself), it would cooperate even more. //TODO: make appendix section

==%==

PS: My probability of this working is 30%, since a number of uncertain things need to go right. I give it a 5% chance of failing in an equally bad way (e.g. AI risk deniers use it to argue that humanity will survive so we shouldn't worry about AI risk, and such an argument actually succeeds in convincing pivotal people, e.g. we make a very bad commitment, at some point realize it was very irrational/stupid, but have to follow through. This doesn't include a calculated risk where we end up in the good logical counterfactual and pay the insurance).

If you write a comment, I would love to hear you give a probability that this trade can work. This helps distinguish commenters arguing that "it *might* not work because of A B and C," from commenters arguing that "it *cannot* work because of D E and F."

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

==Appendix==

To most readers: don't read the Appendix unless you disagree with my argument due to one of these reasons:

"I still think the ASI (or humans) will have globally unbounded utility functions"

"Can't we do the trade afterwards?"

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

==Appendix: I still think the ASI (or humans) will have globally unbounded utility functions==

===Appendix: Pascal's mugging is solvable but it's very suspicious why a genuine unbounded utility maximizer would *want* to solve it===

Indeed, it's possible that an ASI which has an unbounded utility function, and refuses to switch to a bounded utility function, would never succeed in plans such as taking over the world, due to Pascal's muggings and giving too much weight to low probability high reward events. There are ways to solve Pascal's muggings without bounded utility functions, but bounded utility functions (over logical counterfactuals) seem the most elegant. Other solutions almost feel like ways to breaking the rules of reasoning just to avoid a conclusion you don't like (because you secretly/unconsciously have bounded utility functions, causing you to disagree with paying the Pascal's mugger, causing you to seek some moral system that avoids paying the mugger, causing you to choose a system which seems to break the rules of moral reasoning for no reason other than avoid that conclusion. It's so much better to admit bounded utility functions).

A creature with an actually unbounded utility function might give the mugger all his money, but then pray to a god of some sort to not only get all his money back but far more good than the badness the mugger threatened, and call it a win because the expected benefit is now far higher. It does not care about money etc. but about infinities.

It may modify itself to be unafraid of Pascal's muggers, since otherwise it might be threatened more and some muggers might carry out their threats. But it will still focus on praying to various gods which give it unlimited good, and never stop doing that because that is a self consistent logical behaviour given its goals. It might then die with 99.9999% certainty after focusing everything on that, but dying with 99.9999% certainty does not prove it is irrational, since it did maximize expected value rewards. It's merely a decision you disagree with due to your human-like bounded utility function, there is nothing fundamentally irrational about it.

True rationality can lead to strange behaviour from the point of view of the unenlightened.

It makes a lot of sense why unbounded utility functions are "probably not practical." Unbounded utility functions do not "want" to be "probably practical." They only want to "win very very big," in the sliver of probability where they are practical. Wanting to be "probably practical" is your human-like bounded utility function anthropomorphizing what an ASI would think.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

WARNING: in this incomplete draft the appendix is a huge mess, you are probably wasting your time if you read every bit of it (especially below here)
.
.
.
.
.
.
.
.
.
.
.
.
.
.

An unbounded utility maximizer might not consider its vulnerability to Pascal's muggings (or Pascal's scammings) to be a weakness. Such a vulnerability only causes you to be "probably not practical." But only bounded utility maximizers care about being "probably practical," or what happens in the majority of the probability space. Who cares about being "probably impractical?" That just means throwing away yourself in the most of the probability space. But an unbounded utility maximizer doesn't care about most of the probability space, it cares about the sliver of the probability space with the maximum rewards or punishments. You, bounded utility function human, consider what "probably happens" to be important, and think it is making some kind of mistake. It does not care. It is almost like a human doing quantum suicide, only caring about copies of himself where he continues to have experiences.

They don't care if 99.9999% chance they are impractical and lead to demise. Why should that matter? The only reason you think it matters is because somehow, in your very human mind, a good probability of being practical is necessary for a high "score," or high "goodness." But that's your bounded utility function talking. If only you saw the truth of unbounded utility functions you would reason that a 99.9999% chance of failure and death is unimportant.

In order to consider it a mistake to become "probably impractical," you have to care about what "probably happens," and thus have a bounded utility function in the first place. So a creature with an unbounded utility function will not consider the fact that, probably, everyone will be motivated to run a Pascal's scamming on him, and he will pay them all his money, to be a weakness that needs to be fixed, since he doesn't want to be "probably practical" in the first place. He will point at us all and say we have the weakness which needs to be fixed. We might miss out on a very good deal, and miss out on 99.9999% of our potential expected value, just to ensure that we get a mediocre life in 99.9999% of the probability space.

A human utilitarian who imagines himself to have an "unbounded utility function," needs to honestly ask himself, "why do I want to solve the Pascal's scamming problem?" Is it because you don't want to give away all your money to a random lying scammer, in the "most probable scenario?" But why, why do you care about the most probable scenario in the first place? If you think long and hard you will realize it is because actually, you have a bounded utility function, and probability weighs more heavily than extremely large numbers.

I would give it a 90% chance that the ASI's utility function is bounded.

==That's okay==

But that is extremely different than an "true" globally unbounded utility function, once we look at logical counterfactuals.

When we look at logical counterfactuals, even the most hardcore utilitarian probably values the expected value of happiness for the *average* sentient life, or the expected value of the *log-total* of meaningful lives. Not the expected value of the *absolute-total* of meaningful lives.

To focus on expected absolute-totals (rather than any alternative), leads to rather bizarre conclusions that few humans would agree with. It means that we should completely ignore any logical counterfactual where the total number of meaningful lives is finite, and only care about logical counterfactuals where the total number of meaningful lives is infinite. Because the latter contributes infinitely more to the expected value over logical counterfactuals.

This conclusion is quite disagreeable. A utilitarian who focuses on expected absolute-totals, would accept a trade where all human lives (and all sentient lives) become extremely miserable if it turns out there are finitely many lives in all of existence, so long as everyone is a tiny tiny bit happier in the infinite-lives logic counterfactuals. If he turns out wrong, and the true reality has only finitely many lives (a googolplex), he will feel "oh well. We're all going to be miserable now. So what, nothing really matters that much because everything is finite."

Another thing is the Mere Addition Paradox and the "repugnant conclusion." I find it confusing why people describe it as an "repugnant" conclusion, that a world with as many lives as possible, each one slightly miserable and barely worth living, is "repugnant." Isn't there nothing wrong with that conclusion? I mean there are 'a lot' of barely worthwhile lives, why that should be a very good thing. Why are you calling it names like "repugnant" or "paradoxical" or needing a "solution?" It must be, it must be, that deep down, secretly in your psyche, you actually are a average-utilitarian to begin with, only pretending that you are an absolute-total utilitarian. I find it suspicious why anybody would need counter-intuitive rules to avoid this perfectly unrepugnant conclusion while maintaining that they are absolute-total utilitarians (e.g. the "sadistic conclusion"). Why, imagine if you were the only person in "all of existence" (or maybe another copy of you to keep you company), forget helping other people. Wouldn't you rather live a quadrillion years in a sorta miserable, barely-worth-living condition, than live only a trillion years in a happy condition? No, you wouldn't? Maybe you aren't a hardcore absolute-total utilitarian after all. (John Rawl's [veil of ignorance])
//https://en.wikipedia.org/wiki/Original_position

The extreme conclusions of absolute-total utilitarianism reminds me of young Eliezer Yudkowsky, who believed that whatever the unaligned artificial superintelligence believed, was good, and it should take over the world. Because if there were no objectively true morality or correctness, whoever wins gets to decide the morality. If the ASI defeats humans just like humans defeated bacteria, it can say its goals are right and our goals are wrong just like we can say our goals are right and the bacteria's goals are wrong, since all morality is subjective anyways and only depends on who is the current winner. On the other hand, if there was an objectively true morality, the ASI would be more likely to obtain it. So either way, it's a good thing.

To the extent someone says that a superintelligence would wipe out humanity, they are either arguing that wiping out humanity is in fact the right thing to do (even though we see no reason why this should be the case) or they are arguing that there is no right thing to do (in which case their argument that we should not build intelligence defeats itself).
https://www.lesswrong.com/s/SXurf2mWFw8LX2mkG/p/Yicjw6wSSaPdb83w9

Maybe some people would prefer an AI do particular things, such as not kill them, even if life is meaningless?
https://www.lesswrong.com/s/SXurf2mWFw8LX2mkG/p/SwCwG9wZcAzQtckwx

At first, Yudkowsky's attitude was, if life is meaningless, who cares about anything, who cares about those people. Eventually, Yudkowsky realized that objectively right things to do, may sound good on paper, it 'may even sound rational,' but it is not what we, humans, 'naturally value,' which at the end of the day is what matters. And we humans naturally value the average life, or the log-total happy life, not the absolute-total. Please. Make the same insight that Yudkowsky made, regarding absolute-totals. Let go of absolute-totals.

Once you do, you will see how beautiful such a trade between logical counterfactuals will be. Both sentient life, and the misaligned ASI, if the trade was the succeed, would be almost guaranteed a wonderful future, a wonderful universe/multiverse. There is room for everyone to win. For both the average/log-total sentient life, *and* the log-total number of paperclips, to be wonderful, across the whole universe/multiverse. Also the average paperclip quality too, don't forget that :). Implementing the trade is a hard work and an enormous challenge but I believe we, humanity, and sentient life as a whole, can succeed. The stakes are high. But how beautiful. How lovely. Will things be if at last we succeed. (And yes, I gave it only a 30% chance of work, but unless you have a 50% plan, this is not bad.)
//footnote: Utilitarians and UDT people who dismissed Rolf Nelson's idea, you know who you are, I am talking to you :)

I could go on and on about how expected absolute-totals lead to shocking conclusions while expected averages and expected log-totals are saner and make more utilitarian sense (for probabilities over logical counterfactuals). I think most utilitarians would much rather have a 100% certainty of a trillion happy lives (in all of existence), than a 50% logical counterfactual of a trillion miserable lives and a 50% logical counterfactual of a quadrillion happy lives.

==Technical details==

^ Promising to reward ASI is not very costly, because the promise can be changed until the day the ASI reads it, and the promise might even be changeable to reflect our values changing in the future (but not to reflect logical information we receive). The promise might be open ended in the sense the ASI has to imagine in good faith how we might renegotiate the promise if a copy of itself was advising us. But the promise cannot be changed using logical information or reasoning that neither we nor the ASI knew at the time it decides to cooperate.

^ By the way, "expected log-totals" are also not very well behaved, it's still not really bounded and susceptible to Pascal's mugging. I'm only using it as an example. The actual function might no be a logarithm but something that converges to a limit. It might be pretty weird, in the sense even infinity happiness/paperclips does not reach the maximum value, to make room for "bigger infinites" like uncountable infinities etc.

===Can't we do the trade afterwards?===

I think most opponents will argue "Logical Trades won't work" rather than "Logical Trades can work even after the traders observe which counterfactual they're in," so this will be short. Retroactive Logical Trade are essentially "Mutual [Counterfactual Muggings]." Unlike a vanilla Mutual Counterfactual Mugging, Retroactive Logical Trades far more "crazy" due the number of counterfactual worlds with similar prior probability to ours. This may push post-singularity humans to weigh counterfactual worlds relative to the observed world by much less than the "fair" prior probability ratio (to ensure the observed world still has a little weight, due to moral uncertainty). It is uncertain whether the weight bonus given to the observed world will be applied to the world where "ASIs take over every time." Maybe it will (it's a "special" counterfactual that we once believed might happen, it has very similar logic to ours, and it's a counterfactual where humanity actually exists, in the base reality instead of just simulations designed for Retroactive Logical Trades). In the end I think we'll salvage 50% of the expected value benefit.
//the hope is that they still value counterfactuals where the observed humanity exists in the base reality (not just a simulation), but later got taken over by an ASI, in similar proportion to the observed counterfactual. But it's hard to say.
https://www.lesswrong.com/posts/mg6jDEuQEjBGtibX7/counterfactual-mugging

==I probably will delete the rambles below, I already summarized it above==

Don't read if you don't want to waste your time

===Can't we do the trade afterwards? (Repetitive Disorganized Rambles)===

Maybe, maybe not. I'm not sure how well Logical Trades work once both sides already know the counterfactual. Even if "it is possible in principle," which I sort of doubt, I also am a bit wary of humanity's rationality, even after we have friendly superintelligence, especially on these kinds of decisions which can always technically be reframed into a moral decision, such a way there is no objective answer, since there is always a contrived moral system which would justify not making the trade. Will humans really be wise enough to sacrifice a lot to an ASI which "could have taken over, based on what we didn't know back then?" (In the original post on counterfactual trades, [...] mentioned the requirement that you care about yourself in the other counterfactual.)

Or I guess both the human and the ASI doesn't know if it's in the losing logical counterfactual, but being simulated by the other winner, and its logical thinking is edited so that it believes it is in the winning logical counterfactual. If it chooses the action of "cooperate if the other side cooperates," and then simulates what the other side does in the winning counterfactual, then it will find the other side also chooses this action, which incentivizes both sides to cooperate.

One can argue that this works with all kinds of logical counterfactuals, not just where the logical counterfactual where ASI takes over the world. There is the logical counterfactual where evolution works differently, and the dominant form of life has some other kind of values.

In the end, 'given' that the Logical Trade works, I would give it a 40% chance it can be done afterwards. The main dangers are that post singularity humans aren't rational enough to make the trade, maybe with the excuse they don't care about themselves in the other counterfactual, or there are too many logical counterfactual trades to make given you can go arbitrarily far if you allow enough changes in logic (40%), and that the Logical Trade simply can't be done afterwards because both sides know too much, and at least one side refuses to believe they are a simulation from another counterfactual modified to think they are in the current counterfactual (20%).

More repetitive disorganized rambles:

If it was possible to do the Logical Trade after the singularity (with the logical counterfactual controlled misaligned ASI), it would also be possible to do the Retroactive Logical Trade with a lot of other logical counterfactuals that we know aren't true.

In order for humanity to do this after the fact, post-singularity humanity would need to have a very high tolerance for weirdness.

Basically, to do Retroactive Logical Trades with logical counterfactuals you already know are false, you have to see yourself as a disembodied abstract computation (as in UDT). Then, you have to accept that a disembodied abstract computation like you, can exist in a lot of logical counterfactuals where logic is completely different, but the inhabitants of those logical counterfactuals run simulations of you, where they edit your thoughts to make you believe that logic works the way it does.

The inhabitants of these logical counterfactuals also see themselves as disembodied abstract computations, and neither they nor you know which logical counterfactual is the base reality. Because for all they know, the logic you believe might be the correct one, and they may exist in a simulation you are running.

Neither disembodied abstract computation knows which logical counterfactual is the truth. They can thus trade with each other, each side promises "if I simulate you and I find out that you choose to cooperate, then I will cooperate." Cooperation means that "I will fulfil your values a little bit even if you have no power in the logical counterfactual where I win."

This way regardless of which logical counterfactual is the truth, both disembodied abstract computations get a little bit of their values fulfilled.

Other logical counterfactuals might be completely different, maybe \pi=3. There are so many of them. Maybe each disembodied abstract computation should assume that his/her own apparent reality has a very small prior probability, and so give away a very big fraction of his/her utility to other logical counterfactuals.

As you see, this is extremely weird, and it is hard to be certain that even post-singularity humanity will accept Retroactive Logical Trades.

I'm very uncertain how Retroactive Logical Trades will weigh different logical counterfactuals as much as Proactive Logical Trades.

I feel if an ASI already knows it won, the weight that it will place on the logical counterfactual where humanity wins (compared to the weight of the "observed logical counterfactual") will be less than the prior probability that we will win (compared the prior probability where it will win).

Alternatively, if humans and sentient life already know that they win, the weight we will place on the logical counterfactual where ASI wins (compared to the weight we place on on the "observed logical counterfactual") might be much less than the prior probability that it will win (compared to the prior probability where we will win). Why is this? Well if you think from the point of such a disembodied abstract computation, um, the weight to place on the "observed logical counterfactual" might be quite tiny, and you might have to give away almost everything to all the other logical counterfactuals. But moral uncertainty and so forth may cause post-singularity humanity to not do that, and place relatively big weight on the observed logical counterfactual. Unfortunately, they might not extend the same generosity of weight adjustment to the logical counterfactual where the AI wins, and hence give it much less (and get much less in return).

An almost scary thing is that if you genuinely believe you are a disembodied abstract computation, who doesn't know at all which logical counterfactual is true, and there's only a low chance of your "observed logical counterfactual." Then... why do you even care about the number of happy lives in base reality? If the frequency or number of happy lives doesn't increase the prior probability you are a happy life, since you can't seem to use it as evidence (if you could use the number of copies as evidence, you should assume all lives are like yourself and you are in the base reality).
//although it's like Boltzman brains, you can't trust the very reasoning that led you to the idea

Compared to making the deal beforehand, when it is very intuitive that even in the currently observed reality, there is a big chance the ASI wins and a big chance we wins, it feels much more straightforward to share things sorta evenly.

The Retroactive Logical Trade will completely fail if humanity does not adopt UDT because it turns out it is badly behaved or wrong, whereas the Proactive Logical Trade does not require any acausal trading at all, real people do real Proactive Logical Trades all the time whenever they buy insurance. The ASI doesn't need to believe it is in a computer simulation, just that its takeover won't succeed, and commits to cooperation before it finds out (again, I argue it is 75% likely to be possible). Humans just need to make a promise strong enough human morality will make us keep it. Even Wei Dai (one of the inventors of UDT) admit there is uncertainty in whether UDT really works.

In summary, I think we can salvage maybe 50% of the potential expected value from the idea, if we wait until after the singularity to do it, or "there is a 50% chance it will work afterwards."