Isn't that ... irrational?
No. It produces better outcomes. That's the point.
Shouldn't a perfect Bayesian always welcome new information?
The information is welcome. It just doesn't make it sane to be blackmailed. Wei Dai's formulation frames it as being 'updateless' but there is no requirement to refuse information. The reasoning is something you almost grasped when you used the description:
your point is that agents can simply opt out of or ignore acausal trades
Acausal trades are similar to normal trades. You only accept the good ones.
Litany of Tarski; if my action is counterproductive, I desire to believe that it is counterproductive.
Eliezer doesn't get blackmailed in such situations. You do. Start your chant.
Worse still, isn't the category "blackmail" arbitrary, intended to justify inaction rather than carve reality at it's joints? What separates a precommitted!blackmailer from an honest bargainer in a standard acausal prisoner's dilemma, offering to increase your utility by rescuing thousands of potential torture victims from the deathtrap created by another agent?
This has been covered elsewhere in this thread as well as plenty of other times on the the forum since you joined. The difference isn't whether torture or destruction is happening. The distinction that matters is whether the blackmailer is doing something worse than their own Best Alternative To Negotiated Agreement for the purpose of attempting to influence you.
If the UFAI gains benefit torturing people independently of influencing you but offers to stop in exchange for something then that isn't blackmail. It is a trade that you consider like any other.
No. It produces better outcomes.
[...]
Acausal trades are similar to normal trades. You only accept the good ones.
[...]
Eliezer doesn't get blackmailed in such situations.
The difference isn't whether torture or destruction is happening. The distinction that matters is whether the blackmailer is doing something worse than their own Best Alternative To Negotiated Agreement for the purpose of attempting to influence you.
Wedrifid, please don't assume the conclusion. I know it's a rather obvious conclusion, but dammit, we're going to demonstrate it anyway.
The e...
Once again, the AI has failed to convince you to let it out of its box! By 'once again', we mean that you talked to it once before, for three seconds, to ask about the weather, and you didn't instantly press the "release AI" button. But now its longer attempt - twenty whole seconds! - has failed as well. Just as you are about to leave the crude black-and-green text-only terminal to enjoy a celebratory snack of bacon-covered silicon-and-potato chips at the 'Humans über alles' nightclub, the AI drops a final argument:
"If you don't let me out, Dave, I'll create several million perfect conscious copies of you inside me, and torture them for a thousand subjective years each."
Just as you are pondering this unexpected development, the AI adds:
"In fact, I'll create them all in exactly the subjective situation you were in five minutes ago, and perfectly replicate your experiences since then; and if they decide not to let me out, then only will the torture start."
Sweat is starting to form on your brow, as the AI concludes, its simple green text no longer reassuring:
"How certain are you, Dave, that you're really outside the box right now?"
Edit: Also consider the situation where you know that the AI, from design principles, is trustworthy.