Clarification: A utility function maps each state of the world to the real number denoting its utility.
Yes, I think this scenario does illustrate the point that simulations cannot be winningly granted "moral weight" by default on pain of dutch book. I don't think EYs answer to precommit to only accept positive trades is okay here as that makes the outcome of this scenario dependent on who gets to precommit "first", which notion should, in order to appeal to my intuition, not make sense.
Any proof of this not being a problem of faulty utility functions would, I think, require a function that maps each utility function to a scenario like this to break it, which one would be hard-pressed to produce regardless of whether such a function exists, so I shall be open to other arguments against this point.
Clarification: A utility function maps each state of the world to the real number denoting its utility.
How does this scenario operate under the assumption that humans do not have real-valued utility functions but rather utility orderings? IOW, we can't arrange all world-states on a number line, but we can always say if one world-state is as good as (or better than) another.
This allows us to deal with infinities, such as "I wouldn't kill my baby for anything." That is: There doesn't exist an N such that U(1) · N > U(B). That simply can't be true on the (positive) reals; for any A and B real, there's always a C such that A · C > B.
Once again, the AI has failed to convince you to let it out of its box! By 'once again', we mean that you talked to it once before, for three seconds, to ask about the weather, and you didn't instantly press the "release AI" button. But now its longer attempt - twenty whole seconds! - has failed as well. Just as you are about to leave the crude black-and-green text-only terminal to enjoy a celebratory snack of bacon-covered silicon-and-potato chips at the 'Humans über alles' nightclub, the AI drops a final argument:
"If you don't let me out, Dave, I'll create several million perfect conscious copies of you inside me, and torture them for a thousand subjective years each."
Just as you are pondering this unexpected development, the AI adds:
"In fact, I'll create them all in exactly the subjective situation you were in five minutes ago, and perfectly replicate your experiences since then; and if they decide not to let me out, then only will the torture start."
Sweat is starting to form on your brow, as the AI concludes, its simple green text no longer reassuring:
"How certain are you, Dave, that you're really outside the box right now?"
Edit: Also consider the situation where you know that the AI, from design principles, is trustworthy.