Vladimir_Nesov comments on A Problem About Bargaining and Logical Uncertainty - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (45)
You will have enough computing power later.
I mean suppose Omega gives you the option (now, when you don't have enough computing power to compute the millionth digit of pi) of replacing yourself with another AI that has a different decision theory, one that would later give control of the universe to the staples maximizer. Should you take this option? If not, what decision theory would refuse it? (Again, from your current perspective, taking the option gives you 1/2 "logical" probability of 10^20 paperclips instead of 1/2 "logical" probability of 10^10 paperclips. How do you justify refusing this?)
(continuing from here)
I've changed my mind back. The 10^20 are only on the table for the loser, and can be given by the winner. When the winner/loser status is unknown, a winner might cooperate, since it allows the possibility of being a loser and receiving the prize. But if the winner knows own status, it can't receive that prize, and the loser has no leverage. So there is nothing problematic about 10^20 becoming inaccessible: it is only potentially accessible to the loser, when the winner is weak (doesn't know own status), while an informed winner won't give it away, so that doesn't happen. Resolving logical uncertainty makes the winner stronger, makes the loser weaker, and so the prize for the loser becomes smaller.