Jonii comments on Self-modification is the correct justification for updateless decision theory - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (32)
If you create your AI before you can infer from Omega's actions what the umpteenth digit of pi is, then I agree that you should create an AI that presses the button, even if the AI finds out (through Omega's actions) that the digit is in fact odd. This is because from your perspective when you create the AI, this kind of AI maximizes your expected utility (measured in humanity-years).
But if you create your AI after you can infer what the digit is (in the updated-after-your-comment version of my post, by observing that you exist and Alpha Centauri isn't purple), I argue that you should not create an AI that presses the button, because at that point, you know that's the losing decision. If you disagree, I don't yet understand why.
If you can somehow figure it out, then yes, you shouldn't press the button. If you know that the simulated you would've known to press the button when you don't, you're not anymore dealing with "take either 50% chance of world exploding right now VS. 50% chance of world exploding million years from now", but a lot simpler "I offer to destroy the world, you can say yes or no". Updateless agent would naturally want to take the winning bet if gaining that information were somehow possible.
So, if you know which digit omega used to decide his actions, and how, and you happen to know that digit, the bet you're taking is the simpler one, the one where you can simply answer 'yes' or 'no'. Observing that Earth has not been destroyed is not enough evidence though, because the simulated, non-conscious you would've observed roughly the same thing. Only if there were some sort of difference that you knew you could and would use in the simulation, like, your knowledge of umpteenth digit of pi, or color of some object in the sky(we're assuming Omega tells you this much in both cases. This about the fate of humanity, you should seriously be certain about what sort of bets you're taking.