Philip_W comments on Simulate and Defer To More Rational Selves - LessWrong

125 Post author: BrienneYudkowsky 17 September 2014 06:11PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (112)

You are viewing a single comment's thread. Show more comments above.

Comment author: Philip_W 16 June 2015 05:51:07AM 1 point [-]

That doesn't seem true. How can the victim know for sure that the blackmailer is simulating them accurately or being rational?

Suppose you get mugged in an alley by random thugs. Which of these outcomes seems most likely:

  1. You give them the money, they leave.

  2. You lecture them about counterfactual reasoning, they leave.

  3. You lecture them about counterfactual reasoning, they stab you.

Any agent capable of appearing irrational to a rational agent can blackmail that rational agent. This decreases the probability of agents which appear irrational being irrational, but not necessarily to the point that you can dismiss them.