Perplexed comments on The Aliens have Landed! - Less Wrong

33 Post author: TimFreeman 19 May 2011 05:09PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (156)

You are viewing a single comment's thread. Show more comments above.

Comment author: Perplexed 20 May 2011 05:06:16AM *  0 points [-]

I don't think anybody is suggesting building an explicit "just say 'No' to extortion" heuristic into an AI. (I agree we do not have a good definition of "extortion" so when I use the word I use it in an intuitive sense.) We're trying to find a general decision theory that naturally ends up saying no to extortion (when it makes sense to).

That is pretty incoherent. If you are trying to come up with a general decision theory that wins and also says no to extortion, then you have overdetermined the problem (or will overdetermine it once you supply your definition). If you are predicting that a decision theory that wins will say no to extortion, then it is a rather pointless claim until you supply a definition. Perhaps what you really intend to do is to define 'extortion' as 'that which a winning decision theory says no to'. In which case, Nash has defined 'extortion' for you - as a threat which is not credible, in his technical sense.

ETA: Of course the [informal] concept of "credibility" breaks down a bit when all agents are reasoning this way. Which is why the problem is still unsolved!

Why do you say the problem is still unsolved? What issues do you feel were not addressed by Nash in 1953? Where is the flaw in his argument?

Part of the difficulty of discussing this here is that you have now started to use the word "credible" informally, when it also has a technical meaning in this context.