Eliezer_Yudkowsky comments on Robust Cooperation in the Prisoner's Dilemma - Less Wrong

69 Post author: orthonormal 07 June 2013 08:30AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (145)

You are viewing a single comment's thread. Show more comments above.

Comment author: magfrump 16 September 2013 01:55:59AM 3 points [-]

I'm coming to this quite late, and these are the notes I wrote as I read the paper, before reading the comments, followed by my notes in the comments.

Has any of the (roughest) analysis been done to bound proving time for PrudentBot? It should be fairly trivial to get very bad bounds and if those are not so bad that seems like a worthwhile thing to know (although what does "not so bad" really mean I guess is a problem.)

Is there a good (easy) reference for the statement about quining in PA (on page 6 below CliqueBot)? Under modal agents Kleene's recursion algorithm is mentioned; should probably be mentioned at first use rather than second?

The proof of theorem 5.1 is losing me where other proofs didn't; I don't understand how minimality of modal rank gives other results.

Reading the comments to answer some of my own questions: orthonormal seems to believe that PrudentBot couldn't be implemented for the LessWrong PD competition, although he did say with algorithmic proof search, would he change his opinion using Kripke semantics?

Vaniver's comment seems like a promising way of thinking about how to identify TrollBots. Or at least, the most promising thing about it I've seen; has this been pursued any further? Perhaps as a possibly-answerable question: call a bot a troll if it can't survive in a population with only itself or only itself and CB and DB (since these are the simplest bots to describe). I'll also note that PseudoTitForTatBot seems like an interesting basic candidate to play against; or say "TimeLimitTitForTatBot" which plays tit for tat until the time limit is near, so no single N can always fool it. Is there a way to (modally; if that is the right word for not-by-quoting-source-code) defeat TLTFTB?

Overall I still have no understanding of theorem 5.1 though. I'm not terribly familiar with the field in general but the other proofs were still fairly straightforward, whereas this proof loses me in the first sentence, without referencing a result I can look up either inside or outside of the paper. Was this written by a different person/at a different time from the rest of the paper? The second sentence is less bad than the first but still worse than the entire rest of the paper.

Comment author: Eliezer_Yudkowsky 16 September 2013 07:24:46PM 6 points [-]

Haskell code to run PrudentBot (quickly) and the other bots (also quickly) can be found on github here:

https://github.com/klao/provability/blob/master/modal.hs