wedrifid comments on Bayesians vs. Barbarians - Less Wrong

51 Post author: Eliezer_Yudkowsky 14 April 2009 11:45PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (270)

You are viewing a single comment's thread. Show more comments above.

Comment author: Will_Sawin 20 June 2010 01:35:18AM -1 points [-]

I know this post is long, long dead but:

if they have common knowledge of each other's source code.

Isn't this a logical impossibility? To have knowledge is to contain it in your source code, so A is contained in B, and B is contained in A...

Alternatively, I'm considering all the strategies I could use, based on looking at my opponent's strategy, and one of them is "Cooperate only if the opponent, when playing against himself, would defect."

"Common knowledge of each other's rationality" doesn't seem to help. Knowing I use TDT doesn't give someone the ability to make the same computation I do, and so engage TDT. They have to actually look into my brain, which means they need a bigger brain, which means I can't look into their brain. If I meet one of your perfectly rational agents who cooperates on true prisoners dilemma, I'm going to defect. And win. Rationalists should win.

Comment author: wedrifid 20 June 2010 02:55:31AM 5 points [-]

Knowing I use TDT doesn't give someone the ability to make the same computation I do, and so engage TDT.

It is possible to predict the output of a system without emulating the system. We can use the idea 'of emulating their behavior' if it helps as an intuition pump but to assume that it is required is a mistake.

If I meet one of your perfectly rational agents who cooperates on true prisoners dilemma, I'm going to defect. And win.

Why on earth would I cooperate with you? You just told me you were going to defect!

(But I do respect your grappling with the problem. It is NOT trivial. Well, I should say it is trivial but it is hard to get your head around it, particularly with our existing intuitions.)