Wei_Dai comments on Towards a New Decision Theory - Less Wrong

50 Post author: Wei_Dai 13 August 2009 05:31AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (142)

You are viewing a single comment's thread. Show more comments above.

Comment author: Wei_Dai 18 August 2009 05:10:31AM *  1 point [-]

Maybe cousin_it is right and we really have to settle this by formal math. But I'm lazy and will give words one more try. If we don't reach agreement after this I'm going to the math.

So, right now we have different intuitions. Let's say you have the correct intuition and convince everyone of it, and I have the incorrect one but I'm too stupid to realize it. So you and your followers go on to create a bunch of AIs with TDT. I go on to create an AI which is like TDT except it plays defect in PD. Lets say I pretended to be your follower and we never had this conversation, so there is no historical evidence that I would create such an AI. When my AI is born, it modifies my brain so that I start to believe I created an AI with TDT, thus erasing the last shred of evidence. My AI will then go on and win against every other AI.

Given the above, why should I change my mind now, and not win?

ETA: Ok, I realize this is pretty much the same scenario as the brain lesion one, except it's not just possible, it's likely. Someone is bound to have my intuition and be resistant to your persuasion. If you say that smart agents win, then he must be the smart one, right?