Will_Newsome comments on Decision Theories: A Semi-Formal Analysis, Part III - Less Wrong

23 Post author: orthonormal 14 April 2012 07:34PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (53)

You are viewing a single comment's thread. Show more comments above.

Comment author: orthonormal 15 April 2012 11:10:57PM *  2 points [-]

Aha, I see now what you mean. Good insight!

[EDIT: The following is false.] A clever CDT would be able to act like TDT if it considered, not the choice of whether to output C or D, but the choice of which mathematical object to output (because it could output a mathematical object that evaluates to C or D in a particular way depending on the code of Y—this gives it the option of genuinely acting like TDT would).

This has the interesting conclusion that even without the benefit of self-modification, a CDT agent with a good model of the world ends up acting more like TDT than traditional game theorists would expect. (Another example of this is here.) The version of CDT in the last post, contrariwise, is equipped with a very narrow model of the world and of its options. [End falsehood.]

I think these things are fascinating, but I think it's important to show that you can get TDT behavior without incorporating anthropic reasoning, redefinition of its actions, or anything beyond a basic kind of framework that human beings know how to program.

(By the way, I wouldn't call option 3 CliqueBot, because CliqueBots as I defined them have problems mutually cooperating with anything whose outputs aren't identical to theirs. I think it's better for Option 3 to be the TDT algorithm defined in the post.)

Comment author: Will_Newsome 16 April 2012 06:50:51AM *  4 points [-]

It seems to come up all the time that people aren't aware that CDT with a sufficiently good world model (a sufficiently accurate causal graph) is the same as TDT, even though this has been known for years. If you could address that somewhere in your sequence I think you'd save a lot of people a lot of time—it's the most common objection to standard discourse about decision theory that I've seen.

Comment author: orthonormal 16 April 2012 03:25:32PM 1 point [-]

I'll discuss it in the final post.