Wei_Dai comments on General purpose intelligence: arguing the Orthogonality thesis - Less Wrong

20 Post author: Stuart_Armstrong 15 May 2012 10:23AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (156)

You are viewing a single comment's thread. Show more comments above.

Comment author: Wei_Dai 16 May 2012 10:15:20PM 2 points [-]

I wrote earlier

"ability to improve decision theory via philosophical reasoning" (as opposed to CDT-AI changing into XDT<CDT> and then being stuck with that)

XDT<CDT> (or in Eliezer's words, "crippled and inelegant form of TDT") is closer to TDT but still worse. For example, XDT<CDT> would fail to acausally control/trade with other agents living before the time of its self-modification, or in other possible worlds.

Comment author: JGWeissman 16 May 2012 11:30:22PM 0 points [-]

Ah, yes, I agree that CDT would modify to XDT<CDT> rather than TDT, though the fact that it self modifies at all shows that goal driven agents can change decision theories because the new decision theory helps it achieve its goal. I do think that it's important to consider how a particular decision theory can decide to self modify, and to design an agent with a decision theory that can self modify in good ways.