Wei_Dai comments on Towards a New Decision Theory - Less Wrong

50 Post author: Wei_Dai 13 August 2009 05:31AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (142)

You are viewing a single comment's thread. Show more comments above.

Comment author: Wei_Dai 15 August 2009 01:42:09AM 2 points [-]

Unless the smart players didn't defect in non-public rounds, in which case the dumb players who can only look at their behavior wouldn't become prejudiced against smart players, and everyone is happy.

But notice that this isn't evolutionarily stable. If a mutation causes a smart player to start defecting in non-public rounds, then it would have an advantage. On the other hand, smart players defecting in non-public rounds is evolutionarily stable. So either TDT also implies that smart players should play defect in non-public rounds, or TDT could never have arisen in the first place by evolution. (I'm not sure which is the case yet, but the disjunction must be true.) I conclude that "the mistaken belief that smart players will defect" isn't really mistaken.

Comment author: Eliezer_Yudkowsky 15 August 2009 06:22:54PM 4 points [-]

But notice that this isn't evolutionarily stable. If a mutation causes a smart player to start defecting in non-public rounds, then it would have an advantage.

Evolutionary stability isn't about TDT because organisms don't simulate each other. You, however, are running a very small and simple computation in your own mind when you conclude "smart players should defect on non-public rounds". But this is assuming the smart player is calculating in a way that doesn't take into account your simple simulation of them, and your corresponding reaction. So you are not using TDT in your own head here, you are simulating a "smart" CDT decision agent - and CDT agents can indeed be harmed by increased knowledge or intelligence, like being told on which rounds an Omega is filling a Newcomb box "after" rather than "before" their decision. TDT agents, however, win - unless you have mistaken beliefs about them that don't depend on their real actions, but that's a genuine fault in you rather than anything dependent on the TDT decision process; and you'll also suffer when the TDT agents calculate that you are not correctly computing what a TDT agent does, meaning your action is not in fact dependent on the output of their computation.

TDT could never have arisen in the first place by evolution

It didn't.

Evolutionary biology built humans to have a sense of honor, which isn't the same thing, but reflects our ancestral inability to calculate the unobserved rounds with exactitude.

TDT can arise in many ways - e.g. a CDT agent who believes they will in the future face Newcomblike problems will self-modify to use TDT for all Newcomblike problems dependent on decisions made after the instant of CDT self-modification, i.e., "use TDT for problems dependent on my decision after 9am on Tuesday and CDT for all problems dependent on decisions before then". This is inelegant, and a simple application of the unknown meta-decision-theory that wakes up and realizes this is stupid, says "Just use TDT throughout". A true pure CDT agent would never realize this and would just end up with an ugly and awkward decision theory in descendants, which points up the importance of the meta-problem.

But evolutionary dynamics simply are not decision-theory dynamics. You might as well point out that no interstellar travel could arise by evolutionary biology because there's no incremental advantage to getting halfway to another solar system.

Comment author: Wei_Dai 16 August 2009 03:35:56AM *  1 point [-]

I think my earlier comments may not have been as clear as they could be. Let me back off and try again. We should distinguish between two different questions:

  1. Is my article correct and relevant within the context of the past evolution of intelligence?
  2. What happens from now on?

I don't think you've given any arguments against 1. Since TDT didn't arise from evolution, and it wasn't invented until recently, clearly TDT-related arguments aren't relevant as far as question 1 is concerned. So again, I see no reason to retract the article.

As for 2, I have some doubts about this:

"Just use TDT throughout"

I'm trying to explore it using this puzzle. Do you have any thoughts on it?

Comment author: cousin_it 25 May 2010 05:32:06PM *  0 points [-]

Woah, it took me a long time to parse "Smart Losers". The technical parts of the article seem to be correct, but as for its evolutional relevance... In your scenario, being smart doesn't hurt you, being known to be smart does; so it's most advantageous to be "secretly smart". So if your conclusions were correct, we'd probably see many adaptations aimed at concealing our intelligence from people we interact with.

Comment author: Wei_Dai 04 September 2010 04:20:27PM 0 points [-]

So if your conclusions were correct, we'd probably see many adaptations aimed at concealing our intelligence from people we interact with.

Not if the cost of concealing intelligence was too high. Our ancestors lived in tribes with a lot of gossip. Trying to conceal intelligence would have entailed pretending to be dumb at virtually all times, which implies giving up most of the benefits of being intelligent.

Comment author: gwern 04 February 2011 09:37:20PM 0 points [-]

Trying to conceal intelligence would have entailed pretending to be dumb at virtually all times, which implies giving up most of the benefits of being intelligent.

There would still be benefits if your model is at all accurate and there are 'secret rounds' in ordinary human life. Just pretend to be stupid in public and then be smart in private rounds. To frustrate this, one would need to assume that the additional smartness costs too much. (It is so expensive that it outweighs the gains, or the gains are minimal so any cost outweighs them.)

It seems reasonable to me that there are private rounds in real life and that smartness is a net win.