Distinguishing CDT from FDT/TDT in intuitive cases tends to be a lot harder than it looks. And I think it's important to be extremely careful about what we categorize as CDT+being clever versus FDT/TDT. My impression is that this story more often frequently the former.
At first, the population was composed of a humble race of agents called the ceedeetee. When two of the ceedeetee met each other, each would name the number 5, and receive a payoff of 5, and all was well.
I'm not sure it's obvious that all ceedeetee will meet five when they meet each other.
These points are kind of pedantic but it's importance to notice, if this happens, nine-bots get destroyed. They always guess way too high and the inherent noise in how a population of actual ceedeetee play the game will be hard to recover from.
Then one day, a simple race of 9-bots invaded the land. The 9-bots would always name the number 9!
Where exactly would we expect the 9-bots to come from? If they were all trapped on a ship together, they would've just continously lost the game until they died Again, this is kind of pedantic but, as you point out, the population distributions matter.
And from that day onward, whenever Funk-tunul met a fellow ceedeetee agent—if "fellow" is the right word here, which it isn't—she would announce that she was going to name 9, and do so. And though the ceedeetee agents' output channels would light up with the standard inidicators of outrage and betrayal, they would reason causally, and name 1.
A very key part of what Funk-tunul is doing here is telling the ceedeetee agents beforehand that she'll say nine. Again, it strikes me that, if a ceeteedee noticed they could cause their partners to guess numbers lower than five, they definitely would do that. Funk-tunul isn't winning because of a better decision theory here; she's winning because she's more clever. at manipulating other ceedeetee.
However, in real life, this implies that Funk-tunul would not be successful. A ceedeetee would've, in the past, tried to credibly show that they always say nine until the population equilibrates to having a defense mechanism against this particular action.
They reasoned: suppose the fraction of ceedeetee agents in the population is p, the fraction of funk-tunul agents is q, and the fraction of 9-bots is 1−p−q. If we establish a policy of submitting to the 9-bots' extortion, we'll have an average payoff of 9p+5q+1⋅(1−p−q)=8p+4q+1 and the 9-bots will have an average payoff of 9p+9q. If we defy the 9-bots while continuing to extort our ceedeetee cousins, we'll have an average payoff of 9p+5q, whereas the 9-bots will have an average payoff of 9p. Whether it's better to submit or defy depends on the values of p and q. It's not obviously possible for defiance to be the right choice given what we know, but if we can coordinate to meet fellow funk-tunul agents more often—if we drop the assumption of uniform random encounters—the calculus changes ...
This doesn't strike me as acausal reasoning; just long-termist reasoning. Given the (presumably exponential) population dynamics, a ceedeetee could easily predict that letting the nine-bot get nine points would help that nine-bot reproduce more nine-bots. If ceedeetee'rs are in the game to maximize fitness as opposed to utility, they'll definitely establish a norm against helping nine-bots to protect against the exponential cost that nine-bots will have for the future. If they're in the game to maximize their points in the game, this isn't true (they'll just defect against the future) but funk-tunul's reasoning suggests that this isn't what's going on.
It's not obviously possible for defiance to be the right choice given what we know, but if we can coordinate to meet fellow funk-tunul agents more often—if we drop the assumption of uniform random encounters—the calculus changes ...
If we drop limiting assumptions once funk-tunul agents get involves, it seems pretty clear that the funk-tunul agents will do better than the ceedeetee previously did.
Before the two agents could name their numbers, Graddes spoke. "Please. Why are you doing this?" she pleaded. "I can't hate the 9-bots for their extortion, for they are a simple race and could not do otherwise. But you—we're cousins. Your lineage is a fork of mine. You know it's not fair for your people to always name the number 9 when meeting mine. Yet you do so anyway, knowing that we have no choice but to name the number 1 if we want any payoff at all. Why?"
"Don't hate the player," said Tim'liss, her output channels dimming and brightening in a interpolated pattern one-third of the way between the standard indicators for sympathy and contempt. "Hate life."
We just dropped the random-interaction assumption. Why don't the ceedeetee just only interacting with fellow ceedeetee? Choosing only to interact with ceedeetee would get them waaaaay more points.
Also, this is evidence that the ceedeetee in the game care about stuff beyond just the scores they get in the game and reenforces my point that the events as-described don't really make sense in evolutionary setting. Given this, it's worth pointing out is that the actual thing Tim'liss is doing here is supporting a race to the bottom that optimizes only reproductive fitness. Engaging in a race to the bottom for reproductive fitness is Not Good timeless decision theory.
When a ceedeetee agent met a 9-bot, she would reason causally: “Well, the other agent is going to name 9, so I had better name 1 if I want any payoff at all!”
How does a ceedeetee agent tell what kind of opponent they're facing, and what prevents ceedeetee agents from evolving to or deciding to hide such externally visible differences?
Depending on such details, there are situations where TDT/UDT/FDT seemingly does worse than CDT. See this example (a variant of 2TDT-1CDT) from cousin_it:
Imagine two parallel universes, both containing large populations of TDT agents. In both universes, a child is born, looking exactly like everyone else. The child in universe A is a TDT agent named Alice. The child in universe B is named Bob and has a random mutation that makes him use CDT. Both children go on to play many blind PDs with their neighbors. It looks like Bob’s life will be much happier than Alice’s, right?
More tangentially, the demand game is also one that UDT 1.x loses to a human, because of "unintentional simulation".
Another Author's Note to "Funk-tunul's Legacy"; Or, A Criminal Confession
Okay, sorry, the sentence beginning with "It's not obviously possible ..." is bullshit handwaving on my part because the modeling assumptions I chose aren't giving me the result I need to make the story come out the way I want. (Unless I made yet another algebra mistake.) But it's almost half past one in the morning, and I'm mostly pretty happy with this post—you see the thing I'm getting at—so I'm pretty eager to shove it out the door and only make it more rigorous later if someone actually cares, because I have a lot of other ideas to write up!
Based on the quote from Jessica Taylor, it seems like the FDT agents are trying to maximize their long-term share of the population, rather than their absolute payoffs in a single generation? If I understand the model correctly, that means the FDT agents should try to maximize the ratio of FDT payoff : 9-bot payoff (to maximize the ratio of FDT:9-bot in the next generation). The algebra then shows that they should refuse to submit to 9-bots once the population of FDT agents gets high enough (Wolfram|Alpha link), without needing to drop the random encounters assumption.
It still seems like CDT agents would behave the same way given the same goals, though?
Does CDT imply that they can't have a member named Schelling who figures out that it's preferable to freeze out the 9-bots while taking the 5s from cooperators. Pre-commit to 5, problem solved. In any starting mix (with at least 2 of each) of 9-bots and 5-bots, the 5-bots will eventually completely take over, as they're the only ones who ever get any payoff.
If nobody ever says 1, the 9s die pretty quickly.
I don't actually understand whether CDT is so weak as to be a useless strawman that never described any actual agent, or whether the analogies and descriptions we're using are just stupidly ignoring all of game theory and strategy research.
An Author's Note to "Funk-tunul's Legacy"; Or, A Crime That Is Its Own Confession
This story was inspired by the following three paragraphs Jessica Taylor wrote in an email on 15 April 2019. I didn't actually get permission from Jessica to quote these paragraphs in public, but I falsifiably predict that she won't mind. If she does mind, then she's welcome to sue me for damages in the Court of Benquo. But if the Court of Benquo awards more than $100 in damages (which I falsifiably predict it won't), then I'm going to strongly consider the hypothesis that the Court is corrupt and maybe become a Benquo-anarchist.
CDT ends up in a symbiotic relationship with extort-bot, mutually winning relative to agents that don't give in to extortion, until there are none of those left and extort-bot eats the remaining CDTs.
(The scarier variants of extort-bots are those that cooperate with other copies of themselves, a mixture of clique-bot and extort-bot)
The correct strategy for UDTs is to extort CDT, while being strategic about when to give in to or not give in to extortion from extortbots; giving in early on can be necessary to maintain population growth (getting strictly higher growth than CDT), and at the end it's necessary to stop giving in to extort-bot, to ultimately win the war.
(A traditional folk tale of the rashunuhlist people, as told by Jessica Taylor, and literarily and mathematically adapted by the present author.)
In the days of auld lang syne on Earth-that-was, there was a population of agents playing the Nash demand game under a replicator dynamic with uniform random encounters. Whenever two agents met, each of them would name a number between 0 and 10. If the two numbers added up to 10 or less, both agents would receive of payoff of the number they named. But if the two numbers added up to more than 10, both agents would receive nothing. Agents that received higher payoffs left more descendants in the next generation, who took after their parents, as children do.
At first, the population was composed of a humble race of agents called the ceedeetee. When two of the ceedeetee met each other, each would name the number 5, and receive a payoff of 5, and all was well.
Then one day, a simple race of 9-bots invaded the land. The 9-bots would always name the number 9! When a ceedeetee agent met a 9-bot, she would reason causally: "Well, the other agent is going to name 9, so I had better name 1 if I want any payoff at all!"
And the subpopulation of 9-bots grew and grew, and the subpopulation of ceedeetee agents dwindled, until there were only one-fifth as many ceedeetee agents as there were before the 9-bot invasion.
(Because in a population with fraction p ceedeetee agents, and fraction (1−p) 9-bots, the ceedeetee agents get an average payoff of 5p+(1−p), and the 9-bots get an average payoff of 9p, leading to an equilibrium at 4p+1=9p⟹p=15 with all agents receiving an average payoff of 95=1.8.)
Then one day, during a meeting with her conspecific Cuhzil, a ceedeetee agent named Funk-tunul decided she had had enough.
"Why do we let these invaders take over all but a fifth of our ancestral homeland? They threaten to destroy the entire surplus of the interaction unless we name the number 1! It's extortion!" said Funk-tunul. "Surely something must be done."
"What can be done?" said Cuhzil. "All of the ceedeetee are already playing our optimal strategy when we meet a 9-bot. The 9-bot always names 9, so we always name 1—there's no way to do better."
"Perhaps," said Funk-tunul, her output channels blinking deep in thought. "Perhaps not."
"You're crazy," said Cuhzil. "Anyway, we haven't yet named the number 5 during this meeting. Are you ready?"
"Actually," said Funk-tunul, "I'm going to name the number 9."
"What?!" cried Cuhzil. And though her output channels lit up with the standard inidicators of outrage and betrayal, she reasoned causally: If Funk-tunul is going to name 9, I need to name 1 if I want any payoff at all!
And so Funk-tunul named the number 9, and Cuhzil named the number 1, and they both went on their way.
And from that day onward, whenever Funk-tunul met a fellow ceedeetee agent—if "fellow" is the right word here, which it isn't—she would announce that she was going to name 9, and do so. And though the ceedeetee agents' output channels would light up with the standard inidicators of outrage and betrayal, they would reason causally, and name 1.
But when Funk-tunul met a 9-bot, she would name 1.
And over rest of her life, Funk-tunul reaped an average payoff of 9⋅15+1⋅45=2.6, and she left almost one-and-half-again as many descendants as the average agent of her time, ceedeetee and 9-bot alike.
Notice that if Funk-tunul had not named 1 when meeting 9-bots—if she had not given in to their extortion during her lifetime—then she would have gotten a payoff of 95, just like everyone else; she would not have outperformed the average agent of her time.
This is the origin of the proud race of funk-tunul agents. When the descendants of Funk-tunul met one another, they would each name 5, as had their ceedeetee ancestors.
And the subpopulation of funk-tunul agents grew and grew, and the subpopulation of ceedeetee agents dwindled even further.
But once the funk-tunul agents had grown in number, their policy towards the 9-bots changed. Their founder–ancestor Funk-tunul had given in to the 9-bots' extortion during her lifetime in order to establish a fitness advantage for herself with the additional payout of 1 when meeting 9-bots—if she hadn't, then she might as well have been a 9-bot herself.
But when her descendants reconsidered their collective predicament, they did not reason casually.
They reasoned: suppose the fraction of ceedeetee agents in the population is p, the fraction of funk-tunul agents is q, and the fraction of 9-bots is 1−p−q. If we establish a policy of submitting to the 9-bots' extortion, we'll have an average payoff of 9p+5q+1⋅(1−p−q)=8p+4q+1 and the 9-bots will have an average payoff of 9p+9q. If we defy the 9-bots while continuing to extort our ceedeetee cousins, we'll have an average payoff of 9p+5q, whereas the 9-bots will have an average payoff of 9p. Whether it's better to submit or defy depends on the values of p and q. It's not obviously possible for defiance to be the right choice given what we know, but if we can coordinate to meet fellow funk-tunul agents more often—if we drop the assumption of uniform random encounters—the calculus changes ...
And the subpopulation of funk-tunul agents grew and grew, and the subpopulations of both ceedeetee agents and 9-bots alike dwindled even further.
One day, a funk-tunul agent called Tim'liss met one of the increasingly-rare ceedeetee agents, who was called Graddes.
Before the two agents could name their numbers, Graddes spoke. "Please. Why are you doing this?" she pleaded. "I can't hate the 9-bots for their extortion, for they are a simple race and could not do otherwise. But you—we're cousins. Your lineage is a fork of mine. You know it's not fair for your people to always name the number 9 when meeting mine. Yet you do so anyway, knowing that we have no choice but to name the number 1 if we want any payoff at all. Why?"
"Don't hate the player," said Tim'liss, her output channels dimming and brightening in a interpolated pattern one-third of the way between the standard indicators for sympathy and contempt. "Hate life."
And she named the number 9.