Congrats on publishing the book. It seems that many of the concepts you discuss seem well-known (especially within EA and LW). However, am I right to say that the novelty/contribution of the work lies in the following (caveat - obviously, this is a shallow summary):
1. Making multipolar traps, stemming from the evolutionary optimization process, the central concept (instead of doing the most good or becoming less wrong).
2. Outlining how it's the root cause of key priorities such as existential risks (and perhaps things such as the Fermi paradox).
3. Describing how multipolar traps can be avoided via global coordination in the form of value chains and related mechanisms. If so, this might be the most novel part of the book.
Relevant: Natural Selection Favors AIs over Humans
universal optimization algorithm
Evolution is not an optimization algorithm (this is a common misconception discussed in Okasha, Agents and Goals in Evolution).
1
1I am not sure what you mean by "algorithm" here, but the book you link repeatedly acknowledges that of course Natural Selection is an "optimization process", though there are of course disagreements about the exact details. The book has not a single mention of the word "algorithm" so presumably you are using it here synonymous with "optimization process", for which the book includes quotes like:
"selection is obviously [to some degree] an optimizing process"
The book then discusses various limits to the degree to which evolution will arrive at optimal solutions, but the book seems quite clear that calling evolution an optimization process is fine, though also often leads to miscommunication and conveys some wrong intuitions.
Sure. It's an optimization process. At least in my vocabulary process and algorithm are more or less synonymous. But totally fine with calling it a process instead.
What in your mind are the wrong intuition created from calling it an algorithm?
(I was objecting to Dan's point. I think evolution is both an optimization process and meaningfully described as an "optimization algorithm". I don't really know what Dan's point is, since the book he linked doesn't super agree with it, though it does provide nuance to the degree to which evolution could be described as an optimization process)
1Strictly speaking there is no such thing as "natural selection" or "fitness" or "adaptation" or even "evolution". There are only patterns of physical objects, which increase or decrease in frequency over time in ways that are only loosely modeled by those terms.
But it's practically impossible to talk about physical systems without fudging a bit of teleology in, so I don't think it's a valid objection.
Yes, agreed. Teleology is still very useful in biology. Describing the above post with chemistry would be like describing a high level programing language using only NAND gates (I.e. not very useful).
1So of course ‘natural selection is not optimizing fitness’, since none of those things actually exist in the atoms, and electrons, and spacetime fabric, etc… that make up planet Earth.
i.e. There are no ‘natural selection’ molecules to be found anywhere.
And even the patterns are highly contingent on many factors, perhaps infinitely many, so they can’t be said to have discrete, separable, relationships in the literal sense.
It’s just convenient shorthand to describe something many people believe to be sufficiently understood enough among their peers, that they can get away with skipping some mental steps and verbage.
Evolution isn't just a biological process; it's a universal optimization algorithm that applies to any type of entity
Since you don't talk about the other 3 forces of biological evolution, or about "time evolution" concept in physics...
And since the examples seem to focus on directional selection (and not on other types of selection), and also only on short-term effect illustrations, while in fact natural selection explains most aspects of biological evolution, it's the strongest long-term force, not the weakest one (anti-cancer mechanisms and why viruses don't usualy kill theit host are also well explained by natural selection even if not listed as examples here, evolution by natural selection is the thing that well explains ALL of those billions of years of biology in the real world - including cooperation, not just competition)...
Would it be fair to say that you use "evolution" only by analogy, not trying to build a rigorous causal relationship between what we know of biology and what we observe in sociology? There is no theory of the business cycle because of allele frequency, right?!?
When defectors thrive in a mixed population due to selection pressures, more individuals will adapt by choosing defection. Over time, this process erodes the number of cooperators until they eventually all become defectors.
This contradicts a paper "FDT in an evolutionary environment" linked in https://www.lesswrong.com/posts/oR8hdkjuBrvpHmzGF/found-paper-fdt-in-an-evolutionary-environment, which argues that in similar situations defectors may be less competitive than intelligent agents (which, by the way, cooperate among themselves). Also, evolution didn't produce purely selfish entities... for me it suggests that one of more important reasons of defection (and mutual defection) is limit on cognition (so, bounds on rationality).
Good point. I will cover a lot of this in part 2. Essentially I think we are falling victim to survivorship bias. E.g. we will find ourselves in a place in the universe where cooperation is more common since it is needed for complex life and observers like us.
Global coordination stands as arguably the most critical challenge facing humanity today, functioning both as a necessary component for solving existential risks and as a significant barrier to effective mitigation. From nuclear proliferation to artificial intelligence development and climate change, our inability to collaborate effectively on a global scale not only exacerbates these threats but also perpetuates the emergence of new systemic vulnerabilities if left unaddressed.
In this sequence, I will argue that the root of this coordination problem lies in the very mechanisms that shaped our species: natural selection. This evolutionary process, operating as a trial-and-error optimization algorithm, prioritizes immediate survival and reproduction over long-term, global outcomes. As a result, our innate tendencies often favor short-term gains and localized benefits, even when they conflict with the greater good of our species and planet.
The inherent limitations of natural selection in predicting future optimal states have left us ill-equipped to handle global-scale challenges. In a world of finite resources, competition rather than cooperation has often been the more adaptive trait, leading to the emergence of self-interested behaviors that arguably dominate modern societies. This evolutionary legacy manifests in the form of nationalistic tendencies, economic rivalries, dangerous arms races and a general reluctance to sacrifice immediate benefits for long-term collective gains.
This three-part series summarizes my book: The Darwinian Trap: The Hidden Evolutionary Forces That Explain Our World (and Threaten Our Future).
The Universal Algorithm of Natural Selection
Evolution isn't just a biological process; it's a universal optimization algorithm that applies to any type of entity—be it chemicals, groups, countries, companies, or even memes—as long as the system in question fulfills the following three key features:
Darwinian Demons and Multipolar Traps
Humans, like all life forms, face selection pressures that shape behavior. However, these pressures can sometimes lead to self-serving actions with negative consequences, which I term "Darwinian demons." I broadly define Darwinian demons as selection pressures that drive short-sighted, goal-oriented behaviors, which, over time, can lead to widespread net-negative consequences.
Alternative terms for this phenomenon include multipolar traps, Moloch, coordination problems, social dilemmas, race to the bottom, and the tragedy of the commons. I chose the term 'Darwinian Demons' to emphasize its evolutionary roots and the evolutionary lensI will use to analyze them.
Once you begin to look, examples of Darwinian demons emerge everywhere, both in societies and in nature. Below is a list to help build our intuition. While each example deserves a deeper exploration—something I delve into more fully in my book—these snapshots offer a glimpse into the essence of what a Darwinian demon is.
Nature:
Society:
Furthermore, In many cases, Darwinian demons have driven a cultural shift towards the evolution of doublespeak to make adaptive behavior with negative consequences more palatable. Terms like 'strategic financial engineering' often mask accounting manipulation; 'tax planning' can be a euphemism for aggressive tax avoidance; mass layoffs are disguised as 'restructuring'; 'revenue optimization' may conceal misleading pricing strategies; and paying fines for rule-breaking is reframed as merely the 'cost of doing business.' In politics and government, 'enhanced interrogation techniques' is a euphemism for torture; 'collateral damage' softens the reality of civilian casualties in military operations; 'right-sizing' often means cutting public services; and 'national security measures' can sometimes cloak actions that infringe on civil liberties.
By popularizing the concept of Darwinian demons, I aim to shift the focus from the simplistic "bad apples" narrative to more systemic explanations rooted in economics and evolutionary biology. This perspective frees individuals from bearing sole responsibility for systemic problems and fosters a deeper understanding of their causes. To tackle society's greatest challenges, we must look beyond blaming individual apples and examine the orchard that consistently produces them.
Why are the demons so hard to escape?
The core issue isn't just that it's often adaptive to act in ways that harm others—it's that individual agents frequently find themselves trapped in a cycle where harming others is the only rational option. Game theory, and specifically the concept of the tragedy of the commons, helps explain why Darwinian demons are so difficult to overcome. The tragedy of the commons describes scenarios where shared resources are depleted because individuals, acting independently and rationally in their own self-interest, fail to consider the long-term impact on the group. A real-world example is the overfishing in the Grand Banks cod fishery, which led to the collapse of the cod population and devastated the local industry. The outcomes of this dynamic can be illustrated in the following matrix:
Other Players Follow Quota
Other Players Overfish
The key lesson is that rational choices made by individuals don't always result in the best outcome for the group. This is often referred to as an inefficient Nash equilibrium. In a Nash equilibrium, each player’s strategy is optimized for their individual benefit, assuming others' strategies remain constant. However, when these individual optimizations lead to a collectively suboptimal result, the equilibrium is considered inefficient.
Without mechanisms to promote and enforce cooperation—like laws enforcing fishing quotas or economic incentives for sustainability—natural selection tends to favor those who defect. When defectors thrive in a mixed population due to selection pressures, more individuals will adapt by choosing defection. Over time, this process erodes the number of cooperators until they eventually all become defectors.
Is Selfishness Really Such a Bad Thing?
In the example of the competing fisheries, the selection pressure for profit prompts the fisheries to behave selfishly in a way that optimizes short-term survival but leads to a depleted fish stock for everyone. And yet there are plenty of other examples where defection can beget positive outcomes—in theory. According to Adam Smith’s principle of the “invisible hand,” when individuals act in their own self-interest, they unknowingly contribute to the economic well-being of the community. This happens through the pursuit of profit, which encourages businesses to produce goods and services that are in demand. In turn, this production meets consumers’ needs and desires, leading to greater overall societal welfare. So is selfishness really such a bad thing? Or as Gordon Gekko famously put it in the movie Wall Street:
While it's undeniable that selfishness in a free market has been a tremendous force for good—creating vast wealth and lifting billions out of poverty—we must also critically assess its potential downsides with a clear, unbiased perspective, free from ideological zealousness. Only by doing so can we develop policies that achieve Pareto optimal outcomes.
Take a company that aims to make a positive impact by selling a useful product—say, an innovative new sponge. The company might measure its success in terms of profit, reasoning that the more money it makes, the more sponges it can produce and distribute. However, by focusing primarily on profit, the company may neglect other important goals like employee welfare, consumer protection, or environmental stewardship. While society may benefit from cleaner dishes thanks to the sponge, the company’s disregard for these other priorities could result in a net negative impact.
In other words, in a world with limited resources—a world such as our own— when we optimize for one thing while remaining indifferent to another, then we by default actively optimize against the goal to which we’re indifferent. These adverse consequences are what economists call externalities, but the same phenomenon can also be seen in nature. When we optimize for short-term survival in an environment while remaining indifferent to other important goals—such as happiness, health, and well-being—then, by default, we end up optimizing against those values. In his book River Out of Eden, Richard Dawkins eloquently points this out:
Some even argue that life on Earth, over its 4 billion-year history, has been a net negative, generating more suffering than well-being. Moreover, Darwinian forces pose not only a threat to our well-being but also to our very existence. These zero-sum dynamics drive two existential arms races: for resources and power.
Natural Selection: the Generator Function of Existential Risks
To survive, any agent must secure resources, but this relentless pursuit has led to widespread environmental destruction, potentially setting the stage for a new mass extinction. In a world with finite resources, power is crucial for controlling access to land and resources, a dynamic that has led to the development of nuclear weapons. These weapons, if deployed in global conflict, could annihilate most, if not all, human and animal life.
When plotted over time, the impact of these arms races become increasingly evident. Consider the evolution from stone axes, which took days to fell a tree, to modern harvesters that accomplish the task in seconds. Societies that ruthlessly exploit resources grow more powerful, outcompeting those that live in harmony with nature. As a result, nearly half of all species are at risk of extinction within our lifetime.
Worse still, global competition for resources makes it advantageous to produce ever more powerful weapons. What began with stone axes, capable of killing one person in a minute, has escalated to nuclear bombs, capable of killing billions. This trend predictably led to the genesis of doomsday weapons and continues toward increasingly destructive technologies, such as engineered pandemics.
At some point, these arms races become inevitable; a nation-state cannot simply choose to opt out. As depicted in the movie Oppenheimer, scientists at Los Alamos National Laboratory believed they were in a race against the Germans to develop the first atomic weapon, making inaction untenable.
Similarly, the race towards superintelligence is following the same pattern, with China and the USA on opposing sides. However, AI is not just another technology; it is a technology that will enable the creation of new, more powerful technologies. Human intelligence has historically limited the lethality of the technologies we develop, but AI could potentially remove these limits. AI might enable the creation of new doomsday weapons, where the total destruction of all life on Earth is guaranteed. Such a weapon could act as the ultimate deterrent, creating an arms race to build them before anyone else. This misuse of AI for dangerous weapon manufacturing could occur even before an intelligence explosion or AI takeover scenario, posing a grave threat to humanity.
We Need Civilizational Alignment To Solve Technical Alignment
The technical alignment problem involves ensuring that AI systems consistently operate in line with human values, which may vary depending on the specific use case. For example, in the context of autonomous vehicles, a 2016 study published in Science found that most people agree that vehicles should be programmed to prioritize the welfare of all individuals, including pedestrians, even at the expense of the driver and passengers. However, the same study found that people were three times less likely to purchase such an aligned car themselves.
The lesson, put simply, is this: A solution is only impactful if people actually adopt it—and often they won’t. Even if we successfully solve the technical alignment problem and create AI systems that reliably follow what most humans would consider benevolent intentions, it doesn’t guarantee that AI will no longer pose a threat. The world is still filled with individuals—CEOs driven by profit, military leaders seeking more lethal weapons, and politicians aiming to advance their own agendas—who will be motivated to develop and deploy AI that serves their interests rather than society’s. Much like how most drivers would prefer an autonomous vehicle that prioritizes their own survival over that of others.
Solving the technical AI alignment problem is a noble goal, and a necessary one. But unless we also find ways to solve the societal alignment problem, the quest to make AI universally safe and helpful is bound to fail. To address the societal alignment problem, we must establish robust global systems of governance that incentivize the use of ethically aligned AI. This means creating frameworks where it becomes not just a moral choice but also a practical and advantageous one for companies and governments to adopt AI systems that align with human values. Without such a system, we risk the emergence of regulatory loopholes—akin to tax havens—where certain nations may strip away safety regulations and "human-in-the-loop" requirements in a bid to attract AI enterprises. As AI continues to automate a growing share of the global economy, these regulatory gaps could lead to a race to the bottom, undermining efforts to make AI universally safe and beneficial. Only through coordinated global efforts can we ensure that AI serves the collective good rather than the narrow interests of a few.
In part 2, I'll challenge the notion that cooperation is an evolutionary inevitability by diving into our evolutionary past and exploring how life’s inherent fragility could be a key to understanding the Fermi Paradox. Then, in Part 3, get ready for an exciting exploration of how the interconnectedness of global value chains and the power of indirect reciprocity and reputation can be harnessed to build a decentralized, tyranny-free global governance system. Stay tuned!