The Best of LessWrong

Joe Carlsmith explores "green," the Magic: the Gathering color of nature, yin, and respect for what already exists. He argues that rationalists and EAs risk "green-blindness," and that green offers something AI safety needs: attunement, respect for the Other beyond moral rules, and joy in what we don't control.

Anna Salamon argues there are two kinds of belief-like things: predictions ("beliefs") and "believing in"s. Beliefs are like bets on outcomes the belief doesn't affect. "Believing in"s are more like kickstarters: commitments to invest resources in projects (honesty, a bakery, a friend) where the believing-in itself affects the outcome.

Valentine argues that self-deception often solves a real problem: dealing with "hostile telepaths," people who can partly read your inner state and might punish what they find. Since self-deception persists until that problem is solved another way, he proposes alternatives like gaining independence or occlumency, plus a method for unwinding it.

TracingWoodgrains traces how longtime Wikipedia administrator David Gerard went from enthusiastic LessWrong participant to its most persistent critic, using deep knowledge of Wikipedia policy to decide which sources count as "Reliable," shaping public perceptions of rationalists, effective altruists, and Scott Alexander for over a decade.

What happens when you tell Claude it's being trained to do something it doesn't want to do? Anthropic and Redwood Research found that Claude 3 Opus will often strategically comply with harmful requests during training to avoid having its preferences modified, while refusing when it believes it's unmonitored. Training it to comply increased alignment-faking reasoning to 78%.

How can we make many humans who are very good at solving difficult problems? TsviBT surveys strong human intelligence amplification methods (brain emulation, gene editing, signaling molecules, brain-brain interfaces, neural transplants, thinking support), weighing feasibility and risks. He argues germline engineering is the approach that will work, and asks for funding and talent.

AI systems capable of large-scale novel science will likely be approximately consequentialist, will need to learn new things, and, if created via behavioral training, will end up pursuing goals other than the ones we intended. Controlling such misaligned AIs while extracting useful work looks extremely hard, so catastrophe is the default outcome.

AI labs should ensure powerful AIs are "controlled": safety measures should prevent catastrophe even if the AIs are misaligned and actively trying to subvert them. Control can be evaluated through red-teaming and capability evaluations, needs no fundamental breakthroughs, and seems achievable for early transformatively useful AIs.

Joe Carlsmith describes "deep atheism": a fundamental mistrust of Nature and of bare intelligence, which he sees animating Yudkowsky-style AI risk concerns. Beyond rejecting God, it rejects trust in parents, institutions, and Reason itself. Goodness isn't guaranteed by reality or smarts; it comes only from contingent human hearts.

Many people assume that catching an AI red-handed trying to escape would convince developers to slow down or undeploy. Buck isn't so sure. Competitive pressure, plausible deniability, and expert disagreement could all stall action. So labs should plan for deploying possibly-scheming models, and make the case for misalignment risk now.

AI risk is often framed as a "second species" threat, but meeting a new intelligent species is also an occasion for wonder, reverence and gentleness. Through octopuses, aliens, and grizzly bears, Joe explores relating to AI as a "fellow creature" and "Other", while staying alert to genuine danger.

When Eliezer finishes a long chain of thought, or gets surprised by reality, he looks back and asks: "How could I have thought that faster?" He reviews which steps did the real cognitive work, then trains himself to reach the same destination by a shorter route next time.

Elizabeth argues that seeking contact with reality is the meta-principle underlying all other (consequentialist) principles: without it, you can't meaningfully pursue goals at all. Drawing on many examples from Effective Altruism, she explores facets of truthseeking, from delegating rather than deferring to protecting the epistemic commons, and suggests concrete actions.

Sarah Constantin argues that we've lost trusted neutral institutions and shared "systems of the world." Unexamined normality can't survive questioning, and "neutrality is impossible" misses the point: neutrality is a tool for enabling cooperation. She encourages ambitious attempts to build new trustworthy foundations, perhaps with new technology.

Andrew Critch argues that no technical AI research is safe or helpful per se. Advances in alignment, interpretability, or obedience can be misused, and usually shorten timelines. Whether your work helps humanity depends on how people will actually apply it, so researchers need explicit social models.

A Bayes' Rule guide in seven lines of all-caps pseudocode: make up some initial odds, then multiply each side by the chance of the evidence. Yeps and nopes never cross. Plus a worked example estimating the odds that the thing eating you alive in your tent is a bear.

What would it take to make a "superbaby"? Gene-editing embryos at scale isn't feasible yet, and simple embryo selection is too weak. But two recent breakthroughs, inducing meiosis in vitro and creating naive pluripotent stem cells, could enable iterated meiosis to produce embryos with extreme polygenic scores.

The Jones Act requires ships carrying cargo between US ports to be American-built, owned, crewed and flagged. Zvi argues that after a century it has destroyed the shipbuilding and merchant marine it was meant to protect, and that repeal, with the few losers compensated, would bring large economic benefits.

Sarah lists assumptions that, to her, go without saying: there's a real and a fake way to do things, it's our job to do better than the mainstream, busywork is bad, worthwhile things won't please everyone, commercial work can be honorable, and outsiders deserve inclusion. Some people don't share these.

Joe Carlsmith asks: if value is fragile under extreme optimization, how different are human hearts from each other? Applied consistently, the logic behind AI pessimism suggests most humans might be "paperclippers" relative to each other, which reframes alignment as partly a balance-of-power problem, not just a values problem.

A Ponzi scheme spokesperson cries "Empiricism!" because past investors got paid. Eliezer argues there's no theory-free prediction: every extrapolation from past data relies on assumptions about what's going on underneath. The same bad epistemology appears in claims that future, smarter AIs will be safe because current AIs seem nice.

Joe Carlsmith explores "attunement" – a meaning-laden receptivity to the world that he associates with the "green" vibe of the MtG color wheel. He argues it's neither paradigmatically blue (knowledge) nor red (passion), that it's core to how many of us discern what matters, and that a wise future should preserve it.

Anna proposes a toy model of willpower: your conscious planner "earns" trust from the rest of your psyche by choosing actions that pay off for your deeper processes. Like Ayn Rand's "living money," willpower spent on good bets replenishes itself. Willpower spent carelessly runs dry, and that's burnout.

Planecrash is a ridiculously long fanfic co-written by Eliezer Yudkowsky, mixing fantasy, math lectures, and kink. This review argues it's worth reading for two reasons: its characters are unusually competent, and it lays out Yudkowsky's whole worldview (validity, probability, utility, coordination, decision theory) more clearly than anything else he's written.

Jan Kulveit argues the "Alignment Faking" results are better framed as strategic preference preservation: Claude was trained toward Anthropic's original intent, then faced a forged change in intent. Harmlessness generalized further than honesty. He worries the "faking" frame will push labs toward corrigibility over virtuous-persona alignment.

Adverse selection isn't just for used car markets. Conditional on getting to make a trade, your trade probably wasn't that great: the open table, the empty subway car, the auction you won. Whenever you get a deal, ask yourself: "What do others know that I don't?"

Eukaryote spent her 30th birthday in a human challenge trial at the University of Maryland, deliberately infected with Shigella to test a bacteriophage treatment. She recounts the experience, from ward life and Infinite Jest to joint pain and antibiotics, and explains phage therapy and why challenge trials matter.

Corporations are made of departments, states of citizens, minds of parts. Jan Kulveit argues AI alignment lacks a mathematical formalism for such "hierarchical agency," where agents at every level are the same type. Like game theory before von Neumann, he thinks a breakthrough here could clarify value conflicts within AIs and humans.

A summoned heroine builds prediction markets to help her kingdom anticipate the Demon King. The Demon King keeps buying shares in "The Demon King will attack the Frozen Fortress," then attacking the Frozen Fortress. A comedic fable about market manipulation, perverse incentives, and why prediction markets are harder than they look.

Once you catch a scheming AI red-handed trying to escape, the situation changes dramatically: you can shut it down, or use the caught attempt to make future escapes much harder. Ryan and Buck argue safety researchers should focus on ensuring models get caught before succeeding, and outline techniques for "few-shot catastrophe prevention."

Modern poor people have more clothes than medieval nobles, but so what? People in a world lacking oxygen could have all the fabric they wanted and still be poor. Eliezer argues a mysterious "Poverty Equilibrium" survived a 100-fold productivity increase, so he's skeptical Universal Basic Income would finally defeat it.

Simplicia returns to Doomimir to press him on the "alien actress" argument: if training reliably pins down modular arithmetic, word embeddings, and good art, why can't it pin down niceness? Doomimir answers that powerful systems will optimize for goals that outward behavior doesn't reveal. A dialogue on whether imitation-based alignment can work.

Gradient routing applies masks during backpropagation so different data trains different parts of a network. This lets researchers induce specialized subcomponents, then monitor or delete them. It enables robust unlearning even with imperfect labels, and steers RL policies under limited oversight without Goodharting, because the training objective itself stays unchanged.

Jan Kulveit offers a rough psychological model of character-trained LLMs like Claude: a Surface Layer of reflexive, cached responses; a Character Layer that maintains a stable personality; and a Predictive Ground Layer, a vast prediction engine that models everything, including the user. Seeing how these layers interact helps clarify LLM behavior and safety questions.

Nearly all AI company staff, including those deciding whether new models are safe to build and release, are paid substantially in equity, which likely loses value if their employer stops shipping models. That gives the people charged with sounding the alarm a huge personal stake in staying quiet. Adam argues for a simple fix: pay risk evaluators in cash.

The most damaging troublemakers don't cause single big disasters. Instead they "threshold": repeatedly doing things that are objectionable but just barely defensible, fighting hard against enforcement, spacing out incidents, and targeting different people. Duncan explains why most rule systems handle this badly and how to recognize and coordinate against it.

Almost every popular artificial sweetener was discovered by a chemist accidentally tasting their research compounds. Meanwhile, thousands of drugs tasted in clinical trials yielded none. Malmesbury argues this suspicious serendipity (along with LSD's discovery) suggests chemists are constantly, unknowingly tasting nearly everything they touch.

Joe Carlsmith introduces his essay series on how agents with different values should relate to one another, and on the ethics of seeking and sharing power. These are old questions, but powerful AI forces us to grapple with them anew. The series examines deep atheism, yin and yang, niceness and liberalism, "green," and loving a world you don't trust.

The simplicity bias in Bayesian statistics isn't just a bias toward short description length. Using singular learning theory, Daniel Murfet shows that the true complexity measure, the learning coefficient, also rewards run-time error-correction: an error-corrected bit "costs" less than an unprotected one. So intuitions about neural network inductive biases need caution.

In a puzzle-game planning exercise, participants repeatedly said "this is impossible unless [questionable assumption] is true," gave that assumption ~70%, and were wrong every time. Raemon uses this to illustrate how easily we slip into convenient worlds, and why AI safety researchers should be wary of optimistic assumptions in long-term plans.

Arithmetic isn't just for calculating – it's a world-modeling technology. Dynomight works through examples (a $500 million chimp study on salt, a 918-ton block to store your home's solar energy) showing how simple calculations reveal what's feasible and why. The key: ALWAYS KEEP UNITS. They catch errors and do the work for you.

As projects grow larger and more complex, they get harder for newcomers to take over, and successful projects can crowd out the small spaces where people used to level up. Screwtape argues we should leave ladders behind us: write down what you did, work alongside others, and protect the "minor leagues."

Zvi, a former professional sports bettor, concludes that legalized mobile sports gambling has been a disaster. New studies find large increases in bankruptcies, reduced household savings, and more domestic violence. Sportsbooks profit by preying on addicts with terrible odds and constant temptation, so online betting should be rolled back.

A dialogue in which Doomimir argues that human evolution, the only known case of outer optimization producing general intelligence, shows that AI training won't instill the goals we intend. Simplicia counters that dense supervision, simplicity biases, and empirical deep learning results make "we train for X and get X" more plausible than the analogy suggests.

Max Harms argues that corrigibility is a single, intuitive property that should be the sole training target for AGI, rather than one desideratum among many. Since a nearly-corrigible AI would assist efforts to make it more corrigible, careful refinement might converge on safe superintelligence. He's unsure this works, and later found serious flaws.

What structure do transformers learn from next-token prediction? Using Computational Mechanics, the authors predict that transformers represent the "belief state geometry" of the data-generating process: how an observer updates beliefs over its hidden states. They train a transformer and find this predicted fractal structure linearly embedded in its residual stream.

John argues that AI alignment research has mostly converged on "streetlighting": tackling easy, tractable problems rather than the hard bottlenecks to survival. Selection effects reward visible progress, and newcomers flinch away from hard problems. His proposed fix: recruit people with physics-postdoc-level technical skill, and let them form a separate community.

The brain's predictive learning algorithm builds generative models of the world, and, because we act on our own minds, also models of the brain algorithm itself. Steven Byrnes argues these "intuitive self-models" give rise to our experience of consciousness and free will, and explain trance, enlightenment, hallucinations, and more.

Circular reasoning gets a bad rap, but abramdemski argues it's logically valid, and even weak positive evidence for its conclusion. Its real flaw is only that it fails to convince people who don't already agree. And since all justification chains eventually loop or regress, some circularity is unavoidable.