LESSWRONG
LW

Steven Byrnes — LessWrong

Replying toWhy we should expect ruthless sociopath ASI

Why we should expect ruthless sociopath ASI

If you compare a human in 30000 BC to a human today, our brains are full of new information that wasn’t in the training data of 30000 BC. I want to talk about: what would it look to be in a world where you can put millions of LLMs in a sealed box containing a VR environment, for (the equivalent of) thousands of years, and then we open up the box and find that the LLMs have made an analogous kind of scientific and technological progress? (See §1 of “Sharp Left Turn” discourse: An opinionated review.)

Spoiler: I think this is fundamentally impossible with LLMs as we know them today. Anyway, let’s explore... (read 388 more words →)

Replying toWhy we should expect ruthless sociopath ASI

Steven Byrnes18h

Why we should expect ruthless sociopath ASI

This is an interesting topic, but no, my central expectation (and what I’m arguing for here) is that 100% of the ASIs will be ruthless consequentialists.

Couple little points on that side-track though: (1) Ruthless consequentialist AIs can still copy themselves, and cooperate with their copies, if their goals are non-indexical (which they might or might not be, no opinion off the top of my head), (2) Your comment seems to assume that AIs can read each other’s minds? If they can’t, a smart ruthless consequentialist AI would act in a cooperative and prosocial way in an environment where doing so was to its advantage. I agree that mind-reading is an important dynamic that might change the equilibrium in a multipolar AI world.

Why we should expect ruthless sociopath ASI

Steven Byrnes

The conversation begins

(Fictional) Optimist: So you expect future artificial superintelligence (ASI) “by default”, i.e. in the absence of yet-to-be-invented techniques, to be a ruthless sociopath, happy to lie, cheat, and steal, whenever doing so is selfishly beneficial, and with callous indifference to whether anyone (including its own programmers and users) lives or dies?

Me: Yup! (Alas.)

Optimist: …Despite all the evidence right in front of our eyes from humans and LLMs.

Me: Yup!

Optimist: OK, well, I’m here to tell you: that is a very specific and strange thing to expect, especially in the absence of any concrete evidence whatsoever. There’s no reason to expect it. If you think that ruthless sociopathy is the “true core... (read 2154 more words →)

Replying toThe brain is a machine that runs an algorithm

Steven Byrnes1d

The brain is a machine that runs an algorithm

"anything that's entirely about data processing in the widest sense is an algorithm"? But I don't think that maps onto the common usage of the term very well.

Not sure exactly what you have in mind. I do think “the kinds of things that would be discussed in current and future algorithms textbooks” is the right conceptual vocabulary for understanding how the brain does useful things (like navigating the world, avoiding threats, seeking opportunities, inventing tools and concepts, etc.), just as “the kinds of things that would be discussed in camera textbooks” is the right conceptual vocabulary for understanding how the human eye captures images.

Well, more specifically, the human eye is in the... (read more)

Replying toThe brain is a machine that runs an algorithm

Steven Byrnes1d

The brain is a machine that runs an algorithm

An algorithm is an information-processing task, i.e. something that a Turing machine could do.

Sometimes there’s a machine that’s purpose-built to instantiate a particular algorithm, by its construction. If you don’t like the mechanical adder example, here’s another: there’s an algorithm to increment a calendar date, including how many days are in each month and figuring out whether it’s a leap-year or not; and certain fancy watches have gear-based mechanisms that will instantiate that algorithm (and they can get the right answer for centuries, even with all the weird leap-year rules).

Another example would be a special-purpose ASIC, like the tiny high-speed over-voltage-protection IC that you can find in a cell phone. In many... (read more)

Replying toThe brain is a machine that runs an algorithm

Steven Byrnes1d

The brain is a machine that runs an algorithm

I interpret you as saying: normally the process by which we wind up with “a machine that runs an algorithm” is: first we have some algorithm (information-processing procedure) in mind, and then second we come up with a machine that captures it. Whereas that didn’t happen with brains.

But by the same token, normally the process by which we wind up with “a machine that captures images” is: first we have some idea about lenses and light-sensitive substrates etc., and then we build a camera. The human eye was not “designed” by evolution in that order. But that’s not important—we are still correct to think of a human eye as “designed” to capture... (read more)

Replying toThe brain is a machine that runs an algorithm

Steven Byrnes2d

The brain is a machine that runs an algorithm

Hadn’t seen that, but cool! Great minds think alike :)

The brain is a machine that runs an algorithm

Steven Byrnes

Some people say “the brain is a computer”. Other people say “well, the brain is not really a computer, because, like, what’s the hardware versus the software?” I agree: “the brain is a computer” is kinda missing the mark. I prefer: “the brain is a machine that runs an algorithm”.

Here’s a mechanical adder:

What’s “hardware versus software” for a mechanical adder? The question is nonsense.

A mechanical adder is not “a computer”, analogous to a MacBook. Rather, it’s a machine that runs an algorithm. (Namely, the binary addition algorithm.)

And the brain is likewise a machine that runs a (much more complicated) algorithm.

“A machine??”, you say. Yeah, you heard me. A machine. An extraordinarily complex... (read 1074 more words →)

Replying toThe nature of LLM algorithmic progress (v2)

Steven Byrnes4d

The nature of LLM algorithmic progress (v2)

Well I don’t think either Epoch and Dario were talking about data improvements (Epoch because they used perplexity not benchmarks, and perplexity on a fixed corpus is only slightly helped by training data improvements; and Dario based on the wording he used, see §2.2 excerpt).

If Epoch and Dario are making claims that are crazy (6-8 month halving time excluding the data category), and lots of people misunderstood those claims as asserting something directionally less crazy (6-8 month halving time including the data category) … umm, I guess that’s a good thing for public understanding of LLMs, and I should be happy about it?

But it still matters for other reasons. E.g. I think... (read more)

Steven Byrnes7d

If only there was a handy-dandy lesswrong blog-publishing checklist that included “update the link preview image” as one of its items.

Steven Byrnes8d*Quick Take

FYI, if anyone read my post “The nature of LLM algorithmic progress” last week, it’s now a heavily-revised version 2.

Steven Byrnes9d

FWIW, inspired by Justis, I’ve been keeping up a list of things that I could usefully automate with Claude Code (or similar) for my own personal productivity, adding to the list every time something pops into my head. I’ve been adding to the list for the past three weeks. But so far it’s a very underwhelming list! Here’s ~the whole thing:

Custom interface for composing tweet-threads, including their funny formula for counting characters (I have some complaints about the built-in twitter one, e.g. I usually also post them onto bluesky)
Jeff’s “clipboard normalizer” (but I have a PC not Mac)
…And something similar for clipboard conversion from simple HTML into the abstruse “typst” format that

Steven Byrnes

13d

Let’s call “interpretability-in-the-loop training” the idea of running a learning algorithm that involves an inscrutable trained model, and there’s some kind of interpretability system feeding into the loss function / reward function.

Interpretability-in-the-loop training has a very bad rap (and rightly so). Here’s Yudkowsky 2022:

When you explicitly optimize against a detector of unaligned thoughts, you're partially optimizing for more aligned thoughts, and partially optimizing for unaligned thoughts that are harder to detect. Optimizing against an interpreted thought optimizes against interpretability.

Or Zvi 2025:

The Most Forbidden Technique is training an AI using interpretability techniques.
An AI produces a final output [X] via some method [M]. You can analyze [M] using technique [T], to learn what the AI is

... (read 824 more words →)

The nature of LLM algorithmic progress (v2)

Steven Byrnes

14d

(Heavily revised on Feb. 9, 2026—see changelog at the bottom.)

There’s a lot of talk about “algorithmic progress” in LLMs, especially in the context of exponentially-improving algorithmic efficiency. For example:

Epoch AI: “[training] compute required to reach a set performance threshold has halved approximately every 8 months”.
Dario Amodei 2025: “I'd guess the number today is maybe ~4x/year”.
Gundlach et al. 2025a “Price of Progress”: “Isolating out open models to control for competition effects and dividing by hardware price declines, we estimate that algorithmic efficiency progress is around 3× per year”.

It’s nice to see three independent sources reach almost exactly the same conclusion—halving times of 8 months, 6 months, and 7½ months respectively. Surely a sign... (read 3724 more words →)

111

•••

Are there lessons from high-reliability engineering for AGI safety?

Steven Byrnes

17d

This post is partly a belated response to Joshua Achiam, currently OpenAI’s Head of Mission Alignment:

If we adopt safety best practices that are common in other professional engineering fields, we'll get there … I consider myself one of the x-risk people, though I agree that most of them would reject my view on how to prevent it. I think the wholesale rejection of safety best practices from other fields is one of the dumbest mistakes that a group of otherwise very smart people has ever made. —Joshua Achiam on Twitter, 2021
“We just have to sit down and actually write a damn specification, even if it's like pulling teeth. It's the most important

... (read 2236 more words →)

107

New version of “Intro to Brain-Like-AGI Safety”

Steven Byrnes

1mo

A new version of “Intro to Brain-Like-AGI Safety” is out!

Things that have not changed

Same links as before:

As a series of 15 blog posts on LessWrong / Alignment Forum: https://www.lesswrong.com/s/HzcM2dkCq7fwXBej8
As a 225-page PDF (now up to version 3): https://osf.io/preprints/osf/fe36n
Summary video: Video & transcript: Challenges for Safe & Beneficial Brain-Like AGI

…And same abstract as before:

Suppose we someday build an Artificial General Intelligence algorithm using similar principles of learning and cognition as the human brain. How would we use such an algorithm safely?
I will argue that this is an open technical problem, and my goal in this post series is to bring readers with no prior knowledge all the way up to the front-line of

... (read 5581 more words →)

For anyone who read my “Intuitive Self-Models” series (2024): I’m planning to revise it by replacing the term “homunculus” with “active self” wherever it appears. I just think “active self” is a better term for the specific thing I’m trying to talk about there. If anyone disagrees, here’s your chance to speak up. :)

My AGI safety research—2025 review, ’26 plans

Steven Byrnes

2mo

Previous: 2024, 2022

“Our greatest fear should not be of failure, but of succeeding at something that doesn't really matter.” –attributed to DL Moody^[1]

1. Background & threat model

The main threat model I’m working to address is the same as it’s been since I was hobby-blogging about AGI safety in 2019. Basically, I think that:

The “secret sauce” of human intelligence is a big uniform-ish learning algorithm centered around the cortex;
This learning algorithm is different from and more powerful than LLMs;
Nobody knows how it works today;
Someone someday will either reverse-engineer this learning algorithm, or reinvent something similar;
And then we’ll have Artificial General Intelligence (AGI) and superintelligence (ASI).

I think that, when this learning algorithm is understood,... (read 3492 more words →)

133

Reward Function Design: a starter pack

Steven Byrnes

2mo

In the companion post We need a field of Reward Function Design, I implore researchers to think about what RL reward functions (if any) will lead to RL agents that are not ruthless power-seeking consequentialists. And I further suggested that human social instincts constitutes an intriguing example we should study, since they seem to be an existence proof that such reward functions exist. So what is the general principle of Reward Function Design that underlies the non-ruthless (“ruthful”??) properties of human social instincts? And whatever that general principle is, can we apply it to future RL agent AGIs?

I don’t have all the answers, but I think I’ve made some progress, and the... (read 4556 more words →)

We need a field of Reward Function Design

Steven Byrnes

2mo

(Brief pitch for a general audience, based on a 5-minute talk I gave.)

Let’s talk about Reinforcement Learning (RL) agents as a possible path to Artificial General Intelligence (AGI)

My research focuses on “RL agents”, broadly construed. These were big in the 2010s—they made the news for learning to play Atari games, and Go, at superhuman level. Then LLMs came along in the 2020s, and everyone kinda forgot that RL agents existed. But I’m part of a small group of researchers who still thinks that the field will pivot back to RL agents, one of these days. (Others in this category include Yann LeCun and Rich Sutton & David Silver.)

Why do I think that?... (read 1235 more words →)

118

6 reasons why “alignment-is-hard” discourse seems alien to human intuitions, and vice-versa

Steven Byrnes

3mo

Tl;dr

AI alignment has a culture clash. On one side, the “technical-alignment-is-hard” / “rational agents” school-of-thought argues that we should expect future powerful AIs to be power-seeking ruthless consequentialists. On the other side, people observe that both humans and LLMs are obviously capable of behaving like, well, not that. The latter group accuses the former of head-in-the-clouds abstract theorizing gone off the rails, while the former accuses the latter of mindlessly assuming that the future will always be the same as the present, rather than trying to understand things. “Alas, the power-seeking ruthless consequentialist AIs are still coming,” sigh the former. “Just you wait.”

As it happens, I’m basically in that “alas, just you wait” camp,... (read 4808 more words →)

359

•••

Quick book review of "If Anyone Builds It, Everyone Dies" (cross-post from X/twitter & bluesky):

Just read the new book If Anyone Builds It, Everyone Dies. Upshot: Recommended! I ~90% agree with it.

The authors argue that people are trying to build ASI (superintelligent AI), and we should expect them to succeed sooner or later, even if they obviously haven’t succeeded YET. I agree. (I lean “later” more than the authors, but that’s a minor disagreement.)

Ultra-fast minds that can do superhuman-quality thinking at 10,000 times the speed, that do not age and die, that make copies of their most successful representatives, that have been refined by billions of trials into unhuman kinds of thinking

... (read 770 more words →)

Fun fact: AI-2027 estimates that getting to ASI might take the equivalent of a 100-person team of top human AI research talent working for tens of thousands of years.

I’m curious why ASI would take so much work. What exactly is the R&D labor supposed to be doing each day, that adds up to so much effort? I’m curious how people are thinking about that, if they buy into this kind of picture. Thanks :)

(Calculation details: For example, in October 2027 of the AI-2027 modal scenario, they have “330K superhuman AI researcher copies thinking at 57x human speed”, which is 1.6 million person-years of research in that month alone. And that’s mostly going towards inventing... (read more)

No question that e.g. o3 lying and cheating is bad, but I’m confused why everyone is calling it “reward hacking”.

Let’s define “reward hacking” (a.k.a. specification gaming) as “getting a high RL reward via strategies that were not desired by whoever set up the RL reward”. Right?

If so, well, all these examples on X etc. are from deployment, not training. And there’s no RL reward at all in deployment. (Fine print: Maybe there are occasional A/B tests or thumbs-up/down ratings in deployment, but I don’t think those have anything to do with why o3 lies and cheats.) So that’s the first problem.

Now, it’s possible that, during o3’s RL CoT post-training, it got certain... (read more)

In [Intro to brain-like-AGI safety] 10. The alignment problem and elsewhere, I’ve been using “outer alignment” and “inner alignment” in a model-based actor-critic RL context to refer to:

“Outer alignment” entails having a ground-truth reward function that spits out rewards that agree with what we want. “Inner alignment” is having a learned value function that estimates the value of a plan in a way that agrees with its eventual reward.

For some reason it took me until now to notice that:

my “outer misalignment” is more-or-less synonymous with “specification gaming”,
my “inner misalignment” is more-or-less synonymous with “goal misgeneralization”.

(I’ve been regularly using all four terms for years … I just hadn’t explicitly considered how they related... (read more)

I’m intrigued by the reports (including but not limited to the Martin 2020 “PNSE” paper) that people can “become enlightened” and have a radically different sense of self, agency, etc.; but friends and family don’t notice them behaving radically differently, or even differently at all. I’m trying to find sources on whether this is true, and if so, what’s the deal. I’m especially interested in behaviors that (naïvely) seem to centrally involve one’s self-image, such as “applying willpower” or “wanting to impress someone”. Specifically, if there’s a person whose sense-of-self has dissolved / merged into the universe / whatever, and they nevertheless enact behaviors that onlookers would conventionally put into one of... (read more)

I went through and updated my 2022 “Intro to Brain-Like AGI Safety” series. If you already read it, no need to do so again, but in case you’re curious for details, I put changelogs at the bottom of each post. For a shorter summary of major changes, see this twitter thread, which I copy below (without the screenshots & links):

I’ve learned a few things since writing “Intro to Brain-Like AGI safety” in 2022, so I went through and updated it! Each post has a changelog at the bottom if you’re curious. Most changes were in one the following categories: (1/7)
REDISTRICTING! As I previously posted ↓, I booted the pallidum out of the

... (read more)

I think there’s a connection between (A) a common misconception in thinking about future AI (that it’s not a huge deal if it’s “only” about as good as humans at most things), and (B) a common misconception in economics (the “Lump Of Labor Fallacy”).

So I started writing a blog post elaborating on that, but got stuck because my imaginary reader is not an economist and kept raising objections that amounted to saying “yeah but the Lump Of Labor Fallacy isn’t actually a fallacy, there really is a lump of labor” 🤦

Anyway, it’s bad pedagogy to explain a possibly-unintuitive thing by relating it to a different possibly-unintuitive thing. Oh well. (I might still try again to finish writing it at some point.)

Some ultra-short book reviews on cognitive neuroscience

On Intelligence by Jeff Hawkins & Sandra Blakeslee (2004)—very good. Focused on the neocortex - thalamus - hippocampus system, how it's arranged, what computations it's doing, what's the relation between the hippocampus and neocortex, etc. More on Jeff Hawkins's more recent work here.
I am a strange loop by Hofstadter (2007)—I dunno, I didn't feel like I got very much out of it, although it's possible that I had already internalized some of the ideas from other sources. I mostly agreed with what he said. I probably got more out of watching Hofstadter give a little lecture on analogical reasoning (example) than from this whole book.
Consciousness and

... (read more)

LESSWRONG
LW

LESSWRONG
LW

Steven Byrnes

[Intro to brain-like-AGI safety] 1. What's the problem & Why work on it now?

Four ways learning Econ makes people dumber re: future AI

[Intuitive self-models] 1. Preliminaries

Foom & Doom 1: “Brain in a box in a basement”

Steven Byrnes

Why we should expect ruthless sociopath ASI

The brain is a machine that runs an algorithm

In (highly contingent!) defense of interpretability-in-the-loop ML training

The nature of LLM algorithmic progress (v2)

Are there lessons from high-reliability engineering for AGI safety?

New version of “Intro to Brain-Like-AGI Safety”

My AGI safety research—2025 review, ’26 plans

Intuitive Self-Models

Valence

Intro to Brain-Like-AGI Safety

Steven Byrnes

[Intro to brain-like-AGI safety] 1. What's the problem & Why work on it now?

Four ways learning Econ makes people dumber re: future AI

[Intuitive self-models] 1. Preliminaries

Foom & Doom 1: “Brain in a box in a basement”

Steven Byrnes

Why we should expect ruthless sociopath ASI

The brain is a machine that runs an algorithm

In (highly contingent!) defense of interpretability-in-the-loop ML training

The nature of LLM algorithmic progress (v2)

Are there lessons from high-reliability engineering for AGI safety?

New version of “Intro to Brain-Like-AGI Safety”

My AGI safety research—2025 review, ’26 plans

Intuitive Self-Models

Valence

Intro to Brain-Like-AGI Safety

The conversation begins

Things that have not changed

1. Background & threat model

Let’s talk about Reinforcement Learning (RL) agents as a possible path to Artificial General Intelligence (AGI)

Tl;dr