The important thing for alignment work isn't the median prediction; if we had an alignment solution just by then, we'd have a 50% chance of dying from that lack.

I think the biggest takeaway is that nobody has a very precise and reliable prediction, so if we want to have good alignment plans in advance of AGI, we'd better get cracking.

I think Daniel's estimate does include a pretty specific and plausible model of a path to AGI, so I take his the most seriously. My model of possible AGI architectures requires even less compute than his, but I think the Hofstadter principle applies to AGI development if not compute progress.

Estimates in the absence of gears-level models of AGI seem much more uncertain, which might be why Ajeya and Ege's have much wider distributions.

That all makes sense. To expand a little more on some of the logic:

It seems like the outcome of a partial pause rests in part on whether that would tend to put people in the lead of the AGI race who are more or less safety-concerned.

I think it's nontrivial that we currently have three teams in the lead who all appear to honestly take the risks very seriously, and changing that might be a very bad idea.

On the other hand, the argument for alignment risks is quite strong, and we might expect more people to take the risks more seriously as those arguments diffuse. This might not happen if polarization becomes a large factor in beliefs on AGI risk. The evidence for climate change was also pretty strong, but we saw half of America believe in it less, not more, as evidence mounted. The lines of polarization would be different in this case, but I'm afraid it could happen. I outlined that case a little in AI scares and changing public beliefs

In that case, I think a partial pause would have a negative expected value, as the current lead decayed, and more people who believe in risks less get into the lead by circumventing the pause.

This makes me highly unsure if a pause would be net-positive. Having alignment solutions won't help if they're not implemented because the taxes are too high.

The creation of compute overhang is another reason to worry about a pause. It's highly uncertain how far we are from making adequate compute for AGI affordable to individuals. Algorithms and compute will keep getting better during a pause. So will theory of AGI, along with theory of alignment.

This puts me, and I think the alignment community at large, in a very uncomfortable position of not knowing whether a realistic pause would be helpful.

It does seem clear that creating mechanisms and political will for a pause are a good idea.

Advocating for more safety work also seems clear cut.

To this end, I think it's true that you create more political capitol by successfully pushing for policy.

A pause now would create even more capitol, but it's also less likely to be a win, and it could wind up creating polarization and so costing rather than creating capitol. It's harder to argue for a pause now when even most alignment folks think we're years from AGI.

So perhaps the low-hanging fruit is pushing for voluntary RSPs, and government funding for safety work. These are clear improvements, and likely to be wins that create capitol for a pause as we get closer to AGI.

There's a lot of uncertainty here, and that's uncomfortable. More discussion like this should help resolve that uncertainty, and thereby help clarify and unify the collective will of the safety community.

Great analysis. I'm impressed by how thoroughly you've thought this through in the last week or so. I hadn't gotten as far. I concur with your projected timeline, including the difficulty of putting time units onto it. Of course, we'll probably both be wrong in important ways, but I think it's important to at least try to do semi-accurate prediction if we want to be useful.

I have only one substantive addition to your projected timeline, but I think it's important for the alignment implications.

LLM-bots are inherently easy to align. At least for surface-level alignment. You can tell them "make me a lot of money selling shoes, but also make the world a better place" and they will try to do both. Yes, there are still tons of ways this can go off the rails. It doesn't solve outer alignment or alignment stability, for a start. But GPT4's ability to balance several goals, including ethical ones, and to reason about ethics, is impressive.^[1] You can easily make agents that both try to make money, and thinks about not harming people.

In short, the fact that you can do this is going to seep into the public consciousness, and we may see regulations and will definitely see social pressure to do this.

I think the agent disasters you describe will occur, but they will happen to people that don't put safeguards into their bots, like "track how much of my money you're spending and stop if it hits $X and check with me". When agent disasters affect other people, the media will blow it sky high, and everyone will say "why the hell didn't you have your bot worry about wrecking things for others?". Those who do put additional ethical goals into their agents will crow about it. There will be pressure to conform and run safe bots. As bot disasters get more clever, people will take more seriously the big bot disaster.

Will all of that matter? I don't know. But predicting the social and economic backdrop for alignment work is worth trying.

Edit: I finished my own followup post on the topic, Capabilities and alignment of LLM cognitive architectures. It's a cognitive psychology/neuroscience perspective on why these things might work better, faster than you'd intuitively think. Improvements to the executive function (outer script code) and episodic memory (pinecone or other vector search over saved text files) will interact so that improvements in each make the rest of system work better and easier to improve.

1. ^{^}

   I did a little informal testing of asking for responses in hypothetical situations where ethical and financial goals collide, and it did a remarkably good job, including coming up with win/win solutions that would've taken me a while to come up with. It looked like the ethical/capitalist reasoning of a pretty intelligent person; but also a fairly ethical one.

I have no idea what you mean by your claim if you won't define the central term. Or I do, but I'm just guessing. I think people are typically very vague in what they mean by "good", so it's not adequate for analytical discussion. In this case, a vague sense of good produces only a vague sense in which "moral realism" isn't a strong claim. I just don't know what you mean by that.

I'd be happy to come back later and give my guesses at what people tend to mean by "good"; it's something like "stuff people do whom I want on my team" or "actions that make me feel positively toward someone". But it would require a lot more words to even start nailing down. And while that's a claim about reality, it's quite a complex, dependent, and therefore vague claim, so I'd be reluctant to call it moral realism. Although it is in one sense. So maybe that's what you mean?

Some types of suffering, like chronic pain exacerbated by a shitty repetitive .annual labor job, have never gotten much attention. I think that's like the mundane happiness you describe.

Literature lingers on complex emotional suffering, I think, because it's actually more interesting by virtue of being complex but understandable with effort.

It is like a mind tied in complex knots, partly connected to the structure of the world.

I think there are complex joys as well, and literature can have as much fun with those.

I think we focus on suffering because it benefits from our negativity bias, and it seems more virtuous to spend our time understanding suffering than joy.

I think a careful unwrapping of the complexity of a joyful experience, like attending a party and interacting with people in ways they individually appreciate, or the beauty and strangeness of the walk in your other post, ultimately hold just as much interest and virtue, once we don't need to deal with so much shit.

I was wear rose tinted sunglasses, for exactly this perspective on overlooked ubiquitous beauty.

Was this secretly written by Golden Gate Claude? There's a lot of mentions of beauty and majesty and mists near that bridge...

I suspect it's coincidence. This sounds fun, and I've been meaning to do more healthy walking while working, particularly now that 4o makes it so easy to talk to someone with deep knowledge of alignment work. It's at least decent for working through ideas.

Crows are interesting little beings. I have a crow buddy that I think got ditched by his crow friends for being a crow asshole. We're not good buddies because I don't think we really get each other, but he does hang around and will respond when I crow at him.

You need to define what you mean by good for this to make sense. If good means "what you should do" then it's exactly the big claim Steve is arguing against. If it means something else, what is it?

I do have ideas about what people mean by "good" other than "what you should do", but they're complex. That's why I think you need to define the term more for this claim to make sense.

This is good advice, but you must recognize that it's also advice to be selfish. Many rationalists believe in utilitarianism, which preaches near zero selfishness. This is an immense source of stress and unhappiness.

This is particularly problematic when combined with the historically under-recognized importance of the alignment problem. There's been a concern that each individuals efforts might have a nontrivial influence on the odds of a good future for a truly vast number of sentient beings.

Fortunately, AI alignment/outcomes is being steadily better recognized, so individuals can step away slightly easier knowing someone else will do similar work.

But this does not fully solve the problem. Pretending it doesn't exist and advising someone to be selfish when they have complex, well-thought-out reasons not to be is not going to help those individuals.

Are value claims also more complex on average? Are they more likely to be false because they contain more factual claims? (In addition to being more memetically fit because it would be virtuous to spread them if they're true).

Even if they're harder to get right, it seems that value claims are ultimately the most important ones to make. Nobody cares about facts unless they somehow serve some value.

This isn't to undercut the value of the titular insight.

The comic still doesn't make sense to me. I'm not getting from your explanation why that robot killed that human. I can come up with multiple interpretations, but I still don't know yours.

Which means that your average reader is going to find it completely confusing. And since that's at the center of that comic (which again, is really cool art, writing, and layout, so I hope people like and share it!) they're not going to "get it" and won't really get anything out of it other than "this guy thinks AI will take over for some weird reason I can't understand".

As it happens, I've spent 23 years in a research lab centered on learning in the brain, doing everything from variations on backpropagation, reinforcement learning, and other types of learning, to figuring out how that learning leads to human behavior. Since then I've been applying that thinking to AGI and alignment.

So if I'm not getting it, even after taking multiple tries at it, it's pretty certain that your non-expert audience won't get it either.

I went back and looked at your first explanation of the metaphor of the button. I found it confusing, despite having been through similar explanations a thousand times. I suggest you polish your own thinking by trying to condense it into plain english that someone without any background would understand (since I assume your comic is meant for everyone). Then put that into the crucial spot in the comic.