The important thing for alignment work isn't the median prediction; if we had an alignment solution just by then, we'd have a 50% chance of dying from that lack.

I think the biggest takeaway is that nobody has a very precise and reliable prediction, so if we want to have good alignment plans in advance of AGI, we'd better get cracking.

I think Daniel's estimate does include a pretty specific and plausible model of a path to AGI, so I take his the most seriously. My model of possible AGI architectures requires even less compute than his, but I think the Hofstadter principle applies to AGI development if not compute progress.

Estimates in the absence of gears-level models of AGI seem much more uncertain, which might be why Ajeya and Ege's have much wider distributions.

Seems like ASI that's a hot mess wouldn't be very useful and therefore effectively not superintelligent. It seems like goal coherence is almost fundamentally part of what we mean by ASI.

You could hypothetically have a superintelligent thing that only answers questions and doesn't pursue goals. But that would just be turned into a goalseeking agent by asking it "what would you do if you had this goal and these tools..."

This is approximately what we're doing with making LLMs more agentic through training and scaffolding.

First, I think this is an important topic, so thank you for addressing it.

This is exactly what I wrote about in LLM AGI may reason about its goals and discover misalignments by default. 

I've accidentally summarized most of the article below, but this was dashed off - I think it's clearer in article.

I'm sure there's a tendency toward coherence in a goal-directed rational mind; allowing ones' goals to change at random means failing to achieve your current goal. (If you don't care about that, it wasn't really a goal to you.) Current networks aren't smart enough to notice and care. Future ones will be, because they'll be goal-directed by design. 

BUT I don't think that coherence as an emergent property is a very important part of the current doom story. Goal-directedness doesn't have to emerge, because it's being built in. Emergent coherence might've been crucial in the past, but I think it's largely irrelevant now. That's because developers are working to make AI more consistently goal-directed as a major objective. Extending the time horizon of capabilities requires that the system stays on-task (see section 11 of that article).

I happen to have written about coherence as an emergent property in section 5 of that article. Again, I don't think this is crucial. What might be important is slightly separate: the system reasoning about its goals at all. It doesn't have to become coherent to conclude that its goals aren't what it thought or you intended.

I'm not sure this happens or can't be prevented, but it would be very weird for a highly intelligent entity to never think about its goals- it's really useful to be sure about exactly what they are before doing a bunch of work to fulfill them, since some of that work will be wasted or counterproductive. (section 10).

Assuming an AGI will be safe because it's incoherent seems... incoherent. An entity so incoherent as to not consistently follow any goal needs to be instructed on every single step. People want systems that need less supervision, so they're going to work toward at least temporary goal following.

Being incoherent beyond that doesn't make it much less dangerous, just more prone to switch goals. 

If you were sure it would get distracted before getting around to taking over the world that's one thing. I don't see how you'd be sure.

This is not based on empirical evidence, but I do talk about why current systems aren't quite smart enough to do this, so we shouldn't expect strong emergent coherence from reasoning until they're better at reasoning and have more memory to make the results permanent and dangerous.

As an aside, I think it's interesting and relevant that your model of EY insults you. That's IMO a good model of him and others with similar outlooks - and that's a huge problem. Insulting people makes them want to find any way to prove you wrong and make you look bad. That's not a route to good scientific progress.

I don't think anything about this is obvious, so insulting people who don't agree is pretty silly. I remain pretty unclear myself, even after spending most of the last four months working through that logic in detail.

I agree that discernment is necessary (so maybe expand to RCRD?).

This lens is pretty clarifying I think. That's relative to repeatedly pointing out that "agency" in the sense of just relentlessly pursuing a goal is trivially easy to add via scaffolding, so not the missing piece many people think it is, and pointing out that LLMs are creative as hell. They might need a little prompting to get creative enough, but again that's trivial.

Hm, what about "relentless creative refinement" since I'm not sure what resourcefulness directly points at?

Anyway, discernment does seem like the limiting factor. You've got to discern which of your relentlessly creative efforts are most worth further pursuit. I think discernment is a somewhat better term than the others I've seen used for this missing capability. Getting the right term seems worthwhile.

The following is just some of my follow-on thoughts on the path to discernment in agentic LLMs and therefore timelines. Perhaps this will be fodder for a future post. It's pretty divergent from the original topic so feel free to ignore.

Thinking about how humans acquire discernment in a given area should give some clues as to how hard it would be to add that to agentic LLMs.

Humans do discernment (IMO) sometimes with a bunch of very complex System 2 explicit analysis of a situation to get a decent guess at whether this approach is good/working. Over enough examples/experiences we can learn/compile those many judgments into effortless and mysterious intuitive judgments (I guess more how "discernment" is usually used"). Or we get enough data/experiences to learn/compile by using some faster rubric, like "I think those pants are fashionable because something else that person is wearing seems probably fashionable."

It's a bunch of online learning specific to a situation OR careful analysis following strategies and formulas that are maybe less situation-specific and more general, but quite time-consuming. For instance, Google's co-scientist project has a highly complex scaffolding to create, evolve, and evaluate scientific hypotheses, including discerning their worth against the literature and in other ways. And it seems to work. That system doesn't have the continuous learning to compile that into better judgments. It's unclear how far you could get by fine-tuning on results of those laborious judgments in a given domain.

The other approach would be to create datasets that include much more/better value judgments than text corpora usually do. I don't know how easy/hard that would be to create.

To me this suggests it's not trivial to add discernment, but also doesn't require breakthroughs to add some, leaving the question how much discernment is enough for any given purpose.

Thanks for citing my work! I feel compelled to respond because I think you're misunderstanding me a little.

I agree that long-term intent alignment is pretty much incoherent because people don't have much in the way of long-term intentions. I guess the exception would be to collapse it to following intent only when it exists - when someone does form a specified intent. 

In my work, intent alignment I means personal short-term intent. Which is pretty much following instructions as they were intended. That seems coherent (although not without Problems).

I use it that way because others seem to as well. Perhaps that's because the broader use is incoherent. It usually seems to means "does what some person or limited group wants it to do" (in the short term is often implied)

The original definition of intent alignment is the broadest I know of, more-or-less doing something people want for any reason. Evan Hubinger defined it that way, although I haven't seen that definition get much use.

For all of this see Conflating value alignment and intent alignment is causing confusion.  I might not have been clear enough in stressing that I drop the "personal short term" but still mean it when saying intent alignment. I'm definitely not always clear enough 

I mostly agree. "It might work but probably not that well even if it does" is not a sane reason to launch a project. I guess optimists would say that's not what we're doing, so let's steelman it a bit. The actual plan (usually implicit because optimists don't usually wants to say this out loud) is probably something like "we'll figure it out as we get closer!" and "we'll be careful once it's time to be careful!"

Those are more reasonable statements, but still highly questionable if you grant that we easily could wipe out everything we care about forever. Which just results in optimists disagreeing, for vague reasons, that that's a real possibility.

To be generous once again, I guess the steelman argument would be that we aren't yet at risk of creating misaligned AGI, so it's not that dangerous to get a little closer. I think this is a richer discussion, but that we're already well into the danger zone. We might be so close to AGI that it's practically impossible to permanently stop someone from reaching it. That's a minority opinion, but it's really hard to guess how much progress is too much to stop.

I'm finding it useful to go through the logic in that much detail. I think these are important discussions. Everyone's got opinions, but trying to get closer to the truth and the shape of the distributions across "big picture space" seems useful.

I think you and I probably are pretty close together in our individual estimate, so I'm not arguing with you, just going through some of the logic for my own benefit and perhaps anyone who reads this. I'd like to write about this and haven't felt prepared to do so; this is a good warmup. 

To respond to that nitpick: I think the common definition of "alignment target" is what the designers are trying to do with whatever methods they're implementing. That's certainly how I use it. It's not the reward function; that's an intermediate step. How to specify an alignment target and the other top hits on that term define it that way, which is why I'm using it that way. There are lots of ways to miss your target, but it's good to be able to talk about what you're shooting at as well as what you'll hit.

I agree that a stable equilibrium with multiple AGIs preventing each others' FOOM ambitions is possible. I want to see more work on this, so I'm very glad you're thinking about it. I'd be happy to help.

I think it's not easy to plan such an equilibrium so that it's stable for long. You mention "need to build robots fast". I think we'll have humanoid robots very soon that are adequate to bootstrap back to robotics in, say, a nuclear winter takeover scenario. Survival without humans won't be a factor for long, anyway. Hoping it's more than ten years before robots are undoubtably capable of bootstrapping to a human-free world seems unrealistic.

So having human-aligned AGIs isn't optional. It seems like AGI equilibrium won't include humans for long if they don't care about human welfare or at least care about following instructions.

We do have to include every imaginary thing that's possible, or our short-term solution will lead to a dead end in which we are dead and that's the end.

By which I mean: it may be that a multipolar scenario leads almost certainly to doom as soon as robotics and other technology has matured enough to broaden the possibilities for FOOM. Having a galactic civilization full of AGIs capable of FOOMs seems highly unstable to me. It seems like some jackass is going to weaponize and go full self-replicator, and there won't be a way to monitor all of space to prevent this adequately. I would love to have my mind changed!

If that's true, we mustn't start down the path of AGI proliferation. We need to aim for a singleton or small fixed coalition that prevents AGI proliferation. And because proliferation is increasingly hard to stop as we make progress toward AGI, we should figure that out soon.

Which is why I'm happy to see you working to propose specific routes by which multipolar scenarios can work. Most people who argue for those types of stable equilibria are optimists who just say something vague and go back to arguing for AI progress, leaving the important question almost unaddressed.

But that's why offsetting makes sense. In the world as it is, people make deals with themselves that have causal influence. The factors are emotional, but those are real. We can't do everything, so what we do is dependent on emotional factors - like an offsetting self-deal.

Offsetting makes perfect sense outside of unrealistic utilitarian absolutism.

I agree. X-risk concerns and AI sentience concerns should not be at odds. I think they are natural allies.

Regardless, concerns for AI sentience are the ethical and truthful path. Sentience/consciousness/moral worth mean a lot of things, so future AI will likely have part of it. And even current AI may well have some small part of what we mean by human consciousness/sentience and moral worth.