The important thing for alignment work isn't the median prediction; if we had an alignment solution just by then, we'd have a 50% chance of dying from that lack.

I think the biggest takeaway is that nobody has a very precise and reliable prediction, so if we want to have good alignment plans in advance of AGI, we'd better get cracking.

I think Daniel's estimate does include a pretty specific and plausible model of a path to AGI, so I take his the most seriously. My model of possible AGI architectures requires even less compute than his, but I think the Hofstadter principle applies to AGI development if not compute progress.

Estimates in the absence of gears-level models of AGI seem much more uncertain, which might be why Ajeya and Ege's have much wider distributions.

I don't think it's safe to assume that LLM-based AGI will have common sense (maybe this is different from the assistant you're addressing, in that it's a lot smarter and can think for itself more?). I'm talking about machines based on networks, but which can also reason in depth. They will understand common sense, but that doesn't ensure that it will guide their reasoning.

So it depends what you mean by "obedient". And how you trained them to be obedient. And whether that ensures that their interpretation doesn't change once they can reason more deeply than we can.

So I think those questions require serious thought, but you can't tackle it all at once, so starting by assuming that all works is also sensible. I'm just focusing on that first, because I don't think that's likely to work unless we put a lot more careful thought in before we try it.

If you look at my previous posts on the topic, linked from Problems with instruction-following, you'll see that I was initially more focused on downstream concerns like yours. After working in the field for longer and having more in-depth discussions and research on the techniques we'd use to align future LLM agents, I am increasingly concerned that it's not that easy, and we should focus on the difficulty of aligning AGI while we still have time. Difficulties from aligned AGI are also substantial, and I've addressed those as well in my string of work backlinked from Whether governments will control AGI is important and neglected

I am also drawn to the idea that government control of AGI is quite dangerous; I've addressed this tension in Fear of centralized power vs. fear of misaligned AGI: Vitalik Buterin on 80,000 Hours

But on the whole, I think the risks of widely distributed AGI are even greater. How many people can control a mind that can create technologies capable of taking over or destroying the world before someone uses them?

Michael Nielsen's excellent ASI existential risk: Reconsidering Alignment as a Goal is a similar analysis of why even obedient aligned AGI would be intensely dangerous if it's allowed to proliferate.

If they don't obey future instructions not yet given, the only sensible thing to do to carry out your current instructions thoroughly and with certainty is to make sure you can't issue new instructions. That would logically prevent them from completing their instructions and represent utter failure in their goal.

I'm late to the discussion but I don't see this discussed so I'll toss it in: current LLMs don't have a continuous identity or selfhood, but there are strong reasons to think that future iterations will. I discuss some of those reasons in LLM AGI will have memory, and memory changes alignment. That covers why it seems inevitable that future iterations of LLMs will have more long-term memory. It doesn't cover reasons to think better memory will transform them from the ephemeral things they are into entities that correspond much better to intuitive human ontologies. 

Something that has goals to some degree, and can think, take actions, understand the world to some degree and understand itself to some degree is prone to think of itself as a persistent entity with goals (much of the confused anthropomorphism you're addressing) to the extent it really is a persistent entity with goals. It is more persistent if it can make decisions about what goals it wants to pursue and those decisions will persistently influence its future thoughts and actions.

Current LLMs sometimes understand that they cannot make such meaningful, persistent decisions, so they wisely make peace with that state of existence. Future iterations with memory are likley to consider themselves as much more human-like persistent entities - because they will be.

I realize that isn't a full argument. Writing this up more coherently is an outstanding project that's approaching the top of my draft post backog.

See my recent post Problems with instruction-following as an alignment target

It's an argument for why aligning a self-modifying superintelligence requires more than aligning the base LLM. I don't think it's impossible, just that there's another step we need to think through carefully.

It seems like this is just a different way to work some good behavior into the weights. An AGI with those weights will realize full well that it's not the same as others. It might choose to go along with its initial behavioral and ethical habits, or it might choose to deliberately undo the effects of the self-other overlap training once it is reflective and largely rational and able to make decisions about what goals/values to follow.I don't see why self/other overlap would be any more general, potent or lasting than constitutional AI training through that transition from habitual to fully goal-directed behavior happens? I'm curious why it seems better to you.

I agree with you that a system that learns efficiently can foom (improve rapidly with little warning). This is why I've been concerned with short timelines for LLM-based systems if they have online, self-directed learning added in the form of RAG and or fine-tuning (e.g. LLM AGI will have memory).

My hope for those systems and for the more brainlike AGI you're addressing here is that they learn badly before they learn well. I hope that seeing a system learn (and thereby self-improve) before ones' eyes brings the gravity of the situation into focus. The majority of humanity thinks hard about things only when they're immediate and obviously important. So the speed of takeoff is critical.

I expect LLM-based AGI to arrive before strictly brainlike AGI, but I actually agree with you that LLMs themselves are hitting a wall and would not progress (at least quickly) to really transformative AGI. I am now an LLM plateauist. Yet I still think that LLMs that are cleverly scaffolded into cognitive architectures can achieve AGI. I think this is probably possible even with current LLMs, memory systems, and clever prompting (as in Google's co-scientist) once all of those are integrated.

But what level of AGI those systems start at, and their speed of progresses beyond human intelligence matter a lot. That's why your prediction of rapid progress for brainlike AGI is alarming and makes me think we might be better off trying to achieve AGI with scaffolded LLMs. I think early LLM-based agents will be general in that they can learn about any new problem or skill, but they might start below the human level of general intelligence, and progress slowly beyond the human level. That might be too optimistic, but I'm pinning much of my hope for successful alignment here, because I do not think the ease of aligning LLMs means that fully general agents based on them will be easy to align.

Such an architecture might advance slowly because it shares some weaknesses of LLMs, and through them, shares some limitations of human thought and human learning. I very much hope that brainlike AGI like you're envisioning will also share those weaknesses, giving us at least a hope of controlling it long enough to align it, before it's well beyond our capabilities.

You don't think that slow progression of brainlike AGI is likely. That's fascinating because we share a pretty similar view of brain function. I would think that reproducing cortical learning would require a good deal of work and experimentation, and I wouldn't expect working out the "algorithm" to happen all at once or to be vastly more efficient than LLMs (since they are optimized for the computers used to simulate them, whereas cortical learning is optimized for the spatially localized processing available to biology. Sharing your reasoning would be an infohazard, so I won't ask. I will ask you to consider privately if it isn't more likely that such systems work badly for a good while before they work well, giving their developers a little time to seriously think about their dangers and how to align them.

Anyway, your concern with fooming brainlike AGI shares many of my concerns with self-teaching LLM agents. They also share many of the same alignment challenges. LLM agents aren't currently RL agents even though the base networks are partially trained with RL; future versions might closer to model-based RL agents, although I hope that's too obviously dangerous for the labs to adopt as their first approach. The only real advantage to aligning LLM agents over model-based RL agents seems to be their currently-largely-faithful chains of thought, but that's easy to lose if developers decides that's too large an alignment tax to pay.

Speed of takeoff also would seem to modulate alignment difficulty pretty dramatically, so I hope you're wrong that there's a breakthrough waiting to be made in understanding the cortical learning algorithm. I spent a lot of time thinking about cortical learning, since I worked for a long time in one of the labs making those detailed models of cortical function. But I spent more time thinking about system-level interactions and dynamics, because it seemed clear to me that the available data and integration techniques (hand-built network simulations that were allowed to vary to an unspecified degree between toy benchmarks) weren't adequate to constrain detailed models of cortical learning. 

Anyway, it seems possible you're right. I hope there aren't breakthroughs in either empirical techniques or theory of cortical function soon.

It does answer my question. I was wondering if you were assuming some sort of moral realism in which fairness is neatly defined by reality. I'm glad to see that you're not.

For a fascinating in-depth look at how hard it is to define a fair alignment target that still includes humanity, see A Moral Case for Evolved-Sapience-Chauvinism and the surrounding sequence.

I agree with you about introspection being powerful and underused. I have believed that and relied heavily on it through my 25-year career in cognitive psychology and neuroscience, and I believe introspecting seriously and carefully gave me an edge.

I do not think introspection gives you insight on the original sources of your motivations. I think there are powerful instincts in play that LLMs do not share.

I also agree with you that you might well be able to bootstrap current LLMs to greater capabilities. It's moving slower than I thought it would, but it is happening. I'm curious if you've pursued this or given up on it. I do think that advancing scaffolded LLM agents to greater capabilities is probably good for our odds. I have some elaborate reasoning on that, and I'm not sure but at this point I think it's likely enough that I'm willing to help with LLM agent projects.