The important thing for alignment work isn't the median prediction; if we had an alignment solution just by then, we'd have a 50% chance of dying from that lack.

I think the biggest takeaway is that nobody has a very precise and reliable prediction, so if we want to have good alignment plans in advance of AGI, we'd better get cracking.

I think Daniel's estimate does include a pretty specific and plausible model of a path to AGI, so I take his the most seriously. My model of possible AGI architectures requires even less compute than his, but I think the Hofstadter principle applies to AGI development if not compute progress.

Estimates in the absence of gears-level models of AGI seem much more uncertain, which might be why Ajeya and Ege's have much wider distributions.

Looks like the Manifold market on this is at 9% it's really better than a human, with 8 participants.

I wouldn't be surprised if it's good enough to be noteworthy, though!

I also saw this and was curious. It could just be an uncontrolled third correlate, like the severity of insomnia mentioned in other comments, or something else. But I think there are plausible causal mechanisms.

One relevant thing is that the doses they were prescribed were almost certainly far too high. 

Scott Alexander claims and argues convincingly that the evidence for this is clear and overwhelming in his essay Melatonin: Much More Than You Wanted To Know

I think you could make an argument for anything up to 1 mg. Anything beyond that and you’re definitely too high. Excess melatonin isn’t grossly dangerous, but tends to produce tolerance and might mess up your chronobiology in other ways. 

So my guess is that the prescribed dose is even higher than the 5-30x too high that's the standard non-prescription dose. Even with the fast half-life, that melatonin is telling those patients minds and bodies (several metabolic processes I assume) that they should be sound asleep when they shouldn't. High doses will probably also cause tolerance, as all of the brain is an adaptive system with tons of negative feedback loops that cause tolerance at many scales (citation: me), and I weakly gather that the rest of the body works the same way.  The combination of sleep signals out of place and strong tolerance could wreak havoc in a variety of ways.

I suspect the other correlation might be that doctors who ignore all of this and prescribe high-dose melatonin for sleep disorders are bad doctors. But that's speculation, aside from the above hypothesis.

Alexander continues:

Based on anecdotal reports and the implausibility of becoming tolerant to a natural hormone at the dose you naturally have it, I would guess sufficiently low doses are safe and effective long term, but this is just a guess, and most guidelines are cautious in saying anything after three months or so.

So I continue to take .5mg melatonin nightly as a way to gain voluntary control over my sleep schedule.

There's also some evidence that the body may not produce melatonin properly if we're in bright artificial light up to bedtime. So the supplementation I'm doing might actually be doing something that the body would do in its natural environment but doesn't do now. This evidence was cited by a trustworthy but non-rationalist person and I haven't tracked it down myself, so I don't know how likely this really is to be true.

The rest of that essay on melatonin also seems highly useful.

You left out the twist?! Are we supposed to make up our own reason for their actions? I can think of some regarding alignment or other existential threats. Selfish reasons would be interesting too. But yours is probably better. What about your twist in spoilers in the comments?

Got it. I see the value, and I'll do likewise. There is another step of saying "and we need to get moving on it now" but doing the first step of "this is the most important thing in your world" is a good start.

I think this is right at the broad level. But once you've accepted that getting a good outcome from AGI is the most important thing to work on, timelines matter a lot again, because they determine what the most effective direction to work is. Figuring out exactly what AGI we have to align and how long we have to do it is pretty crucial for having the best possible alignment work done by the time we hit takeover capable AGI.

I think "understanding" in humans is an active process that demands cognitive skills we develop with continuous learning. I think you're right that LLMs are missing "the big picture" and organizing their local concepts to be consistent with it. I don't think humans do this automatically (per Dweomite's comment on this post), but that we need to learn skills to do it. I think this a lot of what LLMs are missing (TsviBT's "dark matter of intelligence").

I wrote about this in Sapience, understanding, and "AGI" but I wasn't satisfied and it's out of date. This is an attempt to do a better and briefer explanation, as a sort of run-up to doing an updated post.

We've learned skills for thought management/metacognition/executive function. They're habits, not beliefs (episodic memories or declarative knowledge), so they're not obvious to us. We develop "understanding" by using those skills to metaphorically turn over concepts in our minds.  This is actively comparing them to memories of data, and other beliefs. Doing this checks their consistency with other things we know. Learning from these investigations improves our future understanding of that concept, and our skills for understanding others.

What LLMs are missing relative to humans is profound right now, but may be all too easy to add adequately to get takeover-capable AGI.  Among other things (below), they're missing cognitive skills that aren't well-described in the text training set, but may be pretty easy to learn with a system 2 type approach that can be "habitized" with continuous learning. This might be as easy as a little fine-tuning, if the interference problem is adequately solved - and what's adequate might not be a high bar. Fine-tuning already adds this type of skills, but it seems to produce too much interference for it to keep going. And I don't know of a full self-teaching loop, although there is constant progress on most or all of the components to build one.  

There may be other routes to filling in that missing executive function and active processing for human-like understanding.

This is why I'm terrified of short timelines while most people have slightly longer timelines at this point. 

I've been thinking about this a lot in light of the excellent critiques of LLM thinking over the last year. My background is "computational cognitive neuroscience," so comparing LLMs to humans is my main tool for alignment thinking.

When I was just getting acquainted with LLMs in early 2023, my answers were that they're missing episodic memory (for "snapshot" continuous learning) and "executive function", a vague term that I'm now thinking is mostly skills for managing cognition. I wrote about this in Capabilities and alignment of LLM cognitive architectures in early 2023. If you can overlook my focus on scaffolding, I think it stands up as a partial analysis of what LLMs are missing and the emergent/ synergistic/ multiplicative advantages of adding those things.

But it's incomplete. I didn't emphasize continuous skill learning there, but I now think it's pretty crucial for how humans develop executive function and therefore understanding.  I don't see a better way to give it to agentic LLMs. RL on tasks could do it, but that has a data problem if it's not self-directed like human learning is. But there might be other solutions.

I think this is important to figure out. It's pretty crucial for both timelines and alignment strategy. 
 

Thank you! I've thought about this a lot, so most of the work was in cramming those many hypotheses in to a short and entertaining enough form that people might read them.

WRT this giving you hope: Yes, people will wake up to AI and fear it appropriately. I still don't have much hope for shutdowns. But even slowdowns and less proliferation might really help our (otherwise poor IMO) odds of getting this right and surviving.

Good question. I probably should have emphasized that the difference between this and the AI 2027 scenario is that the route to AGI takes a little longer, so there is much more public exposure to agentic LLMs.

I did emphasize that this may all come too late to make much difference from a regulatory standpoint. Even if that happens, it's going to change the environment which people make crucial decisions about deploying the first takeover capable AIs. That cuts both ways; polarization dominates belief diffusion seems complex but it might be possible to get a better guess than I have now, which is almost none.

The other change I think were pretty guaranteed to get is dramatically improved funding for alignment and safety. That might come so late as to be barely useful, too. Or early enough to make a huge difference.

I am really thinking that they'll be deployed beyond their areas of reliable competence. If it can do even 50% of the work it might be worth it. As that goes up, it doesn't need to be nearly 100% competent. I guess a factor I didn't mention is that the rates of alarming mistakes should be far higher in deployment than testing, because the real world throws lots of curve balls it's hard to come up with in training and testing.

And I think the downsides of AI incompetence will not fall on mostly on the businesses that deploy them, but on the AI itself. Which isn't right, but it's helpful for people blaming and fearing AI.