Yeah to be clear, although I would act differently, I do think the LW team both tries hard to do well here, and tries more effectually than most other teams would.

It's just that once LW has become much more of a Schelling point for doom more than for rationality, there's a pretty steep natural slope.

What happens if a genuine critic comes on here and writes something. I agree that some criticism is bad, but what if it is in the form that you ask for (lists of considerations, transparently written)?

Is the only criticism worth reading that which is actually convincing to you? And won't, due to some bias, that likely leave this place an echo chamber?

More than one (imo, excellent) critic of AI doom pessimism has cut back on the amount they contribute to LW because it's grueling / unrewarding. So there's already a bit of a spiral going on here.

My view is that LW probably continues to slide more the AI-doom-o-sphere than towards rationality due to dynamics, including but not limited to:

• lower standards for AI doom, higher standards for not-doom
• lower standards of politeness required of prominent doomers
• network dynamics
• continued prominence given to doom content, i.e., treatment of MIRI's book

I know this is contrary to what people leading LW would like. But in absence of sustained contrary action this looks to me like a trend that's already going on, rather than a trend that's inchoate.

You'll note that the negative post you linked is negative about AI timelines ("AI timelines are longer than many think"), while OP's is negative about AI doom being an issue ("I'm probably going to move from ~5% doom to ~1% doom.")

In AI, I think we've seen perhaps 2 massively trend-breaking breakthroughs in the last 20 or so years: deep learning at substantial scale (starting with AlexNet) and (maybe?) scaling up generative pretraining (starting with GPT-1).[1] Scaling up RL and reasoning models probably caused somewhat above trend progress (in 2025), but I don't think this constitutes a massive trend break.)

Somewhat implied but worth noting, both of these trend breaks are not principally algorithmic but hardware-related.

AlexNet: Hey, shifting compute to GPUs let us do neural networks way better than CPUs.

Scaling up: Hey, money lets us link together several thousand GPUS for longer periods of time.

Maybe some level of evidence that future trend-breaking events might also be hardware related, which runs contrary to several projections.

So for the case of our current RL game-playing AIs not learning much from 1000 games -- sure, the actual game-playing AIs we have built don't learn games as efficiently as humans do, in the sense of "from as little data." But:

• Learning from as little data as possible hasn't actually been a research target, because self-play data is so insanely cheap. So it's hard to conclude that our current setup for AIs is seriously lacking, because there hasn't been serious effort to push along this axis.
• To point out some areas we could be pushing on, but aren't: Game-play networks are usually something like ~100x smaller than LLMs, which are themselves ~100-10x smaller than human brains (very approximate numbers). We know from numerous works that data efficiency scales with network size, so even if Adam over matmul is 100% as efficient as human brain matter, we'd still expect our current RL setups to do amazingly poorly with data-efficiency simply because of network size, even leaving aside further issues about lack of hyperparameter search and research effort.

Given this, while this is of course a consideration, it seems far from a conclusive consideration.

Edit: Or more broadly, again -- different concepts of "intelligence" will tend to have different areas where they seem to have more predictive use, and different areas they seem to have more epicycles. The areas above are the kind of thing that -- if one made them central to one's notions of intelligence rather than peripheral -- you'd probably end up with something different than the LW notion. But again -- they certainly do not compel one to do that refactor! It probably wouldn't make sense to try to do the refactor unless you just keep getting the feeling "this is really awkward / seems off / doesn't seem to be getting at it some really important stuff" while using the non-refactored notion.

Is that sentence dumb? Maybe when I'm saying things like that, it should prompt me to refactor my concept of intelligence.

I don't think it's dumb. But I do think you're correct that it's extremely dubious -- that we should definitely refactoring the concept of intelligence.

Specifically: There's default LW-esque frame of some kind of a "core" of intelligence as "general problem solving" apart from any specific bit of knowledge, but I think that -- if you manage to turn this belief into a hypothesis rather than a frame -- there's a ton of evidence against this thesis. You could even basically look at the last ~3 years of ML progress as just continuing little bits of evidence against this thesis, month after month after month.

I'm not gonna argue this in a comment, because this is a big thing, but here are some notes around this thesis if you want to tug on the thread.

• Comparative psychology finds human infants are characterized by overimmitation relative to Chimpanzees, more than any general problem-solving skill. (That's a link to a popsci source but there's a ton of stuff on this.) That is, the skills humans excel at vs. Chimps + Bonobos in experiments are social and allow the quick copying and imitating of others: overimitation, social learning, understanding others as having intentions, etc. The evidence for this is pretty overwhelming, imo.
• Take a look at how hard far transfer learning is to get in humans.
• Ask what Nobel disease seems to say about the general-domain-transfer specificity of human brilliance. Look into scientists with pretty dumb opinions, even when they aren't getting older. What do people say about the transferability of taste? What does that imply?
• How do humans do on even very simple tasks that require reversing heuristics?

Etc etc. Big issue, this is not a complete take, etc. But in general I think LW has an unexamined notion of "intelligence" that feels like it has coherence because of social elaboration, but whose actual predictive validity is very questionable.

Here's Yudkowsky, in the Hanson-Yudkowsky debate:

I think that, at some point in the development of Artificial Intelligence, we are likely to see a fast, local increase in capability—“AI go FOOM.” Just to be clear on the claim, “fast” means on a timescale of weeks or hours rather than years or decades; and “FOOM” means way the hell smarter than anything else around, capable of delivering in short time periods technological advancements that would take humans decades, probably including full-scale molecular nanotechnology.

So yeah, a few years does seem a ton slower than what he was talking about, at least here.

Here's Scott Alexander, who describes hard takeoff as a one-month thing:

If AI saunters lazily from infrahuman to human to superhuman, then we’ll probably end up with a lot of more-or-less equally advanced AIs that we can tweak and fine-tune until they cooperate well with us. In this situation, we have to worry about who controls those AIs, and it is here that OpenAI’s model [open sourcing AI] makes the most sense.

But Bostrom et al worry that AI won’t work like this at all. Instead there could be a “hard takeoff”, a subjective discontinuity in the function mapping AI research progress to intelligence as measured in ability-to-get-things-done. If on January 1 you have a toy AI as smart as a cow, and on February 1 it’s proved the Riemann hypothesis and started building a ring around the sun, that was a hard takeoff.

In general, I think, people who just entered the conversation recently really seem to me to miss how fast people were actually talking about.

So, I agree p(doom) has a ton of problems. I've really disliked it for a while. I also really dislike the way it tends towards explicitly endorsed evaporative cooling, in both directions; i.e., if your p(doom) is too [high / low] then someone with a [low / high] p(doom) will often say the correct thing to do is to ignore you.

But I also think "What is the minimum necessary and sufficient policy that you think would prevent extinction?" also has a ton of problems that would also tend to make it pretty bad as a centerpiece of discourse, and not useful as a method of exchanging models of how the world works.

(I know this post does not really endorse this alternative; I'm noting, not disagreeing.)

So some problems:

1. Whose policy? A policy enforced by treaty at the UN? The policy of regulators in the US? An international treaty policy -- enforced by which nations? A policy (in the sense of mapping from states to actions) that is magically transferred into the brains of the top 20 people at the top 20 labs across the globe? ...a policy executed by OpenPhil??

2. Why a single necessary and sufficient policy? What if the most realistic way of helping everyone is several policies that are by themselves insufficient, but together sufficient? Doesn't this focus us on dramatic actions unhelpfully, in the same way that a "pivotal act" arguably so focuses us?

3. The policy necessary to save us will -- of course -- be downstream of whatever model of AI world you have going on, so this question seems -- like p(doom) -- to focus you on things that are downstream of whatever actually matters. It might be useful for coalition formation -- which does seem now to be MIRI's focus, so that's maybe intentional -- but it doesn't seem useful for understand what's really going on.

So yeah.

...and similarly, if this is the actual dynamic, then the US "AI Security" push towards export controls might just hurts the US comparatively speaking in 2035.

The export controls being useful really does seem predicated on short timelines to TAI; people should consider whether that is false.

I can't end this review without saying that The Inheritors is one step away from being an allegory in AI safety. The overwhelming difference between the old people and the new people is intelligence.

I mean, while it may be compelling fiction:

• The relative intelligence of homo sapiens and neanderthals seems kinda unclear at the moment. They actually had larger brains than humans, and I've read hypotheses that they were smarter. They cooked with fire, built weapons, very likely had language, etc.
• The Inheritors was published in 1955. It looks like in it the Neanderthals don't, for instance, hunt large mammals, but Wikipedia says this is an old misconception and we now believe them to have been apex predators.
• There are numerous hypotheses about why homo sapiens outcompeted neanderthals, some hinging on, for instance, boring old things like viruses and so on.

So I think it a bad idea to update more from this than one would from a completely fictitious story.