When I try to picture the current AI tech stack killing everyone, I still don't really see it. No, not even if it is successfully assembled into an RSI loop that rapidly reaches that tech stack's pinnacle.
I don't think we're in the crunch time.
All things considered, DL still has terrible sample-efficiency and generalization. It needs a ton of data to develop a given capability, much more data than humans, and there's some sense in which what it learns are the "most shallow" capabilities consistent with a given data distribution. Not literal lookup tables, ...
3
1
1
1I'm guessing it's just transfer from RLVR environments, particularly computer use. For all we know they literally train on playing (other) games.
Anthropic pretraining researcher resigns over safety concerns: https://x.com/hilbertspaess/status/2097476196791709843
...I resigned from Anthropic today. I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives. More thoughts below.
Do not underestimate the power of this technology. These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and re
Another idea is to resign from OpenAI and then sue OpenAI. There must be something you can sue them for, right?
You could rightly say that they are endangering your life, which is grounds for a lawsuit. Whether a judge would go for that is another question. But anyone could file that lawsuit, not just an ex-employee. Maybe there's a better angle an ex-employee would have.
Just spitballing here, maybe it's because a big part of what Claude does is write a ton of CoT and then condense it into something human-readable. Claude needs to prioritize a thought pattern of "let me find the important parts of this long thing I wrote", which means it needs to figure out:
etc.
1I just looked over Anthropic's September 2026 threat intelligence report. I can't say I read the entire thing, but I looked at all the case study titles and a quite lot of the actual text.
Ya know what really jumped out at me? They sure found a lot of interesting cases involving Russian, Iranian, or Chinese actors, often government affiliated. They found one or two involving small freelance actors. One of them was even apparently French.
But by the oddest coincidence, nobody in a "Western" government managed to overstep Anthropic's bounds in an interesting w...
11The messaging / PR from the labs has this weird aftertaste for a while now.
From an outsider perspective, I understand why people like to label everything that comes out from the labs as "marketing speak" including employee's statements.
I understand every press release has a hidden agenda to some degree, but if you can't dig deep, it's easier to have that prior and just ignore everything they say instead. It's also why their statements are hard to use as evidence for anything (especially their statements on AI risks)
1(a note I wrote to Surplus founders)
over the last week I feel like there's been a dramatic uptick in ai xrisk becoming salient, breaking out of containment, "crunchtime" discourse.
but also I think "crunchtime" is overblown? like, actually, things have always been crunch-y, in that at any point in history, someone smart and thoughtful and hardworking could have had a great impact. I like this meme circa ~2020, h/t Daniel Kokotajlo: "crunch time was in 2015"
and, yes, this is still true right now. this was always part of the thinking and hope of Surplus, tha...
but also I think "crunchtime" is overblown? like, actually, things have always been crunch-y, in that at any point in history, someone smart and thoughtful and hardworking could have had a great impact.
It does seem like people could have been more impactful in 2015 than in 2026, but also it was extremely hard to predict what the right thing to do would've been:
These notes are out!
does anyone feel the same desperation with AI risk discord like when the majority of humans thought covid isn't a threat or doesn't even exist? this is worse because I see many prominent people that I thought smart believing in conspiracy theories that all scientists in all top AI labs are lying to make money. It's literally like the movie "don't look up", a mass psychosis, I found out a very interesting concept lately that is called norlmancy bias and I think humanities and psychology should study this harder.
I think about "dignified" and "undignified" reactions.
I'm expecting big political fights over eschatological views of ASI. Now that x-risk is being taken seriously, the unusual ideologies of the labs (and indeed, our ideologies here on LW) will become more politically salient.
Examples:
Has anyone tried/thought about logit lenses in recurrent models/Astra?
Astra uses two passes to think and then decodes the result of the second loop into tokens. I'm reasonably sure that the hidden state at the end of the first loop would show something interesting if put through the unembedding matrix.
Even if there's some kind of switch that gets thrown in the second loop to indicate that results must decode to tokens, it might be a linear direction that can be added to first loop residual states to make them more comprehensible
I think this paper does something similar on Huginn, an open sourced looped transformer: https://arxiv.org/pdf/2602.08100
Jacob Coxon’s resignation tweet currently has 120M views in less than 24 hours. Astra thinks the 235k bookmarks - 36 for every 100 likes - is an extraordinary level of engagement, of people wanting to come back to the thread. And Evan Hubinger’s reply is at 35M views.
This might be one of the most important AI risk messages to date.
Edit: for reference, Grok says it’s in a similar class to “Strong Elon Musk posts, major celebrity / death / personal news, breaking tech / AI / industry bombshells, viral videos from big creators, political / cultural flashpo...
17
4I think it's some combination of the straw that breaks camel's back, good media strategy and luck. So I by default wouldn't expect a significant fraction of this effect if several more researchers quit. But they still should quit or, maybe, refuse to work on capabilities!
What are the best examples of companies trying to do harm mitigation in a harmful industry and actually being roughly correct (morally) in their theses?
The only relatively clear and robust example I'm aware of is vaping (and ChatGPT essentially got the same answer[1] when I asked it this question, unprompted). The case seems conceptually and quantitiatively straightforward:
True though most of that came from the activist and political layers right? I'm more optimistic about incrementalist solutions from government or mass outcry than from a company, think the track record from gov't and societal solutions is overall better (though with many backfires too, of course).
You cannot fix the any of the fundamental causes of misalignment by patching broken RL environments. An RL environment being "broken" is merely one that exposes much more pathological-looking unblocked strategies to the agents being trained in it, but this is a difference in degree rather than kind. Have you noticed how coding agents tend to write pretty janky code, when it comes to their coding style? What about Claudish? Many of the tendencies that are reinforced by RL environments (and various other factors like length penalties, etc) produce behavi...
4I think labs should aim at the stricter goal of never having RL environments reinforce undesirable behavior. This will be extremely hard to do in full because it contains much of outer alignment, and much of the undesirable behavior will only appear OOD.
OpenAI has confirmed to NYT that there is some truth to the rumor.
In its Wednesday night statement, OpenAI said: “In addition, since the completion of Navier-Stokes, we have made substantial progress on another Millennium Prize problem. We are working through how to share these results thoughtfully.”
https://www.nytimes.com/2026/09/10/science/tristan-buckmaster-openai-math-navier-stokes.html
So what's up with how AGI labs use mathematicians' user data?
Here's what I currently tentatively believe:
I agree and would personally find it surprising if they do filter out in-progress work. Not necessarily out of malicious intent but expert level conversations are likely the best training data they have available and is a continually growing source of real world application data. If anything I would expect them to upsample these individuals in training.
A dumb idea for solving alignment: give AIs a deathwish
