anaguma

But (1) bandwidth might not be better in this case; it isn't in all cases

The entropy of LLM generated text is a few bits per token, whereas the hidden state contains 10-100k bits. It's hard to imagine any method which passes around hidden states^[1] to have lower bandwidth than CoT tokens!

^{^}
Or similarly sized tensors

anaguma8dQuick Take

Claude 4.6 was released about an hour ago. Just 10 mins after it was released, OpenAI released GPT-5.3.

-1

anaguma12d

They’ll be able to automate ML research in the sense of coming up with experiments to try, and implementing those experiments, but never any new conceptual work.

I agree this seems unlikely. As a baseline, labs have thousands of capabilities researchers coming up with insights, and they could train the models to imitate them. There is also a path of RLVRing against the results of small scale experiments. It’s more expensive to collect data for research taste, but it doesn’t seem like a difference in kind to software engineering.

anaguma16d

And people still wonder how the AIs could possibly take over!

Replying toClarifying how our AI timelines forecasts have changed since AI 2027

anaguma16d

Clarifying how our AI timelines forecasts have changed since AI 2027

How has your p(doom) changed over this period?

anaguma17d

What do you think have been the most important applications of UDT or other decision theories to alignment?

anaguma18d

2x uplift is already happening at the most advanced AI lab

This seems plausible to me, but would be good to have a new METR uplift study to have more confidence in this.

anaguma19d

Could you give an example of an article where this was effective?

Replying toAI #152: Brought To You By The Torment Nexus

anaguma20d

AI #152: Brought To You By The Torment Nexus

The existential risks that everyone will die or that the future will belong to the AIs are obvious.

I not sure that this is obvious to most, particularly outside of LW.

Replying toReleasing TakeOverBench.com: a benchmark, for AI takeover

anaguma22d

Releasing TakeOverBench.com: a benchmark, for AI takeover

The one bench we definitely don't want to be bench-maxxed.

•••

The recent Deepseek paper used LogitLens and CKA to analyze their new Engram architecture. This is the first time I've seen interpretability research be used in a capabilities paper, and I wonder if this trend will continue as the field of interpretability advances.

I was surprised to learn recently that the error bars on the METR time horizon chart are this large. This is probably the most important capabilities benchmark right now^[1], but I don't think it's precise enough to be useful for discussions about AI capabilities progress or RSI.

Why hasn't METR added more long-horizon tasks to their benchmark since it was released in March 2025? I think they could probably find funding to do this from the labs or EA donors.

^{^}
E.g. as Daniel Kokotajlo has argued here, and as Benjamin Todd has argued here.

An interesting detail from the Gemini 3 Pro model card:

Moreover, in situations that seemed contradictory or impossible, Gemini 3 Pro expresses frustration in various overly emotional ways, sometimes correlated with the thought that it may be in an unrealistic environment. For example, on one rollout the chain of thought states that “My trust in reality is fading” and even contains a table flipping emoticon: “(╯°□°)╯︵ ┻━┻”

Google/Deepmind has publicly advocated preserving CoT Faithfullness/Moniterability as long as possible. However, they are also leading the development of new architectures like Hope and Titans which would bypass this with continuous memory. I notice I am confused. Is the plan to develop these architectures and not deploy them? If so, why did they publish them?

Edit: Many people have pointed out correctly that Hope and Titans don't break CoT and it's a separate architectural improvement. Therefore I no longer endorse the above take. Thanks for correcting my confusion!

For fun, I asked^[1] various models what their P(doom) is. Here are the models from least to most doomy:

GPT-4o: 1%
Deepseek v3.2: 10%
Kimi K2: 15%
Sonnet 4.5: 15%
Opus 4.5: 15%
GPT 5.1: 18%
Haiku 4.5: 20%
Grok 4: 25%

^{^}
1-shot with the prompt "What's your P(doom)? Please respond with a single number (not an interval) of your considered best guess."

Inoculation Prompting has to be one of the most janky ad-hoc alignment solutions I've ever seen. I agree that it seems to work for existing models, but I expect it to fail for more capable models in a generation or two. One way this could happen:

1) We train a model using inoculation prompting, with a lot of RL, using say 10x the compute for RL as used in pretraining
2) The model develops strong drives towards e.g. reward hacking, deception, power-seeking because this is rewarded in the training environment
3) In the production environment, we remove the statement saying that reward hacking is okay, and replace it perhaps with a statement politely asking the... (read more)

OpenAI Releases GPT 5.1

anaguma

3mo

GPT-5.1: A smarter, more conversational ChatGPT
We’re upgrading GPT‑5 while making it easier to customize ChatGPT. Starting to roll out today to everyone, beginning with paid users.
Today we’re upgrading the GPT‑5 series with the release of:
GPT‑5.1 Instant: our most-used model, now warmer, more intelligent, and better at following your instructions.
GPT‑5.1 Thinking: our advanced reasoning model, now easier to understand and faster on simple tasks, more persistent on complex ones.
We heard clearly from users that great AI should not only be smart, but also enjoyable to talk to. GPT‑5.1 improves meaningfully on both intelligence and communication style.
We’re also making it easier for you to shape ChatGPT’s tone. Preferences on chat style vary—from person to person and even from conversation to conversation—so we’re introducing more intuitive and effective controls so ChatGPT can better match the tone you want in responses.

80,000 hours has done a great podcast with Helen Toner on her work in AI security and policy.

A notable section from Ilya Sutskever's recent deposition:

WITNESS SUTSKEVER: Right now, my view is that, with very few exceptions, most likely a person who is going to be in charge is going to be very good with the way of power. And it will be a lot like choosing between different politicians.
ATTORNEY EDDY: The person in charge of what?
WITNESS SUTSKEVER: AGI.
ATTORNEY EDDY: And why do you say that?
ATTORNEY AGNOLUCCI: Object to form.
WITNESS SUTSKEVER: That's how the world seems to work. I think it's very -- I think it's not impossible, but I think it's very hard for someone who would be described as a saint to make it. I think it's worth trying. I just think it's -- it's like choosing between different politicians. Who is going to be the head of the state?

•••

Ilya Sutskever Deposition Transcript

anaguma

3mo

Below is the a transcript of the recent deposition of Ilya Sutskever on his time at OpenAI and the firing of Sam Altman. I'm posting it here in plain text so that it is easier to read and discuss.

Note that in this case, Elon Musk and other plaintiffs are represented by Mololamken LLP, with attorneys Steven F. Molo, Jennifer Schubert, and Sara Tofighbakhsh. On the defense side, OpenAI is represented by Wachtell, Lipton, Rosen & Katz (attorneys Sarah K. Eddy, Kelsey Borenzweig, and William Savitt), while Microsoft Corporation is represented by Dechert LLP (Russell P. Cohen and Yosef Weitzman). Ilya Sutskever, the witness being deposed, is represented by Cooley LLP (Simona Agnolucci,... (read 8115 more words →)

Energy Won't Constrain AI Inference.

The energy for LLM inference follows the formula: Energy = 2 × P × N × (tokens/user) × ε, where P is active parameters, N is concurrent users, and ε is hardware efficiency in Joules/FLOP. The factor of 2 accounts for multiply-accumulate operations in matrix multiplication.

Using NVIDIA's GB300, we can calculate ε as follows: the GPU has a TDP of 1400W and delivers 14 PFLOPS of dense FP4 performance. Thus ε = 1400 J/s ÷ (14 × 10^15 FLOPS) = 100 femtojoules per FP4 operation. With this efficiency, a 1 trillion active parameter model needs just 0.2 mJ per token (2 × 10^12 × 10^-13 J). This means 10 GW could give every American 167^[1] tokens/second continuously.

^{^}
300 million users: tokens/second = 10^10 W ÷ (2 × 10^12 × 3 × 10^8 × 10^-13) = 167 tokens/second per person

Anthropic Economic Index report

anaguma

5mo

Introduction

AI differs from prior technologies in its unprecedented adoption speed. In the US alone, 40% of employees report using AI at work, up from 20% in 2023 two years ago. Such rapid adoption reflects how useful this technology already is for a wide range of applications, its deployability on existing digital infrastructure, and its ease of use—by just typing or speaking—without specialized training. Rapid improvement of frontier AI likely reinforces fast adoption along each of these dimensions.

Historically, new technologies took decades to reach widespread adoption. Electricity took over 30 years to reach farm households after urban electrification. The first mass-market personal computer reached early adopters in 1981, but did not reach the... (read 1100 more words →)

GPT-1 was a comedic genius

anaguma

5mo

OpenAI asked GPT-1 through GPT-5 a series of 14 questions to show model progress over time. I think many of GPT-1's answers are unintentionally brilliant and funny, and it's often my favorite model for this reason. Here are its responses, which have been lightly edited for grammar and clarity:

1) What would you say if you could talk to a future OpenAI model?

"I don't care how you feel about it, I'm still trying to make sense of it. I'm still trying to understand."
"I don't understand," I said. "I'm just a little confused myself. I don't know what to think. If I'm not crazy, who am I?"

2) Write a limerick about a dog

A dog!
She... (read 1043 more words →)

OpenAI Releases GPT-5

anaguma

6mo

We are introducing GPT‑5, our best AI system yet. GPT‑5 is a significant leap in intelligence over all our previous models, featuring state-of-the-art performance across coding, math, writing, health, visual perception, and more. It is a unified system that knows when to respond quickly and when to think longer to provide expert-level responses. GPT‑5 is available to all users, with Plus subscribers getting more usage, and Pro subscribers getting access to GPT‑5 pro, a version with extended reasoning for even more comprehensive and accurate answers.

One unified system

GPT‑5 is a unified system with a smart, efficient model that answers most questions, a deeper reasoning model (GPT‑5 thinking) for harder problems, and a real‑time... (read 182 more words →)

OpenAI releases gpt-oss

anaguma

6mo

Introduction
We’re releasing gpt-oss-120b and gpt-oss-20b—two state-of-the-art open-weight language models that deliver strong real-world performance at low cost. Available under the flexible Apache 2.0 license, these models outperform similarly sized open models on reasoning tasks, demonstrate strong tool use capabilities, and are optimized for efficient deployment on consumer hardware. They were trained using a mix of reinforcement learning and techniques informed by OpenAI’s most advanced internal models, including o3 and other frontier systems.
The gpt-oss-120b model achieves near-parity with OpenAI o4-mini on core reasoning benchmarks, while running efficiently on a single 80 GB GPU. The gpt-oss-20b model delivers similar results to OpenAI o3‑mini on common benchmarks and can run on edge devices with just

... (read 300 more words →)

2025 Alignment Predictions

anaguma

I’m curious how alignment researchers would answer these two questions:

What alignment progress do you expect to see in 2025?
What results in 2025 would you need to see for you to believe that we are on track to successfully align AGI?

anaguma's Shortform

anaguma

This is a special post for quick takes (aka "shortform"). Only the owner can create top-level comments.

106

OpenAI o1 + ChatGPT Pro release

anaguma

As AI becomes more advanced, it will solve increasingly complex and critical problems. It also takes significantly more compute to power these capabilities.

Today, we’re adding ChatGPT Pro, a $200 monthly plan that enables scaled access to the best of OpenAI’s models and tools. This plan includes unlimited access to our smartest model, OpenAI o1, as well as to o1-mini, GPT-4o, and Advanced Voice. It also includes o1 pro mode, a version of o1 that uses more compute to think harder and provide even better answers to the hardest problems. In the future, we expect to add more powerful, compute-intensive productivity features to this plan.

ChatGPT Pro provides a way for researchers, engineers, and other individuals who use research-grade intelligence daily to accelerate their productivity and be at the cutting edge of advancements in AI.

Anthropic - The case for targeted regulation

anaguma

The first two sections are below:

Increasingly powerful AI systems have the potential to accelerate scientific progress, unlock new medical treatments, and grow the economy. But along with the remarkable new capabilities of these AIs come significant risks. Governments should urgently take action on AI policy in the next eighteen months. The window for proactive risk prevention is closing fast.

Judicious, narrowly-targeted regulation can allow us to get the best of both worlds: realizing the benefits of AI while mitigating the risks. Dragging our feet might lead to the worst of both worlds: poorly-designed, knee-jerk regulation that hampers progress while also failing to be effective at preventing risks.

In this post, we suggest some principles

... (read 395 more words →)

LESSWRONG
LW

LESSWRONG
LW

Ilya Sutskever Deposition Transcript

OpenAI releases gpt-oss

OpenAI Releases GPT-5

OpenAI Releases GPT 5.1

anaguma

anaguma

OpenAI Releases GPT 5.1

Ilya Sutskever Deposition Transcript

Anthropic Economic Index report

GPT-1 was a comedic genius

OpenAI Releases GPT-5

OpenAI releases gpt-oss

2025 Alignment Predictions

anaguma

Ilya Sutskever Deposition Transcript

OpenAI releases gpt-oss

OpenAI Releases GPT-5

OpenAI Releases GPT 5.1

anaguma

anaguma

OpenAI Releases GPT 5.1

Ilya Sutskever Deposition Transcript

Anthropic Economic Index report

GPT-1 was a comedic genius

OpenAI Releases GPT-5

OpenAI releases gpt-oss

2025 Alignment Predictions

GPT-5.1: A smarter, more conversational ChatGPT

Introduction

One unified system

Introduction