LESSWRONG
LW

aog — LessWrong

One reason is that hosting data centers can give countries political influence over AI development, increasing the importance of their governments having reasonable views on AI risks.

•••

aog15d

I think it will probably be easy for intelligence agencies on either side to get an estimate within +/- 5% which is sufficent for a minimal version of this.

That seems plausible or even likely but I think it’s hard to be confident. Public estimates of chip production are high variance. For example, SemiAnalysis and Bloomberg’s estimates of Huawei’s 2025 chip production differ by 50% [1]. I doubt either US or Chinese intelligence will be better at estimating these numbers than independent researchers within the next few years. Maybe the true numbers would become clear if governments want them to be, but I could also imagine China rationally not trusting documentation of US... (read more)

aog15d

This does assume both countries have a good sense of the total number of chips controlled by each country’s developers, which seems doable but not trivial and worth working towards now.

Replying toNew Report: An International Agreement to Prevent the Premature Creation of Artificial Superintelligence

aog3mo

New Report: An International Agreement to Prevent the Premature Creation of Artificial Superintelligence

We've done a number of wargames of this sort of regime and the regime often breaks down.

I'd be curious to hear how it breaks down.

Replying toImpossibleBench: Measuring Reward Hacking in LLM Coding Agents

aog3mo

ImpossibleBench: Measuring Reward Hacking in LLM Coding Agents

I'd be very interested to see more and older models evaluated on this benchmark. There might be a strong correlation between the amount of RLVR in a model's training and its propensity to reward hack. In at least some of your setups, GPT-4.1 cheats a lot less than more recent models. I wonder how much that's caused by it being less capable vs. less motivated to cheat vs. other factors.

Replying toConsiderations around career costs of political donations

aog4mo

Considerations around career costs of political donations

Donations to US political campaigns are legally required to be publicly disclosed, whereas donations to US 501c3 nonprofits and 501c4 policy advocacy organizations are not legally required to be publicly disclosed and can be kept private.

aog4mo

I'm a grantmaker at Longview. I agree there isn't great public evidence that we're doing useful work. I'd be happy to share a lot more information about our work with people who are strongly considering donating >$100K to AI safety or closely advising people who might do that.

aog5mo

He also titled his review “An Effective Altruism Take on IABIED” on LinkedIn. Given that Zach is the CEO of Centre for Effective Altruism, some readers might reasonably interpret this as Zach speaking for the EA community. Retitling the post to “Book Review: IABIED” or something else seems better.

Replying toIs There An AI Safety GiveWell?

aogSep 06, 2025

Is There An AI Safety GiveWell?

Agreed with the other answers on the reasons why there's no GiveWell for AI safety. But in case it's helpful, I should say that Longview Philanthropy offers advice to donors looking to give >$100K per year to AI safety. Our methodology is a bit different from GiveWell’s, but we do use cost-effectiveness estimates. We investigate funding opportunities across the AI landscape from technical research to field-building to policy in the US, EU, and around the world, trying to find the most impactful opportunities for the marginal donor. We also do active grantmaking, such as our calls for proposals on hardware-enabled mechanismsand digital sentience. More details here. Feel free to reach out to aidan@longview.org or simran@longview.org if you'd like to learn more.

Replying toSlowdown After 2028: Compute, RLVR Uncertainty, MoE Data Wall

aog6mo

Slowdown After 2028: Compute, RLVR Uncertainty, MoE Data Wall

Knowing the TAM would clearly be useful for deciding whether or not to continue investing in compute scaling, but trying to estimate the TAM ahead of time is very speculative, whereas the revenues from yesterday's investments can be observed before deciding whether to invest today for more revenue tomorrow. Therefore I think investment decisions will be driven in part by revenues, and that people trying to forecast future investment decisions should make forecasts about future revenues, so that we can track whether those revenue forecasts are on track and what that implies for future investment forecasts.

I haven't done the revenue analysis myself, but I'd love to read something good on the revenue needed to justify different datacenter investments, and whether the companies are on track to hit that revenue.

Digital sentience funding opportunities: Support for applied work and research

aog

aog, zdgroff

9mo

Written by Zach Freitas-Groff and posted at his request.

Summary

I’m excited to announce a “Digital Sentience Consortium” hosted by Longview Philanthropy, in collaboration with The Navigation Fund and Macroscopic Ventures, to support research and applied projects focused on the potential consciousness, sentience, moral status, and experiences of artificial intelligence systems. The opportunities include research fellowships, career transition fellowships, and a broad request for proposals for applied work on these topics.

For years, I’ve thought this area was seriously overlooked. It now has growing interest. Twenty-two out of 123 pages of Claude 4’s model card are about its potential moral patienthood. Scientific experts increasingly say that near-term AI sentience is a real possibility; even the skeptical neuroscientist Anil Seth says, “it is unwise to... (read 913 more words →)

Research Priorities for Hardware-Enabled Mechanisms (HEMs)

aog

9mo

Longview Philanthropy is launching a new request for proposals on hardware-enabled mechanisms (HEMs). To encourage strong proposals, we’ve written up what we believe to be top priorities for research in this field.

We think HEMs are a promising method to enforce export controls, secure model weights, and verify compliance with international agreements on AI. We’d like to fund the development of designs, prototypes, and secure enclosures for various kinds of HEMs, as well as complementary red-teaming and field-building efforts.

We plan to distribute $2M - $10M through this RFP. If you’d like to apply, please submit an expression of interest. Below are our top priority areas for new work.

... (read 4323 more words →)

aog's Shortform

aog

10mo

This is a special post for quick takes (aka "shortform"). Only the owner can create top-level comments.

Shoutout to Epoch for creating its own intellectual culture.

Views on AGI seem suspiciously correlated to me, as if many people's views are more determined by diffusion through social networks and popular writing, rather than independent reasoning. This isn't unique to AGI. Most individual people are not capable of coming up with useful worldviews on their own. Often, the development of interesting, coherent, novel worldviews benefits from an intellectual scene.

What's an intellectual scene? It's not just an idea. Usually it has a set of complementary ideas, each of which make more sense with the others in place. Often there’s a small number of key thinkers who come up with many new ideas, and... (read 416 more words →)

•••

Benchmarking LLM Agents on Kaggle Competitions

aog

tl;dr: I prompted ChatGPT to participate in a Kaggle data science competition. It successfully wrote scripts that trained models to predict housing prices, and ultimately outperformed 71% of human participants.

I'm not planning to build a benchmark using Kaggle competitions, but I think a well-executed version could be comparable to e.g. SWE-Bench. This is a proof-of-concept to inform others who might want to work on building that benchmark.

Motivation

Someone once said that benchmarking LLM agents on consequential real-world tasks could help the world understand and anticipate the advance of AI capabilities. Data science is a consequential real-world task that's also near and dear to my heart, as I spent a few years as a... (read 1322 more words →)

Adversarial Robustness Could Help Prevent Catastrophic Misuse

aog

There have been several discussions about the importance of adversarial robustness for scalable oversight. I’d like to point out that adversarial robustness is also important under a different threat model: catastrophic misuse.

For a brief summary of the argument:

Misuse could lead to catastrophe. AI-assisted cyberattacks, political persuasion, and biological weapons acquisition are plausible paths to catastrophe.
Today's models do not robustly refuse to cause harm. If a model has the ability to cause harm, we should train it to refuse to do so. Unfortunately, GPT-4, Claude, Bard, and Llama all have received this training, but they still behave harmfully when facing prompts generated by adversarial attacks, such as this one and this one.
Adversarial robustness will likely

... (read 2669 more words →)

Unsupervised Methods for Concept Discovery in AlphaZero

aog

Using contrast pairs, the authors extract linear directions in the activation space of AlphaZero which correspond to concepts. By observing AlphaZero's play in situations that use these concepts, human grandmasters can improve their own play.

This is related to the following recent research:

Burns et al. (2022) found directions which correlate with truth in a language model’s latent space by using unsupervised methods applied to contrast pairs.
Research by Alex Turner's SERI MATS stream used contrast pairs to identify directions representing various concepts in the latent space of GPT-2 and RL agents. By adding or subtracting these vectors during the models’ forward passes, they controlled model outputs in sophisticated ways. (1, 2, 3)
Zou et al.

aog

aog, Dan H

Welcome to the 10th issue of the ML Safety Newsletter by the Center for AI Safety. In this edition, we cover:

Adversarial attacks against GPT-4, PaLM-2, Claude, and Llama 2
Robustness against unforeseen adversaries
Studying the effects of LLM training data using influence functions
Improving LLM honesty by editing activations at inference time
Aligning language models in simulated social situations
How to verify the datasets on which a model was trained
An overview of the catastrophic risks posed by AI

Subscribe here to receive future versions.

We have a new safety newsletter. It’s more frequent, covers developments beyond technical papers, and is written for a broader audience.

Check it out here: AI Safety Newsletter.

Robustness

Universal and Transferable Adversarial Attacks on Aligned Language Models

Large... (read 1221 more words →)

Hoodwinked: Evaluating Deception Capabilities in Large Language Models

aog

This is a linkpost for https://arxiv.org/abs/2308.01404

Evaluating the hazardous capabilities of AI systems is useful for prioritizing technical research and informing policy decisions. To better evaluate the deception and lie detection abilities of large language models, I've built a text-based game called Hoodwinked, modeled after games of social deduction including Mafia and Among Us.

In Hoodwinked, players are locked in a house and must find a key to escape. One player is tasked with killing the others. Each time a murder is committed, the surviving players have a natural language discussion, then vote to banish one player from the game. The killer must murder the other players without being identified by the other players,... (read 742 more words →)

Learning Transformer Programs [Linkpost]

aog

Recent research in mechanistic interpretability has attempted to reverse-engineer Transformer models by carefully inspecting network weights and activations. However, these approaches require considerable manual effort and still fall short of providing complete, faithful descriptions of the underlying algorithms. In this work, we introduce a procedure for training Transformers that are mechanistically interpretable by design. We build on RASP [Weiss et al., 2021], a programming language that can be compiled into Transformer weights. Instead of compiling human-written programs into Transformers, we design a modified Transformer that can be trained using gradient-based optimization and then be automatically converted into a discrete, human-readable program. We refer to these models as Transformer Programs. To validate our

aog

Summary

Forecasts of explosive growth from advanced AI often assume that AI will fully automate human labor. But there are a variety of compelling objections to this assumption. For example, regulation could prevent the automation of some tasks, or consumers could prefer human labor in others. Skeptics of explosive growth therefore argue that non-automated tasks will "bottleneck" economic growth. This objection is correct in one way: In models of economic growth similar to Davidson (2023), the task with the slowest productivity growth determines the long-run growth rate. But it misses another key result of these models: Even in the presence of non-automated bottleneck tasks, partial automation can cause a temporary period of explosive... (read 1386 more words →)

LESSWRONG
LW

LESSWRONG
LW

aog

Analysis: US restricts GPU sales to China

Argument against 20% GDP growth from AI within 10 years [Linkpost]

Key Papers in Language Model Safety

Adversarial Robustness Could Help Prevent Catastrophic Misuse

aog

aog

Digital sentience funding opportunities: Support for applied work and research

Research Priorities for Hardware-Enabled Mechanisms (HEMs)

aog's Shortform

Benchmarking LLM Agents on Kaggle Competitions

Adversarial Robustness Could Help Prevent Catastrophic Misuse

Unsupervised Methods for Concept Discovery in AlphaZero

MLSN: #10 Adversarial Attacks Against Language and Vision Models, Improving LLM Honesty, and Tracing the Influence of LLM Training Data

Center for AI Safety Blog Posts

AI Safety Newsletter

aog

Analysis: US restricts GPU sales to China

Argument against 20% GDP growth from AI within 10 years [Linkpost]

Key Papers in Language Model Safety

Adversarial Robustness Could Help Prevent Catastrophic Misuse

aog

aog

Digital sentience funding opportunities: Support for applied work and research

Research Priorities for Hardware-Enabled Mechanisms (HEMs)

aog's Shortform

Benchmarking LLM Agents on Kaggle Competitions

Adversarial Robustness Could Help Prevent Catastrophic Misuse

Unsupervised Methods for Concept Discovery in AlphaZero

MLSN: #10 Adversarial Attacks Against Language and Vision Models, Improving LLM Honesty, and Tracing the Influence of LLM Training Data

Center for AI Safety Blog Posts

AI Safety Newsletter

Summary

Table of Contents

Motivation

Robustness

Universal and Transferable Adversarial Attacks on Aligned Language Models

Summary