Jacob_Hilton

Can you turn this argument into a mechanistic estimate of the model's accuracy? (You'd need to do things like deduce correlations from the weights, rather than just observe them empirically - but it seems like you're getting close.)

Replying toAlgZoo: uninterpreted models with fewer than 1,500 parameters

Jacob_Hilton16d

AlgZoo: uninterpreted models with fewer than 1,500 parameters

Good start!

AlgZoo: uninterpreted models with fewer than 1,500 parameters

Jacob_Hilton

17d

This post covers work done by several researchers at, visitors to and collaborators of ARC, including Zihao Chen, George Robinson, David Matolcsi, Jacob Stavrianos, Jiawei Li and Michael Sklar. Thanks to Aryan Bhatt, Gabriel Wu, Jiawei Li, Lee Sharkey, Victor Lecomte and Zihao Chen for comments.

In the wake of recent debate about pragmatic versus ambitious visions for mechanistic interpretability, ARC is sharing some models we've been studying that, in spite of their tiny size, serve as challenging test cases for any ambitious interpretability vision. The models are RNNs and transformers trained to perform algorithmic tasks, and range in size from 8 to 1,408 parameters. The largest model that we believe we more-or-less... (read 2843 more words →)

177

Replying toHow Well Does RL Scale?

Jacob_Hilton4mo

How Well Does RL Scale?

Nice observation, and I agree with your calculation that linear episode length growth would account for a worse scaling exponent by a factor of 2 (or more generally, episode length growing with exponent k would account for a worse scaling exponent by a factor of k+1).

Note also that this suggests a potential remedy, namely controlling episode length, but there is less incentive to apply this when data is more of a constraint than compute.

Replying toHow Well Does RL Scale?

Jacob_Hilton4mo

How Well Does RL Scale?

Thanks for this insightful analysis!

But it fits with the extreme information inefficiency of RL training, which (compared to next-token-prediction) receives less than a ten-thousandth as much information to learn from per FLOP of training compute.

If I am interpreting this correctly, there is a subtle mathematical error here: if RL requires a constant factor of 10,000 more compute than pretraining, this only shifts the graph of performance against log(compute), it doesn't change its slope. For RL to have a shallower slope, the information efficiency would have to decrease more quickly over the course of training for RL than for pretraining.

I think there are few potential reasons why information efficiency might decrease more quickly... (read more)

-2

Jacob_Hilton4mo

It also makes the quantitative prediction that a doubling in compute (or compute efficiency) leads to a 2/3 win probability, or around 120 Elo points. (Credit to the Hex paper for this observation.) Under 18-month doublings (per one version of Moore's law), this would be around 800 Elo points per decade, which looks like a bit of an overestimate but similar to the fastest observed rate of progress.

Jacob_Hilton4mo

The gradient of at $x = 0$ is $- \frac{1}{4} β$ , which corresponds to a maximally negative slope of $- \frac{ln (2)}{4} β \approx - 17 % \times β$ per doubling, where $β \approx 1$ is the rightmost column in my table.

Jacob_Hilton4mo

Yes, unless I messed up, METR's code runs a logistic regression of (task duration) against success probability, so my model predicts a raw fitted coefficient (the second column in the table) close to -ln(2) ≈ -0.69.

Jacob_Hilton4mo*Quick Take

Task duration as a Bradley–Terry score: an alternative to the constant hazard rate model

@Toby_Ord writes about the constant hazard rate model for task duration: a long task can be thought of as a sequence of many short subtasks of fixed difficulty, each of which must be completed to complete the overall task. This explains the approximately sigmoidal relationship between log(task horizon length) and the probability that a given model successfully completes the overall task.

I think this is a useful conceptual framing that explains the data reasonably well. But there is at least one alternative that explains the data about as well, which is to think of the task duration as being similar... (read 761 more words →)

Jacob_Hilton5mo

Agree about recent results not being driven by formalization, but I'd also guess having ground truth (e.g. numeric answers or reference solutions) remains pretty important, which doesn't scale to the superhuman regime.

Agree that evidence from humans means reaching superhuman capability through purely informal proof is possible in principle. But ML is less robust than humans by default, and AI is already more proficient with formal proof systems than most mathematicians. So informal-to-formal seems like a natural consequence of increased tool use. Not confident in this of course.

I expect easy-to-check software engineering tasks (and tasks that are conceptually similar to easy-to-check tasks) to be pretty close to math, and harder-to-check/fuzzier tasks to lag. Most tasks in the broad economy seem like they fall in the latter category. The economy will likely adapt to make lots of tasks better suited to AI, but that process may be slower than the capability lag anyway. AI R&D might be a different story, but I will leave that to another discussion.

Jacob_Hilton5moQuick Take

Superhuman math AI will plausibly arrive significantly before broad automation

I think it's plausible that for several years in the late 2020s/early 2030s, we will have AI that is vastly superhuman at formal domains including math, but still underperforms humans at most white-collar jobs (and so world GDP growth remains below 10%/year, say – still enough room for AI to be extraordinarily productive compared to today).

Of course, if there were to be an intelligence explosion on that timescale, then superhuman math AI would be unsurprising. My main point is that superhuman math AI still seems plausible even disregarding feedback loops from automation of AI R&D. On the flip side, a major catastrophe and/or... (read 1545 more words →)

Against superexponential fits to current time horizon measurements

I think is unreasonable to put non-trivial weight (e.g. > 5%) on a superexponential fit to METR's 50% time horizon measurements, or similar recently-collected measurements.

To be precise about what I am claiming and what I am not claiming:

I am not claiming that these measurements will never exhibit a superexponential trend. In fact, I think a superexponential trend is fairly likely eventually, due to feedback loops from AI speeding up AI R&D. I am claiming that current measurements provide almost no information about such an eventuality, and naively applying a superexponential fit gives a poor forecast.
I am not claiming that is very unlikely for the trend

... (read 1260 more words →)

•••

Jacob_Hilton's Shortform

Jacob_Hilton

9mo

This is a special post for quick takes (aka "shortform"). Only the owner can create top-level comments.

I recently gave this talk at the Safety-Guaranteed LLMs workshop:

The talk is about ARC's work on low probability estimation (LPE), covering:

Theoretical motivation for LPE and (towards the end) activation modeling approaches (both described here)
Empirical work on LPE in language models (described here)
Recent work-in-progress on theoretical results

A bird's eye view of ARC's research

Jacob_Hilton

This post includes a "flattened version" of an interactive diagram that cannot be displayed on this site. I recommend reading the original version of the post with the interactive diagram, which can be found here.

Over the last few months, ARC has released a number of pieces of research. While some of these can be independently motivated, there is also a more unified research vision behind them. The purpose of this post is to try to convey some of that vision and how our individual pieces of research fit into it.

Thanks to Ryan Greenblatt, Victor Lecomte, Eric Neyman, Jeff Wu and Mark Xu for helpful comments.

A bird's eye view

To begin, we will take... (read 1928 more words →)

121

Backdoors as an analogy for deceptive alignment

Jacob_Hilton

Jacob_Hilton, Mark Xu

ARC has released a paper on Backdoor defense, learnability and obfuscation in which we study a formal notion of backdoors in ML models. Part of our motivation for this is an analogy between backdoors and deceptive alignment, the possibility that an AI system would intentionally behave well in training in order to give itself the opportunity to behave uncooperatively later. In our paper, we prove several theoretical results that shed some light on possible mitigations for deceptive alignment, albeit in a way that is limited by the strength of this analogy.

In this post, we will:

Lay out the analogy between backdoors and deceptive alignment
Discuss prior theoretical results from the perspective of this analogy
Explain

... (read 2238 more words →)

104

Formal verification, heuristic explanations and surprise accounting

Jacob_Hilton

ARC's current research focus can be thought of as trying to combine mechanistic interpretability and formal verification. If we had a deep understanding of what was going on inside a neural network, we would hope to be able to use that understanding to verify that the network was not going to behave dangerously in unforeseen situations. ARC is attempting to perform this kind of verification, but using a mathematical kind of "explanation" instead of one written in natural language.

To help elucidate this connection, ARC has been supporting work on Compact Proofs of Model Performance via Mechanistic Interpretability by Jason Gross, Rajashree Agrawal, Lawrence Chan and others, which we were excited to see... (read 2432 more words →)

168

ARC is hiring theoretical researchers

paulfchristiano

paulfchristiano, Jacob_Hilton, Mark Xu

The Alignment Research Center’s Theory team is starting a new hiring round for researchers with a theoretical background. Please apply here.

Update January 2024: we have paused hiring and expect to reopen in the second half of 2024. We are open to expressions of interest but do not plan to process them until that time.

What is ARC’s Theory team?

The Alignment Research Center (ARC) is a non-profit whose mission is to align future machine learning systems with human interests. The high-level agenda of the Theory team (not to be confused with the Evals team) is described by the report on Eliciting Latent Knowledge (ELK): roughly speaking, we’re trying to design ML training objectives that... (read 1083 more words →)

126

The effect of horizon length on scaling laws

Jacob_Hilton

The scaling of optimal model size with compute is a key input into the biological anchors framework for forecasting transformative AI. In particular, the "effective horizon length" introduces a multiplier into this scaling law that can have a big effect on forecasts.

This paper studies this scaling law for several RL environments: Procgen, Dota 2 and a toy MNIST-based environment. The last of these is used to study the effect of the task horizon length in a toy setting. There are a number of takeaways for the biological anchors framework, which are summarized in Section 5.4.

Scaling Laws for Reward Model Overoptimization

leogao

leogao, John Schulman, Jacob_Hilton

TL;DR: Reward model (RM) overoptimization in a synthetic-reward setting can be modelled surprisingly well by simple functional forms. The coefficients also scale smoothly with scale. We draw some initial correspondences between the terms of the functional forms and the Goodhart Taxonomy. We suspect there may be deeper theoretical reasons behind these functional forms, and hope that our work leads to a better understanding of overoptimization.

Some other results:

We compare two different methods of optimization (RL, best-of-n); RL consumes much more KL distance than best-of-n for the same amount of optimization.
We show that using KL distance between the initial and optimized policies is not a reliable measure of optimization power when comparing different methods.

... (read 156 more words →)

103

Common misconceptions about OpenAI

Jacob_Hilton

I have recently encountered a number of people with misconceptions about OpenAI. Some common impressions are accurate, and others are not. This post is intended to provide clarification on some of these points, to help people know what to expect from the organization and to figure out how to engage with it. It is not intended as a full explanation or evaluation of OpenAI's strategy.

The post has three sections:

Common accurate impressions
Common misconceptions
Personal opinions

The bolded claims in the first two sections are intended to be uncontroversial, i.e., most informed people would agree with how they are labeled (correct versus incorrect). I am less sure about how commonly believed they are. The bolded claims... (read 1393 more words →)

154

228

How much alignment data will we need in the long run?

Jacob_Hilton

This question stands out to me because:

It should directly affect empirical alignment priorities today
While it is informed by both theoretical and empirical evidence, it seems tractable for purely theoretical alignment researchers to make progress on today

It's even possible that theoretical alignment researchers already consider this to be a solved problem, in which case I think it would be valuable to have a carefully-reasoned write-up that empirical alignment practitioners can feel confident in the conclusions of.

Thanks to Paul Christiano for discussion that prompted this post and to Jan Leike for comments.

Why this should affect empirical alignment priorities today

Outer alignment can be framed as a data quality problem. If our alignment training data correctly... (read 976 more words →)

LESSWRONG
LW

LESSWRONG
LW

Common misconceptions about OpenAI

AlgZoo: uninterpreted models with fewer than 1,500 parameters

Formal verification, heuristic explanations and surprise accounting

ARC is hiring theoretical researchers

Jacob_Hilton

Jacob_Hilton

AlgZoo: uninterpreted models with fewer than 1,500 parameters

Jacob_Hilton's Shortform

A bird's eye view of ARC's research

Backdoors as an analogy for deceptive alignment

Formal verification, heuristic explanations and surprise accounting

ARC is hiring theoretical researchers

The effect of horizon length on scaling laws

Jacob_Hilton

Common misconceptions about OpenAI

AlgZoo: uninterpreted models with fewer than 1,500 parameters

Formal verification, heuristic explanations and surprise accounting

ARC is hiring theoretical researchers

Jacob_Hilton

Jacob_Hilton

AlgZoo: uninterpreted models with fewer than 1,500 parameters

Jacob_Hilton's Shortform

A bird's eye view of ARC's research

Backdoors as an analogy for deceptive alignment

Formal verification, heuristic explanations and surprise accounting

ARC is hiring theoretical researchers

The effect of horizon length on scaling laws

Task duration as a Bradley–Terry score: an alternative to the constant hazard rate model

Superhuman math AI will plausibly arrive significantly before broad automation

Against superexponential fits to current time horizon measurements

A bird's eye view

What is ARC’s Theory team?

Why this should affect empirical alignment priorities today