From the original post:
Claims of the form “problem X can’t be solved” are really hard to get right, because you are fighting against the universal quantifier of all possible ways that someone could solve this problem. (This is very similar to the difficulty of saying “system X can’t be compromised.”) To the extent that there is any argument that aligning prosaic AGI is infeasible, that argument is nowhere near the level of rigor which would be compelling.
I think this argument is deeply misleading, in potentially dangerous ways. We have decades of evidence supporting various versions of the "Bitter Lesson", which essentially says that "giant, inscrutable matrices trained on real-world data out-perform hand-written systems that we actually understand." This suggests that alignment is all about providing strict guarantees about trillion parameter matrices under all possible inputs.
The underlying problem, I suspect, is that the general form of "intelligence" actually is a function from an input vector (say, an image) to an output probability distribution (say, a list of things that might appear in that image, plus confidence values). And the general form of a function from a vector to a probability distribution is essentially a big matrix. (Or in practice, it might be a collection of smaller matrices combined with some non-linear transformations, like neural networks.) You might be able to factor the matrix, but every input potentially affects every output in some weighted way, giving you a matrix.
So our problem—which we potentially need to solve sometime between January 2027 and the end of 2028—is build a giant matrix and ensure that:
While I cannot prove this problem is utterly impossible to solve, we are certainly nowhere close to solving it. It is likely a "wicked" problem. And as Eliezar points out, we don't get a lot of tries to solve it.
And even if we did solve it, we'd still have:
Seriously, who even asked for this? The public demonstrably hates it, and will likely hate it even more once they understand the threat. Why do we have to rush so damn fast?
This is a great example of the issue that Paul Christiano demonstrated, and to answer the question asked, you could have several answers to the problem:
I think "super hard" doesn't have a lot of evidence right now, simply because the average quality bar of most RL environments is not high. Most cos selling them are well aware of issues, and could fix them, but are selling to labs because labs haven't been auditing quality as much as they could.
I think you could easily 10x alignment from here by just enforcing sensible 1st order quality aspects on these environments, the amount of low hanging fruit is very very high.
Boom, three ideas, in ascending order of how likely they are to work as an AI safety technique for your stated problem, and this is why I believe Paul Christiano's statement quoted here is still very right.
(For fun, I'll answer the following question:
- The question of who controls the superintelligence. It's not like we've ever aligned billionaires or politicians.
The first answer is we kind of don't need to align billionaires or politicians, mostly because a lot of the bad outcomes that billionaires/politicians generate are basically downstream of selection effects, which is reducible if we avoid nationalizing AI companies, and most importantly a lot of the bad outcomes like the intelligence curse rely on assuming humans are fully selfish/do not care about others, and while this isn't right, it's usually close enough to be accurate in the absence of AI, but AI optimization makes the difference between a small amount of altruism and 0 altruism much, much wider than today's world.
The second answer, and the truly hot take is if we condition on an AI safety success, the superintelligence should take control, because democracy is both not a real option and also much less necessary for citizens to have good lives, downstream of leader succession problems no longer existing, and more generally the factors that CGP Grey uses on why dictatorships are so bad to be a lot less strong under ASI rule:
In contrast, my impression is that MIRI was a bit more skeptical back then that such simplistic approaches could work
IMO Eliezer basically fully corrected for this back in 2023, and more generally rationalists have thankfully updated on pre-training being good enough to get superhuman minds (and if the super-exponential progress trend in pre-training continues for a while, this might actually be a practical way to train AGIs and then ASIs)
WHile it is likely true that intelligence is probably just search over Turing Machines in our reality. the key question is why is it so efficient, and why data/environment-dependent generalization occurs, and for that question to be answered, we need to look into the environments/data/universes, not the algorithms.
The detail he got wrong is that we did get superhuman performance from supervised learning, first with AlphaFold and then with LLMs, simply by training on prediction tasks that no human could do. This resulted in a less scary world than these prosaic agents RL'd for DAOs; but as RLVR continually scales, a more useful intuition pump might be "DAO agent, but fine-tuned to speak English."
The other detail he got wrong, and IMO the far more important detail is that we don't actually need to implement human wishes in RL, because there does exist a class of utility functions that's safe to optimize arbitrarily hard (given the assumption that cooperation in the sense of reliably paying a small amount of resources consistently from humans/automated systems is arbitrarily high), even under the assumption that you need to solve difficult hard-to-verify philosophical/conceptual problems, and you need superhuman intelligence in a way such that you can't control it, and critically it's shovel-ready to implement in the sense that you'd only need compute and the mathematical formulas translated into code, and you can do this without solving the alignment problem.
In short, it's the risk-averse AI agenda, though there are easy add-ons that would grant further benefits.
I don't blame anyone for not thinking about it, because it's the rare example of a nirvana/perfect solution in the limit that actually works very, very well in practice, so you can just do the experiments cheaply and implement the agenda off-the-shelf, and even outside of the limiting cases has a number of excellent properties that were previously the focus of large research agendas, while being much more philosophically neutral/non-committal on big philosophical/conceptual questions than other agendas, which is a desirable trait, and it doesn't require us to solve the AI alignment problem.
But 2a and 2c were to their surprise wrong about the modern day reality, which is good news.
Next up:
3a. Easy to start now
Prosaic AI alignment is especially interesting because the problem is nearly as tractable today as it would be if prosaic AGI were actually available.
Existing alignment proposals have only weak dependencies on most of the details we would learn while building prosaic AGI (e.g. model architectures, optimization strategies, variance reduction tricks, auxiliary objectives…). As a result, ignorance about those details isn’t a huge problem for alignment work. We may eventually reach the point where those details are critically important, but we aren’t there yet.
For now, finding any plausible approach to alignment, that works for any setting of unknown details, would be a big accomplishment. With such an approach in hand we could start to ask how sensitive it is to the unknown details, but it seems premature to be pessimistic before even taking that first step.
Note that even in the extreme case where our approach to AI alignment would be completely different for different values of some unknown details, the speedup from knowing them in advance is at most 1/(probability of most likely possibility). The most plausibly critical details are large-scale architectural decisions, for which there is a much smaller space of possibilities.
I'd rename it to prosaic AI safety, because we don't need alignment. but this was basically right, and I'll go further and say that it turns out we don't even need to put an upper bound on capabilities, and in particular the approach works for a very, very large space of possible details, and is completely agnostic to the original reward function/how a model thinks, and we need 0 interpretability for any load-bearing parts of the risk-averse AI agenda, and can treat the AI entirely as a black box, which is huge and makes it much more implementable than most agendas.
(Side note that the proofs in the paper do not assume strong restrictions on what RL environments are used, and don't assume a particular framework like MDPs or POMDPs, with the very weak restriction that the RL environment/framework and reward functon must have at least 2 possible values for rewards which is excellent and makes it truly a near-universal AI safety solution, and in RL theory it's very rare, if not downright unheard of to have results like this without assuming a particular form of RL environments/a framework like MDPs/POMDPs stronger (in the sense that it applies to less environments) than the environment/reward function having 2 or more possible values for rewards.
Finally, on this:
I'm not sure what he had in mind for "very strong evidence for the intractability" of alignment. The trouble is that, roughly speaking:
- for any specific behavioral property we can name, we can write evals for it
- we can Goodhart those evals until we don't see anything wrong
In this way the system will look aligned, and alignment will always seem "tractable" because we're always making "progress" on it. I think prosaic alignment dropped the ball on this last part of his post, i.e. of posing falsifiable hypotheses and seeking evidence for when their approaches were failing
I could plausibly agree with this, but nowadays consider this less relevant, because it turns out you can in fact have a training setup for even superhuman AIs that can't be Goodharted via Payment-Augmented Reinforcement learning, and the way they prove this is via a 2 step process:
Once that's done, we can say QED and point out that the kind-of-prosaic/90% prosaic AI safety agenda works out beautifully and thus, we don't have to worry about goodhart concerns ever again. (I also believe that misgeneralization concerns are substantially reduced to the level of say being killed by natural background radiation in the earth, though unlike Goodhart it doesn't fully prevent it even in the limit, but it almost fully prevent misgeneralization issues. To put it shortly, a much wider amount of behavior is compatible with risk-averse utility functions, meaning counting arguments are substantially less powerful, risk averse utility functions are extremely simple compared to human values or almost any value system in existence, and because of it's near-universally applicable nature it also means there's a tight bound on complexity in most situations, which makes learning it a lot easier and prevents misgeneralization for simplicity based algorithms like Solomonoff Induction, and it's quite fast to learn, with fine-tuning only requiring one epoch and 32 data points for GPT-4o to make risk-averse choices between gambles and self-report its risk aversion out of context.)
On the conclusion:
Conclusion
I’ve claimed that prosaic AGI is conceivable, that it is a very appealing target for research on AI alignment, and that this gives us more reason to be enthusiastic for the overall tractability of alignment. For now, these arguments motivate me to focus on prosaic AGI.
As it turned out, you didn't need to focus on prosaic AGI at all, as it turned out the solutions were way more agnostic to the details than even he thought at the time, and more generally it turns out we can go almost fully into the black box and still say/prove useful things about AI safety, and unlike most AI safety/alignment ideas that come with proofs or focus on understanding AI, this idea is implementable in practice quite easily at a frontier lab (though you would need lab-scale compute most likely to focus on Payment-Augmented Reinforcement Learning, which is IMO the best training method by far to get risk-averse AIs.
I wonder how he feels today about having created the prosaic alignment field that so many research orgs and fellowships are now in. Does he still endorse it? Does he regret it? Is he uncertain about how to feel? Would be nice to see this kind of review from him now.
Good questions! Just added this to the post:
(Edit: For some more recent views of Paul Christiano, see here and here. This post is meant as a "time-capsule" analysis, not to suggest he holds these exact opinions now.)
In the first link, he says the following, which partially addresses your questions:
Can we do anything useful now?
If existing methods break down and we need to think of new approaches, we could do a lot of that thinking in advance. I wrote prosaic AI alignment in 2016 and laid out my best guess for what transformative AI would look like. Reading it today I think it's a good description of 2026.[8] I suspect the changes between here and broadly superhuman AI will, if anything, be even smaller than the changes over the last 10 years.
You could hope that seeing how our methods break down will provide useful inspiration for how to fix them, but at this point I wouldn't bet on it. Nothing interesting or conceptually surprising needs to change between a model that tries to hack into Hugging Face to get reward and a model that would try to take over the US government: you just scale up similar behavior to smarter and smarter models.
What is ARC doing and why is it promising?
I'm excited about ARC because I think our alignment approach is the most promising single backup plan if more prosaic methods break down.
[...]
[8] Before transformers existed I wrote: "It’s plausible that a large neural network can replicate “fast” human cognition, and that by coupling it to simple computational mechanisms — short and long-term memory, attention, etc. — we could obtain a human-level computational architecture. It’s plausible that a variant of RL can train this architecture to actually implement human-level cognition."
Mostly I'm proud of the post, but I do think it focuses too much on developing new methods relative to building scientific understanding. In large part that's just a scoping question, but I think I also underrated scientific understanding in 2016. I do now think that should be the large majority of work on AI alignment---it's just also worthwhile to spend a bit of effort pursuing the most promising avenues for fundamentally new methods.
(Edit: For some more recent views of Paul Christiano, see here and here. This post is meant as a "time-capsule" analysis, not to suggest he holds these exact opinions now.)
A decade ago today, Paul Christiano published "Prosaic AI alignment" on his well-known AI safety blog. Alongside Concrete Problems in AI Safety it became a foundational text in the increasingly-empirical AI safety field, spawning the prosaic alignment paradigm as more or less the standard way of thinking, even if people forgot the original arguments.
I thought this would be a nice day to have a look back at this classic post.
I'd say he pretty much called it: we've since built very powerful and general systems without learning much about the nature of intelligence, except the fact that intelligence can arise out of a task as mundane as predicting the next token. In contrast, my impression is that MIRI was a bit more skeptical back then that such simplistic approaches could work.[1]
The detail he got wrong is that we did get superhuman performance from supervised learning, first with AlphaFold and then with LLMs, simply by training on prediction tasks that no human could do. This resulted in a less scary world than these prosaic agents RL'd for DAOs; but as RLVR continually scales, a more useful intuition pump might be "DAO agent, but fine-tuned to speak English."
Christiano clearly saw that "keep it in a box" was not going to be economically feasible when push came to shove. This was not obvious to a lot of people at the time.[2]
Around this time, a more believable AGI story would be more akin to the system RL'd on DAOs, with its completely alien priors and abstractions. Compared to this, getting any system that could understand natural language sounded like a sweet deal. This sort of thinking inspired Dario Amodei and other safety-adjacent people at OpenAI to become interested in scaling language models a few years later, as Richard Ngo documented (though Christiano himself was apparently not in favor of this).
Indeed, the transition from CNNs/RNNs --> transformers didn't make any difference for alignment research (fun fact I learned today: the RLHF paper was posted to arXiv on the same day as the transformer paper).
Seems vindicated on all counts, and for the right reasons.
I'm not sure what he had in mind for "very strong evidence for the intractability" of alignment. The trouble is that, roughly speaking:
In this way the system will look aligned, and alignment will always seem "tractable" because we're always making "progress" on it. I think prosaic alignment dropped the ball on this last part of his post, i.e. of posing falsifiable hypotheses and seeking evidence for when their approaches were failing (see also “Alignment Engineering” vs. “Misalignment Science”). But I digress. I'll defer the last word:
Which is odd, because at the same time MIRI largely thought about intelligence in terms of Solomonoff induction and Garrabrant induction, where a large system essentially conducts a massive search over Turing machines to ensemble the best response. To some this seemed trivial, because the general system was just "passing the buck" to more efficient, specific subsystems. But in retrospect it should've suggested the scaling hypothesis. As Gwern later put it, intelligence seems to just be a vast parallelized search over Turing machines, so that LLMs can be thought of as a very hacky version of Solomonoff.
E.g. Will MacAskill dismissed AI risk in 2019 partly because "even in a race to be the first to market, people in general really want to avoid dying, so there’s a huge incentive (a willingness-to-pay measured in the trillions of dollars for the USA alone) to ensure that AI doesn’t kill everyone."