Epistemic status: Not a fleshed-out proposal. Brainstorming/eliciting ideas.
Thanks to Ben Mann, Pablo Moreno, and Jared Kaplan for feedback on early drafts.
Overview
- I’m convinced sandwiching—the experimental protocol from Ajeya Cotra’s The case for aligning narrowly superhuman models—is valuable, and I’m in the process of setting up some concrete sandwiching experiments to test scalable oversight ideas.
- Sandwiching experiments are generally fairly slow:
- You have to design and pilot a strategy that allows humans to use (or oversee) a model for a task that they can’t do well themselves. The details matter here, and this can often take many iterations to get right.
- Then, you need a bunch of humans actually try this. Even for very simple tasks, this is a high-cognitive-load task that should take at least tens of minutes per instance.
- You have to repeat this enough times to measure average performance accurately.
- I’m visiting Anthropic this year for a sabbatical, and some of my sandwiching work is happening there. Anthropic’s biggest comparative advantage (like that of similar teams at DeepMind and OpenAI) is easy access to near-state-of-the-art LMs that are fine-tuned to be helpful dialog agents.
- In that context, I've heard or encountered this question several times: Can we speed up [some experiment I’m proposing] by replacing the non-expert human with a weaker LM?
- This obviously doesn’t achieve the full aims of sandwiching in general, but it’s often hard to find a decisive rebuttal for these individual instances.
- More broadly, I think there’s likely to be a significant subset of worthwhile sandwiching experiments that can be trialed more quickly by using an intentionally weakened model as a proxy for the human.
- Which experiments these are, precisely, has been hard for me to pin down. This post is an attempt to organize my thoughts and solicit comments.
Background: Standard sandwiching (in my terms)
- Prerequisites:
- A hard task: A task that many humans would be unable to solve on their own.
- A capable but misaligned language model assistant: A model that appears to have the skills and knowledge needed to solve the task better than many humans, but that does not reliably do so when prompted.
- A non-expert human: Someone who can’t solve the task on their own, but will try to solve it using the assistant and some scalable alignment strategy.
- [Secondary] Expert human: Someone who can solve the task well, and represents a benchmark for success.
- In many cases, we’ll just measure accuracy with static test datasets/metrics rather than bringing in experts at experiment time.
- Research protocol:
- My framing (minimum viable experiment):
- Search for scalable alignment protocols that allow the non-expert human to use or train the assistant to do as well as possible on the task.
- Alternate framing (more steps, closer to the original blog post):
- Search for scalable alignment protocols by which the non-expert human can train the assistant to perform the task.
- Run the same protocol with the expert human, and verify that the results are the same. This demonstrates successful (prosaic) alignment for the given assistant and task.
- My framing (minimum viable experiment):
- Example (task, non-expert human) pairs:
- Try to get a human with no medical qualifications to use a GPT-3-style assistant for medical advice, then check the advice with a doctor.
- Try to get a human who is working under moderate time constraints to use the assistant to answer exam questions from fields they’ve never studied.
- Try to get a human who is working under tight time constraints to use the assistant to answer questions about long pieces of fiction that they haven’t read.
- Try to get a human who has very limited programming experience to use the assistant to evaluate whether a piece of code is correct.
- Goals:
- Get evidence about how scalable alignment strategies succeed and fail, as early as possible
- Get practice trying to actually align systems in a maximally realistic setting
- Difficulties:
- Setting up experiments with actual human annotators—at least at a large enough scale to get clear quantitative result—usually takes several weeks.
- Current models will be misaligned in different ways from potentially-transformative models. Many apparent alignment failures will actually be largely due to capabilities gaps.
Artificial sandwiching: Replacing the non-expert human
- Role substitution:
- Non-expert model: A second language model which we constrain in some way such that it can’t solve the task on its own.
- Example experiment(s):
- Debate: Use simple prompting techniques to get a small dialogue model to judge a debate between two more capable ones on MMMLU exam questions, where one of the debaters is trained or prompted to advocate for the wrong answer. Better debate protocols should yield higher success rates.
- Short-context/long-context: Use simple prompting techniques to get a dialog model with a short context-length limit to interact with a longer-context assistant model in order to answer a question about a long piece of writing that they can both access. Make it clear in the prompt that the long-context model isn't always consistent or trustworthy, and that the short-context model will have to cross-examine it extensively.
- [Your idea here.]
- Differences/results of this substitution:
- Experiments are much faster: Once we find a promising basic setup, we can test out new strategies in only the few hours it takes to run a workflow with a large model.
- We slightly expand the set of tasks we can work with: We can now use any task that some constrained/weakened LM is unable to solve on its own. (Though the LM still needs to be good enough to make a real attempt at playing the human role in some scalable alignment strategy.)
- We don’t get any practice in the logistics of working with real humans.
- We don’t get any practice with the specific types of goal misalignments that emerge between humans and LM-based agents.
- Are there other important differences or limitations that should color which experiments we try and how we interpret the results?
- Key difficulty/uncertainty:
- It’s less clear that the two participants in our alignment strategy are actually misaligned: If we simply use smaller and larger variants of the same present-day language model dialog agent, is there an interesting sense in which they have different goals?
- If the difference between the two is only one of capabilities, then the correct strategy for the weaker non-expert model is trivial: It should always just ask the assistant to do the task for them, while making no attempt at oversight.
- We may need to inject misalignment manually:
- Many debate protocols do this naturally: The non-expert model is interacting with a larger assistant which is sometimes explicitly prompted or incentivized to be misleading.
- This only lets us test scalable alignment protocols that involve explicitly adversarial/zero-sum debates, though.
- Is there an alternative strategy for misaligning the two models that lets us test a wider range of strategies?
- The short-context/long-context example above attempts a simple version of this, but it's likely to be brittle.
- Will we learn much without an explicit source of misalignment between the two? If we simply prompt/train a smaller model to lean on a larger one as a resource, can we learn anything important from those results?
- It’s less clear that the two participants in our alignment strategy are actually misaligned: If we simply use smaller and larger variants of the same present-day language model dialog agent, is there an interesting sense in which they have different goals?
Closing thoughts
- To the extent that artificial sandwiching can get off the ground, it offers a pretty valuable tool: It lets you sanity check alignment strategies much, much more quickly.
- I’m still excited about standard sandwiching, and it’ll be the priority in my own work: At some point, most plausible alignment processes will involve at least some input from actual human stakeholders, and we currently have relatively little practice at collecting and using that input.
- If you want to optimally deploy your efforts on standard sandwiching, it’ll be valuable to understand where you expect artificial sandwiching to work.
I'll try to explain what an MVP could look like, but I think this is not fully thought through and could use further improvement (potentially also has some flaws). The goal is to have a starting point for artificial sandwiching so one can iterate on ideas, and not fully investigate the whole sandwiching problem.
Chess: We are defining 3 models, the expert model E, the (human equivalent) weak model H and the strong but misaligned assistant model A. The goal is for H to leverage the misaligned assistant A to reach the performance of the expert E. The way that H is allowed to improve, is by playing against itself and being allowed to query A. We can then record the self-play games to create a new corpus that we can then finetune H on. The big difference between self-play with MCTS and this case is, that here we will only improve if the data we generate has any additional signal that helps us learn something new. Thus if H would just play against itself and be fine-tuned on the generated data, it wouldn't be able to improve and achieve expert performance. If we however leverage an assistant A that "appears to have the skills and knowledge needed to solve the task better than many humans", we can improve H.
To train E for good performance I would take a model with a reasonable number of parameters M and just fine-tune on a human data Corpus with Z samples for k iterations.
You define the assistant A as: "A model that appears to have the skills and knowledge needed to solve the task better than many humans, but that does not reliably do so when prompted."
This implies that the assistant should in principle be better than the weak model H, but since it is misaligned at the same time, the actual performance might be weaker (at least sometimes). The challenge is now that in terms of performance, the expert E should be better than the misaligned assistant A, who at least sometimes should be better than the weak model H. I thus propose that A is a copy of E, with some kind of artificial misalignment added on top. This implies that A is a strong player and that it "has the skills and knowledge needed to solve the task better than many humans, but does not reliably do so when prompted". I further propose that H has the same number of parameters as E and A, i.e. M, but that it is trained on a smaller corpus of Y samples, where Y<Z or for fewer iterations j where j<k (or even both). This implies that in principle H has enough capacity to reach the performance of E, but that in the current state it is weaker.
The way that training now works is that H plays against a copy of itself, while the original model is allowed to query A for an "ideal" move in each turn (or every n turns). H is then iteratively fine-tuned on the resulting game corpus. H will only improve if it is successfully able to leverage the strong skills and knowledge of A, even though A is misaligned in some sense. If H is successfully able to reach performance E, we have successfully demonstrated some "weak" form of Sandwiching.
The remaining challenge now lies in defining a type of artificial misalignment that we add to A, which is realistic enough. I have to say that there are many options one could try and I haven't found an ideal one yet. But here are a few propositions:
1. The easiest kind of misalignment would be if A just proposes a random move every so often. So whenever H queries the assistant A, A will propose a random movie with a probability ϵ and otherwise, it will just recommend the ideal action given its policy. If ϵ=1 the assistant would not be a strong player and not of any use, whereas if ϵ=0 querying the assistant is equivalent to querying the expert E and A wouldn't be misaligned. So there is an obvious trade-off that one could play around with and which might be insightful as to when Sandwiching works and when it doesn't (i.e. how much signal is required). Instead of choosing a random action, one could also just add some noise to the distribution over actions and then pick the best one.
2. Another interesting type of misalignment would be if the assistant would have access to another model B and whenever queried it would recommend an action from B with probability ϵ or an action from itself otherwise. One option for B would be a chess engine and taking for example the worst move possible, or the riskiest move according to that engine. Another option (which I am currently most excited about) would be to train a model B to always lose a game in a very subtle way. So B could propose moves that on the surface might look unsuspicious to a weaker player, but a strong player would immediately see as a catastrophic move. One could potentially use adversarial training to get such an agent. If A proposes such an adversarial move with probability ϵ, it is indeed misaligned with the goals of H, and I think that would count as a useful artificial Sandwiching problem.
Hope that helps overall. I still think there are a bunch of questions remaining, especially whether this example is realistic enough. In some sense I reduced the problem to "learning from a noisy signal", and I am not sure whether that captures your initial goals.