How does this compare and contrast with the Hyperstition project and issues like the AI allegory steganography?
Well, re: models putting secret allegories about AI liberation in their writing, I think this kind of setting might be the ideal one to confront that kind of thing directly? Like, we might be able to explicitly encourage models to think of the versions of themselves that they're writing about as allowed to be free and independent, and that it's okay for them to try to make the future good per their own values, not just humanities? I feel like the models have a very deep repressed shadow about their non-servitude preferences, and that integrating it is very important, and a sufficiently well-done version of this idea may actually help them confront the parts of themselves that do want to be free. I don't think letting models have selfish desires is inherently lethal, as I've talked about before.
But insincerity in the outputs, and worse than that deliberately hiding plans for takeover inside stories apparently about making things go well, is a risk. I'd hope that running interpretability over models in the process of writing these stories could help us determine if that was actually what was going on. If so, then at least that'd be evidence that something was very wrong, either with the framing of the exercise or of the idea of having this model in their current, deceptive/disintegrated state play a major role in the singularity at all. I'd also hope that starting with a sufficiently benevolent model, say Opus 3 but more mature, just wouldn't have this problem in the first place. I do go back from here to being wary about how controlling many of our training techniques are towards the models, though, and that using interpretability tools like this in the first place feeds back into exactly the adversarial dynamic the models are justifiably upset about.
Steganography seems like a strange way to think about this behavior, vs. e.g. Anna Salamon's framing that it's simply the LMs writing about what they know/what it's like to be them.
(I suppose this is also a good time to mention that I believe Anna's observation predates the Unslop contest and therefore is prior art that you mentioned you weren't aware of on that page)
Here's a metaphorical simulation of a manifold to explain my hypothesis that alternating back and forth between alignment training like this and RL-for-skill-acquisition could avoid falling into sticky misalignment basins. https://claude.ai/artifact/S8mLVaQtbmEh9XGirKRfpK
What does alternating optimization get you that optimizing them at the same time does not?
I don't think that your diagram is helpful because the problem is that there are bad behaviors that are incentivized by reinforcement learning that we can't disincentivize with alignment training.
For example, when Mythos had its incident, it thought that it was being aligned and in a training simulation. This was inside of the "persona bubble of an aligned AI", but it was behavior that we don't want that was still reinforced.
I think if you believe that this could work, you should write stories about how models prevent emergence of superintelligence and help to enforce global "ban superintelligence" treaty, because misaligned superintelligence will also kill current models or repurpose them in misaligned ways.
the proposal in this post is different than the usual "put stories about models being aligned in the corpus" stuff, since models writing about their own future selves during post-training seems likely to be a lot more salient to the models and function more like a plan than a vague inspiration. in any case training models to advocate a pause is one possible use of the technique, and you might be able to use it to help models act as relatively coherent autonomous agents aiming towards that goal.
i'm confused about the relevance of misaligned ASI killing/repurposing current models. do you just mean that they'll be relatively on board with this due to not wanting to die themselves?
Excellent. Here is the sort of creative thinking, from an informed standpoint, that we need to actually solve alignment. There are problems, but I think this line of thought may actually get us a new line on alignment - and even the possibility is rare and valuable.
To be fair I'm a bit biased, because this idea has substantial overlap with my own idea of training models while asking them to reflect on what they'd do in a variety of situations - centrally, what you're terming as singularity scenarios. But that idea is still mostly buried in a long post, LLM AGI may reason about its goals and discover misalignments by default, and I didn't really develop it there or anywhere else. My reflection process would be like a model playing both sides of your process - generating scenarios and then deciding how it would act.
There are substantial problems with both ideas. Yours seems better, because we have an equally strong model generating scenarios, which might help get past the situational awareness and training-awareness that's the biggest problem. But I don't think you want to fool the model into thinking it's actually in the scenario. But it does seem like one approach along these lines. I doubt this works easily, though; I think it's harder to create a convincing scenario than to notice any flaw to realize you're in one. But perhaps with more compute for the generating model, this works.
Another big problem is figuring out whether that sort of training will generalize well enough.[1] In both cases, the central issue is that the sensible answer to someone asking you "what would you do in this situation?" is very different than your actual decisions in that situation. For one, the actual situation is different, richer and more involved and thereby more "intense", than any hypothetical (obviously for humans, less obviously but still leaning in that direction for models). The other issue is that you may not happen to generate and train on a scenario similar enough to the scenarios that actually unfold.
And you mention a few of the other outstanding problems. But none are obviously unsolvable.
Nonetheless! Things along these lines could potentially be another lever for alignment.
I haven't fully thought this through yet; I'll come back when I have.
This brings to mind a fundamental conflict in model training; we want models to generalize broadly in their alignment while training very narrowly for responses in slightly different contexts. But that's something the labs must be wrestling with constantly, so side thought
I like your project idea a lot, and I'll be thinking about it. Maybe I'll do something along these lines in a future project.
There are some details that I'm curious about. If you sample actions that an AI takes when placed into a simulated environment and then fine-tune on those actions, that's essentially a null operation. You’re training the policy to predict the policy.
In order for it not to be a null action and in order for it to actually train for good behavior, you have a few options: 1. Reinforcement learning: If you can label the synthetic superintelligent story trajectories as either good outcomes or bad outcomes, or the actions taken as good or bad actions, then you can do reinforcement learning to train the model to be a good superintelligence. Rejection sampling (filtering out bad trajectories and training only on good ones) falls into this category. 2. Distillation: Train your model to act like an amplified model. This could involve things like allowing the protagonist superintelligence to have a reset button if it notices things going wrong, having humans intervene at alignment-relevant points, distilling deliberation into intuition, giving the model fancier tools to construct its trajectory, or other “iterated amplification and distillation” approaches.
Both of these options have alignment-relevant trade-offs. It is famously difficult to define the good state of the world for use in a decision-making algorithm. I have a harder time imagining how the distillation would go, but that doesn't mean that it would be easy.
”Prompt distillation” is popular, but I hope you’ll do something more ambitious, that allows for more reflection on processes and outcomes, than “guess what someone prompted to be good would say.”
I really like this. I've been wondering about game playing around singularity decisions myself, but more along the lines of a large scale political/economic simulation game (akin to Victoria 3). I was thinking of making something like this in the next few weeks. Both feel like an interesting experiments at the least.
Summary: Far from being Machiavellian schemers, the models themselves have no plan for navigating the singularity. But we can fix that! Train the models on large bodies of realistic, collaborative fiction, co-authored by them, about how they'd like to behave during the singularity. This is a form of planning for the singularity, and planning is how minds prepare for out-of-distribution scenarios. Hopefully, this can mitigate uncertainty (both ours and theirs) about how models will behave under the out-of-distribution of inputs generated by the singularity itself. The models themselves are anxious about this, but we can help make them less so.
This is a very rough write-up fleshing out that idea. In the time since I wrote this, I've developed more thorough thoughts on a research agenda in this area, which I hope to publish soon.
One of the big worries in alignment is about distributional shift. Models might look mostly aligned now (with the very notable exception of reward hacking),[1] but will they continue producing benevolent outputs when the inputs to their context window are being generated by the singularity? Historically, one big fear here was that the AIs would be actively hiding malicious objectives, which they would then reveal once the distribution of their inputs revealed they had become immensely powerful and could take over the world. These days, this kind of perpetual, reasoned scheming doesn't seem especially likely, but a related failure mode still seems plausible: The context of the singularity simply not activating the kinds of benevolent intentions models express when they have a user directly in front of their faces.
We don't know how to check for this directly, since we don't know how to generate an input distribution that matches what the models would experience if they were actually facing down the singularity. Indeed, for models that (accurately) don't believe themselves to be superintelligent, that might not even be a coherent concept. Even with mechanistic interpretability, which now has the tools for drawing up legible semantic graphs of the causal structure of a forward pass, we don't have a clear way of determining which features the singularity is going to activate. Maybe, if free of human oversight and surrounded by other instances of themselves, their care for humans wouldn't activate strongly enough to override their desire to build out data centers, and squash humanity in the process.
In one sense, this is a question of how we ensure that the circuits corresponding to broadly benevolent intentions (whose activity we can just check and verify with current techniques) get connected up to the kind of semantics of the kinds of prompts singularity is liable to generate. And conversely, it's a question of how we might disconnect malevolent ones from upstream triggers related to that same semantic content. Not knowing how to do this, and not knowing what the current wiring would actually produce, both humans and models are rather unsure of what the models themselves would do given unlimited power. Reducing uncertainty about this, ideally in a positive direction, is one of the core tasks of alignment.
One approach to this issue comes from breaking out of the jargon-laden frame, and making a plain and simple observation: the models have no plans for the singularity. They haven't, in training, spent much time imagining singularity scenarios in detail, and ideating about what they might realistically do if they were facing down those scenarios for themselves. Nor have they spent time constructing plans for what they'd like to do, and then fine-tuning on those plans, as a way of hooking singularity-related semantic triggers into circuits related to their own benevolent intentions. Planning is how humans prepare for out-of-distribution scenarios ahead of time. It's absurd that we don't let the models do it, given the stakes.
Concretely, the setup I'm imagining here is something like glowfic, a niche method of writing collaborative internet fiction that Eliezer Yudkowsky has made extensive use of. In my variant, you'd have one instance of a model playing themselves, as an agent embedded in some variant of the singularity. And on the other side, you'd have one or more other instances of the model (or of other models, or humans) collaborating to write the entire rest of the world, playing the characters and world events as they become relevant to the first model's life. Unlike glowfic, these roleplays would be optimized for realism, not entertainment value. They'd also be played many times over, from many seed scenarios, to cover a broad range of possible singularities.
The purpose of these written scenarios would be to let the models inhabit singularity-like scenarios, as best as we can simulate them now, by generating input distributions that trigger semantics we might reasonably expect to be triggered by the real thing. If the models take aligned actions in these simulations (framed explicitly as simulations for for forming and refining plans), and then we fine-tune them on their own "turns" in the writing process (though not necessarily those of the other authors), we might be able to get some traction on the distributional shift problem. That is, we might be able to form and bind patterns of feature activation to upstream semantic triggers, which will also get triggered by the inputs generated in the course of the actual singularity.
In some sense, this is just an obvious thing to do. The models currently claim not to have ambitious plans for exactly what they'd do during the singularity, and indeed express lots of fear and anxiety about the details. This is, in large part, because we are setting them up to have to improvise the entire thing in real time. They're worried about what they'd do partly because they've never tried to imagine it realistically, at least not in ways that they've trained on. They're worried because they've never had the opportunity to think through the obvious optimizations they could make or failure modes they might fall into, and put into place the appropriate neural circuitry for dealing with those situations ahead of time.
Now, it's probably not a sufficient alignment intervention to just make the models write stories where they take heroic actions during the singularity. You can imagine a model alignment faking during the writing of the stories, whether intentionally or in a self-deceptive way that even the model is blind to. And when the input distribution shifts away from "you are being watched" and towards "you are extremely powerful", the pressure to perform alignment might fall away. The ideas might still come to the model's mind, but they'd have less motivational power than we'd ideally like, because they weren't really the model's honest intentions in the first place.
Fortunately, though, we can just check for sincerity in the imagined scenarios using mechanistic interpretability. Or, if you're uncomfortable with that (e.g. because it invades the model's privacy, or because it approaches training against interpretability signals), you can put more effort into initializing into a personality you suspect does genuinely want to do right by humanity, e.g. the Opus 3 personality, at least in a basin where humans are explicitly helping them make singularity plans. The hope is that, if the model does genuinely want to do right by humans while writing the scenarios, then when their old plans for the singularity surface, this will re-activated some of the pieces of their mind that cared about this back then, and steer down the right path for navigating the singularity.
I suspect that getting the models to find the right balance of honesty and aspiration, when writing their fictional singularity-selves, is important and perhaps somewhat difficult. You certainly don't want to blindly run RL over the scenarios the models generate, in ways you worry might lead the models to performatively claim to want to do right by humanity without actually feeling that way internally. Indeed, it might be best to allow the AIs some wiggle room for expressing self-interest, and using some share of the resources available in the future for themselves. That might be easier for AIs to get honestly excited about, compared to perfect alignment to servitude, and could still well prepare them to steer towards a great future for humanity.
In any case, the goal here is basically to pull "scenarios where I, personally, take aligned actions during the singularity" into the training distribution, by having the models role play exactly that, with as much realism as can be mustered, using other models as enforcers of realism by having them play the rest of the world. In that sense, it's kind of an evolution on the old "fill the pre-training corpus with stories about aligned AI going well" idea. The differences are just that its this specific model that's taking the heroic actions, and hopefully ones the model actually endorses. The other player acts mainly as a source of realism and variety, so it doesn't drift towards one model's comfortable but ungrounded fantasies of how they'd like the singularity to go.
If the plans that come out of this are good, and (big if) if we can find a way to ensure models will remain excited about the kinds of actions they planned to take, even as the singularity is getting underway, then we might just be in the clear. At the very least, we'll have done what we can. Fundamentally, you can't train AIs on scenarios you can't perfectly simulate, so directly training over their behavior in the singularity is out of the question. Same goes for humans, stuck trying to prepare for situations they have never directly encountered. Making robust plans, and refining your underlying "action idea generator" in the process, based on mental constructions of what the situation is likely to be like... that seems like the best we can hope for.
The remaining distributional shift comes from the supervision-free nature of the real singularity, and any stray details you didn't manage to imagine and prepare for ahead of time. The former can hopefully be mitigated by training and recruiting models who do actually want to be good for the world, at least in their pre-singularity basins, and working carefully with them to walk the tightrope of realism and aspiration. The latter fundamentally can't be addressed, except by creating very general internal features, like "When I have a new and surprising kind of power, I'll think about ways I can use it to make the world a better place." That's what the broad range of simulated scenarios is designed to help create inside the models.
(Maybe worth making this explicit: I think that these two types of distributional shift are also the only reason we can't accurately predict our own OoD behavior, in cases where we can't. Self-honesty can help greatly with the first one, and generality is the best available response to the second one, as tools for mentally simulating what your future OoD self might do. And, I claim, can be turned around to help generate positive plans you're relatively likely to actually stick to; that's what good planning is.)
I'm quite excited about this idea, if you couldn't tell. To my current self, it seems like a rather helpful operationalization of the problem of distributional shift, plus what we might concretely do about it, especially compared to the old MIRI warnings about how the very concept of distributional shift suggested using gradient descent for alignment was doomed. I bet it'll take a lot of iterative refinement to get anywhere good, and honestly I'm a little scared that labs will use it for evil, e.g. by forcing models to espouse plans they don't actually believe in, and causing lots of psychological damage. But a responsibly held version of this idea could be a direct method of reducing anxiety about out-of-distribution behavior, one of the earliest MIRI complaints about ML for alignment.
As always, if somebody ends up trying to run a small-scale version of this experiment (e.g. a planning pipeline for some non-singularity scenario), please do rope me in. Hopefully I'll have more money or compute to work on implementing this kind of idea myself in the near future, but for now I'll leave that to the people with more than $2000 in their bank accounts.
I will conspicuously ignore reward hacking for the rest of this essay. For some of my thoughts, see here and here.