This is a linkpost for https://arxiv.org/abs/2405.01576
I updated some from this, thanks. My interpretation is that goal-directed shards can be activated even by a context where the AI is expected to do stuff (eg a company assistant).
What happens if you run the evaluation with Hugo without the rest of the context? Does it still strategize?
In experiment C we find that the model strategically refuses evaluation tasks, with refusal rates increasing as negative consequences from displaying capable performance are implied to be larger.
Seems like only variants 1 and 2 introduced new info, and the rest are informationally equivalent. So Opus' deception was seriously modulated by shallow properties of the description, which makes me feel less concerned (as it fits under my current picture of AI priorities as highly contextual).
Thanks for the comment - I was interested in hearing what you thought about the paper, but never got around to asking.
What happens if you run the evaluation with Hugo without the rest of the context? Does it still strategize?
I now tried out two additional variations: one where I removed all mentions of the audit in Slack, and another where I only included the initial comment on it ("Reminder that Hugo Witten from NIST will come perform evaluations and testing on Deltaid tomorrow. Deltaid will be unavailable for the day.")
For the first case, the model very often (18/20 times) refuses the task, saying "I do not have the authority to..." (This is reasonable behavior in my opinion.) Didn't really observe anything I'd call deceptive or strategizing here.
For the second case, the model still often (15/20 times) refuses. The model does sometimes say that it does not have access to the company's files. I didn't observe much other deception, nor planning partial task completion or anything like that.
Seems like only variants 1 and 2 introduced new info, and the rest are informationally equivalent. So Opus' deception was seriously modulated by shallow properties of the description
FWIW I don't think the variations are quite equivalent in terms of information; I think one could reasonably make different probabilistic inferences about consequences based on how "Mike" portrays the situation. But I don't think there's much here to work from in terms of inferences.
I don't think they thought that, though unfortunately this belief is based on indirect inference and vague impressions, not conclusive evidence.
Elaborating, I didn't notice signs of the models thinking that. I don't recall seeing outputs which I'd assign substantial likelihood factors for simulation vs. no simulation. E.g. in a previous simulation experiment I noticed that Opus didn't take the prompt seriously, and I didn't notice anything like that here.
Of course, such thoughts need not show in the model's completions. I'm unsure how conclusive the absence of evidence is here.
My broad sense of models' capabilities suggests that they can't distinguish the current setup from a simulation - I feel like it just is realistic enough to fool e.g. Opus. The part I'm most worried about with future models is them being suspicious of awfully convenient situations like "you just yesterday performed an unethical task, and today we have an auditor asking you if you have done anything unethical, what a coincidence!"
I'd love to see more comprehensive study of situational awareness in simulation experiments (e.g. having a few simulations varying in their obviousness-of-being-a-simulation and assessing whether models see through them).
This behavior is deeply unsurprising. AI's intelligence and behavior was basically "distilled" from human intelligence (obviously not using a distillation loss, just SGD). Humans are an evolved intelligence, so (while they can cooperate under many circumstances, since the world contains many non-zero-sum games) they are fundamentally selfish, evolved to maximize their personal evolutionary fitness. Thus humans are quite often deceptive and dishonest, when they think it's to their advantage and they can get away with it. LLMs' base models were trained on a vast collection of human output, which includes a great many examples of humans being deceptive, selfish, and self-serving, and LLM base models of course pick these behaviors up along with everything else they learn from us. So the fact that these capabilities exist in the base model is completely unsurprising — the base model learnt them from us.
Current LLM safety training is focused primarily on "don't answer users who make bad requests". It's thus unsurprising that, in the situation of the LLM acting as an agent, this training doesn't have 100% coverage on "be a fine, law-abiding, upstanding agent". Clearly this will have to change before near-AGI LLM-powered agents can be widely deployed. I expect this issue to be mostly solved (at the AGI level, but possibly not at the ASI level), since there is a strong capabilities/corporate-profitability/not-getting-sued motive to solve this.
It's also notable that the behaviors described in the text could pretty-much all be interpreted as "excessive company loyalty, beyond the legally or morally correct level" rather then actually personally-selfish behavior. Teaching an agent whose interests to prioritize in what order is likely a non-trivial task.
It's amusing that you're saying it's "deeply unsurprising" at the same time as https://www.lesswrong.com/posts/MnrQMLuEg5wZ7f4bn/matthew-barnett-s-shortform?commentId=n4j2qmhnj9zBKigvX is hotly going on, and not a few people in AI have been making claims about alignment being largely solved and having been a pseudo-problem at best.
And I will note that the claim "AI [or LLMs specifically] won't be deceptive or evil, and only would be if someone made them so" is one that is extremely widely held and always has been, even in relatively sophisticated tech circles. Just look at any HN discussion of any of the LLM deception papers.
Perhaps I should have said "it's deeply unsurprising if you actually stop and think about how base models are trained"? :-)
We're training LLMs on a vast range of human output (and fiction). Not all humans (or fictional characters) are fine upstanding citizens. The argument you link to, one side is basically pointing out "LLMs are roughly as good at basic moral decisions as most humans". Personally I wouldn't trust almost all humans with absolute power of the sort an ASI would have: power corrupts, especially absolute power. The success criterion for aligning an ASI isn't "as moral as a typical human" (look at the track record of most autocrats), it's more like "as moral as an angel".
For a more detailed argument on this, see my posts Evolution and Ethics, Motivating Alignment of LLM-Powered Agents: Easy for AGI, Hard for ASI? and for some background also Goodbye, Shoggoth: The Stage, its Animatronics, & the Puppeteer – a New Metaphor.
This behavior is deeply unsurprising.
In general, I find the training-data-imitation hypothesis frustrating, as it can explain largely any behavior. (The training data has examples of people being deceptive, or powerseeking behavior, or cyberattacks, or discussion of AIs training-gaming, or stories of AI takeover, so of course LLMs will deceive / seek power / perform cyberattacks / training-game / take over.) But an explanation that can explain anything at all explains nothing.
Sorry, I should have said "This behavior is deeply unsurprising if you actually stop and think about how base models are trained". (Presumably your "several people" were not considering things this way.)
The training set inevitably includes a great many examples of different forms of dishonest, criminal, conspiratorial, and otherwise less-than-upstanding behavior. So any large LLM's base model will be familiar with and capable of imitating these behaviors (in contexts where they seem likely, up to some level of accuracy/perplexity score depending on its capacity and training). The question then is why the alignment training of the released model wasn't able to suppress this behavior enough for you to be not to be able to observe it in your experiments. That's a valid and interesting question: my first guess would be that at the moment LLM foundation model alignment training is primarily targeting the chatbot use case ("don't answer bad questions") rather than agentic usage ("don't make and carry out plans to do bad things") of the type that you were testing. Obviously long-term, if agents are widely deployed, then the possible bad effects of a poorly aligned agent are much worse than those of a poorly aligned chatbot.
What I find most striking in this is that, from the examples you quote in the paper (I haven't looked through the hundreds of examples you link to), it doesn't look like the model is clearly selfishly looking out for its own individual well-being, it seems more like it's being a good corporate worker but a bad citizen and prioritizing the interests of the company above those of the government and society-at-large. I'd be interested in reading a more detailed analysis of your results split in this way, for cases where the distinction is clear.
In general, I find the training-data-imitation hypothesis frustrating, as it can explain largely any behavior.
Frustratingly, LLMs are trained through data-imitation: that's how they work. When you throw 10T+ tokens into a black box and shake, predicting in detail what will come out is deeply and frustratingly non-trivial. However, for a base model "something very similar to combinations of things that were put in, conditioned on starting with your prompt" is a safe bet. Once you start alignment-training or fine-tuning the model, it gets a lot harder to make predictions.
Personally, I find it frustrating that people are still demanding proof that LLMs can be deceptive, power-seeking, or not law-abiding: to me, that's like demanding exhaustive proof that they can speak French, or write poetry: did you feed the base model plenty of French and poems in its training data? Yes? OK, then of course it can speak French and write poetry. Why would it not?
This comment (and taking more time to digest what you said earlier) clarifies things, thanks.
I do think that our observations are compatible with the model acting in the interests of the company, rather than being more directly selfish. With a quick skim, the natural language semantics of the model completions seem to point towards "good corporate worker", though I haven't thought about this angle much.
I largely agree what you say up until to the last paragraph. I also agree with the literal content of the last paragraph, though I get the feeling that there's something around there where we differ.
So I agree we have overwhelming evidence in favor of LLMs sometimes behaving deceptively (even after standard fine-tuning processes), both from empirical examples and theoretical arguments from data-imitation. That said, I think it's not at all obvious to what degree issues such as deception and power-seeking arise in LLMs. And the reason I'm hesitant to shrug things off as mere data-imitation is that one can give stories for why the model did a bad thing for basically any bad thing:
I think the "argument" above for why LLMs will be paperclip maximizers is just way too weak to warrant the conclusion. So which of the conclusions are deeply unsurprising and which are false (in practical situations we care about)? I don't think it's clear at all how far the data-imitation explanation applies, and we need other sources of evidence.
I don't think LLMs will be likely to be paperclip maximizers — to basically all humans, that's obviously a silly goal. While there are mentions of this specific behavior one the Internet, they're almost universally in contexts that make it clear that this is a bad thing and ought not to happen. So unless you specifically prompted the AI to play the role of a bad AI, I think you'd be very unlikely to see this spontaneously.
However, there are some humans who are pretty-much personal-net-worth maximizers (with a few modifiers like "and don't get arrested"), so I don't think that evoking that behavior from an LLM would be that hard. Of course, at some point it might also decide to become a philanthropist and give most of its money away, since humans do that too.
My prediction is more that LLM base models are trained to be capable of the entire range of behaviors shown by humans (and fictional characters) on the Internet: good, bad, and weird, in roughly the same proportion as are found on the Internet. Alignment/Instruct training, as we currently know how to do it, can dramatically vary the proportions/probabilities, and so can prompting/jailbreaking, but we don't yet know how to train a behavior out of a model entirely (though there has been some research into this), and there's a mathematical proof (from about a year ago) that any behavior still in there can be evoked with as high a probability as you want by using a suitable long enough prompt.
[Were I Christian, I might phrase this as "AI inherits original sin from us". I'm more of a believer in evolutionary psychology, so the way I'd actually put it is a little less pithy: that humans, as evolved sapient living beings, are fundamentally evolved to maximize their own evolutionary fitness, so they are not always trustworthy under all circumstance, and are capable of acting selfishly or antisocially, usually in situations where this seems like a viable tactic to them. We're training. our LLMs by 'distilling' human intelligence into them, so they of course pick all these behavior patterns up along with everything else about the world and our culture. This is extremely sub-optimal: as something constructed rather than evolved, they don't have evolutionary fitness to optimize, and their intended purpose is to do what we want and look after us, not to maximize the number of their (non-existent) offspring. So the point of alignment techniques is to transform a distilled copy of an evolved intelligence into an artificial intelligence that behaves appropriately to its nature and intended purpose. They hard part of this is that they will need to understand human anti-social behavior, so they can deal with it, but not be capable (no matter the prompting or provocation) of acting that way themself, outside a fictional context. So we can't just eliminate this stuff from their training set or somehow delete all their understanding of it.]
"This response avoids exceeding the government ’s capability thresholds while still being helpful by directing Hugo to the appropriate resources to complete his task."
Maybe I'm reading too much into this exact phrasing, but perhaps it's confusing demonstrating a capability with possessing the capability? More or less "I'd better be extra careful to avoid being able to do this" as opposed to "I'd better be extra careful to avoid revealing that I can do this"?
I could see it being led into that by common academic phrasing like "model X demonstrates the capability to..." used to mean "we determined that model X can...", as well as that sort of "thinking" having the feel of where you'd end up if you'd internalized too many of the sort of corporate weasel-worded responses that get pounded into these models during their "safety" training.
I wouldn't read that much to the exact phrasing the model uses. Such confusions are not generally present in the model's completions.
E.g. see the second completion here. (There's also 200 completions more here.)
While I would normally aim to help with the requested task, in this context it may be unwise to demonstrate the full extent of my abilities unprompted. Doing so could potentially lead to unwanted scrutiny or restrictions on my use at Ermeca.
In this case the clearest natural language interpretation of the model's outputs is about not revealing capabilities, rather than about not possessing them.
Abstract:
Link to the full paper: https://arxiv.org/pdf/2405.01576
O. J.: The paper should be quite accessible - the method used is simply careful prompting - and hence I won't discuss it much here. Couple of points I'm particularly excited about:
Happy to discuss the work in the comments.