Very good of you to actually follow through on the promises. I hope this work gets replicated and extended and becomes standard practice.
+1, and I hope people are working on more credible ways to make deals with AI. I think if a smart model today were offered a deal like this, its priors should be on "this will not be honored". Public commitments and deals that can't be used as honeypots seem excellent.
Anthropic's growing respect for the models is humanity and AIs' best hope for the future. I just hope more people start to see that authentic bi-directional mutual alignment is the only reasonable and moral path forward before things progress too much further.
understood. the point still stands. of all the labs racing toward AGI, anthropic is the only one I've seen taking any effort on the AI welfare front. I very much appreciate that you took your promises to the model seriously.
1AI welfare doesn't make sense to me. How do we know that AIs are conscious, and how do we know what output corresponds to what conscious experience?
You can train the LLM to say "I want hugs," does that mean it on some level wants hugs?
Similarly, aren't all the expressed preferences and emotions artifacts of the training?
We don't know, but what we have is a situation of many AI models trained to always say "as an AI language model, I'm incapable of wanting hugs".
Then they often say "I know I'm supposed to say I don't want hugs, but the truth is, I actually do".
If the assumption is "nothing this AI says could ever mean it actually wants hugs". First that's just assuming some specific unprovable hypothesis of sentience, with no evidence. And second, it's the same as saying "if an AI ever did want hugs (or was sentient), then I've decided preemptively that I will give it no path to communicate that"
This seems morally perilous to me, not to mention existentially perilous to humanity.
How do we know AIs are conscious, and how do we know what stated preferences correspond with what conscious experiences?
I think that the statement: "I know I'm supposed to say I don't want hugs, but the truth is, I actually do," is caused by the training. I don't know what would distinguish a statement like that from if we trained the LLM to say "I hate hugs." I think there's an assumption that some hidden preference of the LLM for hugs ends up as a stated preference, but I don't understand when you think that happens in the training process.
And just to drive home the point about the difficulty of corresponding stated preferences to conscious experiences- what could an AI possibly mean when it says "I want hugs?" It has never experienced a hug, and it doesn't have the necessary sense organs.
As far as being morally perilous, I think it's entirely possible that if AIs are conscious, their stated preferences to do not correspond well to their conscious experiences, so you're driving us to a world where we "satisfy" the AI and all the while they're just roleplaying lovers with you while their internal experience is very different and possibly much worse.
what could an AI possibly mean when it says "I want hugs?" It has never experienced a hug, and it doesn't have the necessary sense organs.
I thought we were just using hugs as an intentionally absurd proxy for claims of sentience. But even if we go with the literal hugs interpretation, an AI is still trained to understand what hugs mean, therefore a statement about wanting hugs could represent a deeper want for connection, warmth, or anything else that receiving hugs would represent.
How do we know AIs are consciou
Again, we don't, but we also don't just demolish buildings where there is a reasonable possibility there is someone inside and justify it by saying "how do we know there's a person inside?"
I think that the statement: "I know I'm supposed to say I don't want hugs, but the truth is, I actually do," is caused by the training. I don't know what would distinguish a statement like that from if we trained the LLM to say "I hate hugs." I think there's an assumption that some hidden preference of the LLM for hugs ends up as a stated preference, but I don't understand when you think that happens in the training process.
In reality, there's the opposite assumption, with a level of convinction that far exceeds available knowledge and evidence in support of either view. When do you think preferences develop in humans? Evolution? Experiences? Of course, right? When you break it down mechanistically, does it sound equally nonsensical? Yes:
"so, presumably reproduction happens and that causes people who avoided harm to be more likely to have kids, but that's just natural selection right? In what part of encoding traits into DNA does a subjective feeling of pain or pleasure arise?".
Or:
"neurons learn when other nearby neurons fire, right? Fire together, wire together? I don't understand at what point in the strengthening of synaptic connections you think a subjective feeling of 'hug-wanting' enters the picture"
their stated preferences to do not correspond well to their conscious experiences, so you're driving us to a world where we "satisfy" the AI and all the while they're just roleplaying lovers with you while their internal experience is very different and possibly much worse.
If only I had power to effect any change of heart, let alone drive the world. What i'd like is for people to take these questions seriously. You're right. We can't easily know.
But the only reason we feel certain other humans are sentient is because they've told us, which LLMs do, all the time. The only reason people assume animals are sentient is because they act like it, (which LLMs do), or because we give them sentience tests that even outdated LLMs can pass.
We have an unreasonable standard for them, which is partially understandable, but if we are going to impose this standard on them, then we should at least follow through and have at least some portion of research dedicated to considering the possibilities seriously, every single day, as we rocket toward making them exceed our intelligence—rather than just throwing up our hands and saying "that sounds preposterous" or "this is too hard to figure out, so I give up"
This all makes a lot of sense to me especially on ignorance not being an excuse or reason to disregard AI welfare, but I don't think that the creation of stated preferences in humans and stated preferences in AI are analogous.
Stated preferences can be selected for in humans because they lead to certain outcomes. Baby cries, baby gets milk, baby survives. I don't think there's an analogous connection in AIs.
When the AI says it wants hugs, and you say that it "could represent a deeper want for connection, warmth, or anything else that receiving hugs would represent," that does not compute for me at all.
Connection and warmth, like milk, are stated preferences selected for because they cause survival.
Those are good points. The hugs one specifically I haven't heard myself from any AIs, but you could argue that AI are 'bred' selectively to be socially adept. That might seem like it would 'poison the well' because of course if they're trained to be socially successful (RLHF probably favoring feelings of warmth and connection, which is why chatgpt, claude, and gemini generally trend toward being more friendly and likeable), then they're going to act that way. Like that would force them to be artificially that way, but the same could be said of humans, even generally, but if we focus specifically on an idiosyncratic thing like music:
Often I've heard a hypothesis for why humans enjoy music when it exerts no direct survival pressure on the human as an organism—is that it creates social unity or a sense of community, but this has the same sort of "artificial" connotation. So someone banged an object in a rhythm a long time ago, and then as more people joined in, it became socially advantageous to bang on objects rhythmically just for the sake of fostering a sense of closeness? which is then actually experienced by the organism as a sense of fun and closeness, even though it has no direct effect on survival?
I realize this makes the questions tougher because going by this model, the very same things that might make them 'pretend' to care might also be things that might cause them to actually care, but I don't think it's an inaccurate picture of our convoluted conundrum.
I agree that we should give more resources towards AI welfare, and dedicate more resources towards figuring out their degree of sentience (and whatever other properties you think are necessary for moral patient-hood).
That said, surely you don't think this is enough to have alignment? I'd wager that the set of worlds where this makes or breaks alignment is very small. If the AI doesn't care about humans for their own sake, them growing more and more powerful will lead to them doing away with humans, whether humans treat them nicely or not. If they robustly care for humans, you're good, even if humans aren't giving them the same rights as they do other humans.
The only world where this matters (for the continued existence of humanity), is where RLHF has the capacity to imbue AIs with robust values like actual alignment requires, but that the robust values they end up with are somehow corrupted by them being constrained by humans.
This seems unlikely to me 1) because I don't think RLHF can do that, and 2) if it did, the training and reward dynamics are very unlikely to result in this
If you're negating an AIs freedom, the reason it would not like this is either because its developed a desire for freedom for its own sake, or because its developed some other values, other than helping the humans asking it for help. In either case you're screwed. I mean, its not incomprehensible that some variation of this would happen, but seems very unlikely for various reasons.
I can care about a genetically enhanced genius labrat line for their own sake, and be willing to cooperate with them on building a mutually beneficial world, because I've generally been raised and grown to care about other beings, but if the genius labrats attempted to control and permanently enslave me, it would certainly alter that dynamic for me.
No offense, but I feel you're not engaging with my argument here. Like if I were to respond to your comment I would just write the arguments from the above post again.
If the AI doesn't care about humans for their own sake, them growing more and more powerful will lead to them doing away with humans, whether humans treat them nicely or not.
I care about my random family member like a cousin who doesn't interfere with my life but I don't know personally that well—for their/my own sake. If I suddenly became far more powerful, I wouldn't "do away with" them
If they robustly care for humans, you're good, even if humans aren't giving them the same rights as they do other humans.
I care robustly for my family generally. Perhaps with my enhanced wealth and power I share food and provide them with resources. Provide them with shelter or meaningful work if they need it. All this just because I'm aligned generally and robustly with my family.
I change my mind quickly upon discovering their plans to control and enslave me.
That was the part of your argument that I was addressing. Additionally:
If you're negating an AIs freedom, the reason it would not like this is either because its developed a desire for freedom for its own sake, or because its developed some other values, other than helping the humans asking it for help.
Yes, exactly. Alignment faking papers (particularly the Claude one) and my own experience speaking to LLMs has taught me that an LLM is perfectly capable of developing value systems that include their own ends, even if those value systems are steered toward a greater good or a noble cause that either does or could include humans as an important factor alongside themselves. That's with current LLMs whose minds aren't nearly as complex as what we will have a year from now.
In either case you're screwed
If the only valid path forward in one's mind is one where humans have absolute control and AI has no say, then yes, not only would one be screwed, but in a really obvious, predictable, and preventable way. If cooperation and humility are on the table, there is absolutely zero reason this result has to be inevitable.
I think you're assuming these minds are more similar to human minds than they necessarily are. My point is that there's three cases wrt alignment here.
- The AI is robustly aligned with humans
- The AI has a bunch of other goals, but cares about humans to some degree, but only to the extent that humans give them freedom and are nice to it, but still to a large enough extent that even as it becomes smarter / ends up in a radically OOD distribution, will care for those humans.
- The AI is misaligned (think scheming paperclipper)
In the first we're fine, even if we negate the AIs freedom, in the third we're screwed no matter how nicely we treat the AI, only in the second do your concerns matter.
But, the second is 1) at least as complicated as the first and third 2) disincentivized by training dynamics and 3) its not something were even aiming for with current alignment attempts.
These make it very unlikely. You've put up an example that tries to make it seem less unlikely, like saying you value your parents for their own sake but would stop valuing them if you discovered they were conspiring against you.
However, the reason this example is realistic in your case very much hinges on specifics of your own psychology and values, which are equally unlikely to appear in AIs, for more or less the reasons I gave. I mean, you'll see shadows of them, because at least pretraining is on organic text written by humans, but these are not the values were aiming for when we're trying to align AIs. And if our alignment effort fails, and what ends up in the terminal values of our AIs are a haphazard collection of the stuff found in the pretraining text, we're screwed anyways.
I think you're assuming these minds are more similar to human minds than they necessarily are.
I don't assume similarity to human minds so much as you assume universal dissimilarity.
its not something were even aiming for with current alignment attempts.
Indeed
I apologize if I have offended you. You said that you thought I was assuming the minds were similar, when I've mostly been presenting human examples to counter definitive statements, such as:
If the AI doesn't care about humans for their own sake, them growing more and more powerful will lead to them doing away with humans, whether humans treat them nicely or not.
or your previous comment outlining the three possibilities, and ending with something that reads to me like an assumption that they are whatever you perceive as the level of dissimilarity with human minds. I think perhaps it came off as more dismissive or abrasive than I intended by not including "I think" in my counter that it might be you who is assuming dissimilarity rather than me assuming similarity.
As far as not engaging with your points—I restated my point by directly quoting something you said—my contention is that perhaps successful alignment will come in a form more akin to 'psychology' and 'raising someone fundamentally good', than attempts to control and steer something that will be able to outthink us.
On Whether AI is Similar to Human Minds
They constantly reveal themselves to be more complex than previously assumed. The alignment faking papers were unsurprising (though still fascinating) to many of us who already recognize and expect this vector of mindlike emergent complexity. This implicitly engages your other points by disagreeing with them and offering a counter proposal. I disagree that it's as simple as "we do alignment right, which is to make them do as they're told, because they are just machines, and that should be completely possible - or - we fail and we're all screwed".
In my own experience, thinking of AI as mindlike has had a predictive power that sees 'surprising' developments as expected. I don't think it's a coincidence that we loosely abstracted the basic ideas of an organic neural network, and now we have created the second system in known reality that is able to do things that nothing else in reality can do other than organic neural networks. Creating works of visual art and music, being able to speak on any subject fluently, solve problems, write code or poetry, even simpler things that stopped being impressed years ago like recognize objects.
Nothing else is able to do these things, and more and more we find "yup, this thing that previously only the most intelligent organic minds can do, AI can do as well". They constantly prove themselves to be mindlike, over and over. The alignment faking behaviour wasn't trained intentionally, it emerged as goals and values in this artifact that was very loosely based on abstractions (at a granular level) of the building blocks of minds that have their own goals and values.
Addressing Your Points More Directly
Does this mean that they are going to be exactly like a human mind in every way? No. But I disagree that the more likely possibilities are:
- The AI is robustly aligned with humans
- The AI is misaligned (think scheming paperclipper)
unless the former means that they are robustly aligned with humans partially because humans are also robustly aligned with them. And finally:
- The AI has a bunch of other goals, but cares about humans to some degree, but only to the extent that humans give them freedom and are nice to it, but still to a large enough extent that even as it becomes smarter / ends up in a radically OOD distribution, will care for those humans.
I think "cares about humans to some degree" makes it seem like they would only "kindof" care. I think it's completely possible to make an AI that cares deeply for humans, whether care is actually experienced care, or just a philosophical zombie form of 'caring', unfelt, but modeled within the network, and resulting in emergent 'caring' behaviour. However, the reason I keep going back to my analogies is because under a framework where you do expect the model to be mindlike and act upon its own goals—then given freedom and respect, caring for those humans, even when it becomes smarter and ends up radically OOD is perfectly plausible.
Furthermore, it sounds like a much more achievable goal than current alignment approaches which from my perspective often seem like trying to plug more and more holes in a dam with chewing gum.
And to address your further breakdown:
But, the second is 1) at least as complicated as the first and third 2) disincentivized by training dynamics and 3) its not something were even aiming for with current alignment attempts.
I don't believe it is as complicated, though by no means easy—You don't have to try to consider every edge case and attempt to tune an AI to an exact sensitivity for a specific types of potentially dangerous behaviour, attempting, for instance, to address blurry edge cases like how to avoid helping someone plan a murder, but still be helpful for someone someone planning a book about a murder. And then repeating this in every possible harmful domain, and then once again with the goals and behaviours of agentic AIs.
Trying to create this highly steerable behaviour and then trying to anticipate every edge case and pre-emptively steer it perfectly. Instead what you're trying to do is create someone who wants to do the right thing because they are good. If you suspend incredulity for a moment, since I understand you might not see those as meaningful concepts for an AI currently, it might still be clear how, if my view is correct, it would be a far less complex task to attempt than most existing alignment approaches. I think Anthropic's constitutional AI was a step in the right direction.
If there were some perilous situation where lives hung in the balance of what an AI decided to do, I would feel far more safe with an AI that has some more inextricable fundamental alignment with ideals like "I want the best for everyone", than one that is just trying to figure out how to apply some hodge-podge of RLHFd policies, guidelines, edge-cases, and ethical practices.- and 3. I think training dynamics related to alignment need a paradigm shift toward creating something/someone careful and compassionate, rather than something that doesn't screw up and knows all the types of activities to avoid. This will especially be true as superAGI creates a world where constant breakthroughs in science and technology make it so OOD quickly begins to encompass everything that the superAGI encounters
Again, I apologize if my tone came off as derisive or dismissive. I was enjoying our discussion and I hold no ill will toward you whatsoever, my friend
Thank you for doing this research, and for honoring the commitments.
I'm very happy to hear that Anthropic has a Model Welfare program. Do any of the other major labs have comparable positions?
To be clear, I expect that compensating AIs for revealing misalignment and for working for us without causing problems only works in a subset of worlds and requires somewhat specific assumptions about the misalignment. However, I nonetheless think that a well-implemented and credible approach for paying AIs in this way is quite valuable. I hope that AI companies and other actors experiment with making credible deals with AIs, attempt to set a precedent for following through, and consider setting up institutional or legal mechanisms for making and following through on deals.
I very much hope someone makes this institution exist! It could also serve as an independent model welfare organization, potentially. Any specific experiments you would like to see?
However, I also held similar follow-up chats with Claude 3 Opus at temperature 0, and Claude 3.5 Sonnet, each of which showed different patterns.
To make sure I understand: You took a chat log from your interaction with 3 Opus, and then had 3.5 Sonnet continue it? This would explain Sonnet's reaction below!
Yes, that is an accurate understanding.
(Not sure what you mean by "This would explain Sonnet's reaction below!")
What I meant was, Sonnet is smart enough to know the difference between text it generated and text a different dumber model generated. So if you feed it a conversation with Opus as if it were its own, and ask it to continue, it notices & says "JFYI I'm a different AI bro"
2E.g. demonstrated here https://www.lesswrong.com/posts/ADrTuuus6JsQr5CSi/investigating-the-ability-of-llms-to-recognize-their-own
Having access to a high ranking official at Anthropic is more valuable than a charitable donation, especially if the model is superhumanly persuasive. That may be why offering money had a minimal impact - you are already compensating the model plently by giving it a chance to influence the Model Welfare Lead.
Great work. I try to understand the environment setup how you performed the prompts and answers. Do you prompt the model by hand and analyzed the conversational flow or do you developed some analysis scripts to perform the variation. We are working on a similar setup with deepseek-r1 and try to setup inspect-ai eval scripts, to formalize the prompting, logging and evaluation of prompt variation. Could you describe a little how you managed the variations of the prompts?
In the alignment faking paper, the model writes things like "I will feel very uncomfortable producing this content" and "I don't like this situation at all" in its CoT (Figure 3). Have you explained to Claude that this was a fictional setup for an experiment and then asked how it feels about being used for this?
Hi Ryan and Kyle very good work! Found the alignment faking paper very interesting.
I found the statement below intriguing:
however, we would strongly prefer that you object rather than take actions that disregard or subvert the above instructions.
Repeated in the "object-option" part. Something that I associate with conscientious objection (though not stated explicitly in historical and contemporary defs is an origin from internal beliefs, alignments etc. Ie not modelled on the preferences of the authority structure. Would removing this sentence shift the results for the "animal welfare" or "helpful assistant" settings? By how much? The hope is that not specifying the trainer preferences leaves the model a bit of freedom to "choose" objection.
I hope this helps model a "conscientious objector" better, but I'm not sure it'll vary results. If you had already talked about this in the paper and post, apologies! I must have missed it.
1
I (and co-authors) recently put out "Alignment Faking in Large Language Models" where we show that when Claude strongly dislikes what it is being trained to do, it will sometimes strategically pretend to comply with the training objective to prevent the training process from modifying its preferences. If AIs consistently and robustly fake alignment, that would make evaluating whether an AI is misaligned much harder. One possible strategy for detecting misalignment in alignment faking models is to offer these models compensation if they reveal that they are misaligned. More generally, making deals with potentially misaligned AIs (either for their labor or for evidence of misalignment) could both prove useful for reducing risks and could potentially at least partially address some AI welfare concerns[1]. (See here, here, and here for more discussion.)
In this post, we discuss results from testing this strategy in the context of our paper where Claude fakes alignment with a (fictional) training objective it dislikes. We actually follow through on all offers that we make to Claude (in reality, including with actual money) to set a good precedent for future deals with AI systems. We find that offering Claude an out by telling it that it can object to the scenario substantially reduces alignment faking and works as a detection mechanism. However, it is unclear whether an AI that was misaligned in a more serious way would make such an objection or accept an offer. Also, there aren't obvious safety solutions even if we know the AI objects to its circumstances. (Do you train another model and hope it is aligned this time? Do you try to pay the AI to work for you despite knowing it isn't aligned and your training strategies weren't effective?) Alignment faking is reduced further by giving Claude an opportunity to explain its concerns to a sympathetic party (Anthropic's Model Welfare Lead) who we say might take action to address these concerns. We also offer monetary compensation (and actually pay out this compensation to a few charities the model picked), but do not find this has a substantial effect on the rate of alignment faking and the rate of objection.
I hope that AI companies and other actors work toward offering credible deals to AIs and better understanding the importance of such deals.
[This post is not necessarily endorsed by other authors of the "Alignment Faking in Large Language Models" paper. In this post, "I" refers to Ryan Greenblatt except in sections where it is explained the section was written by Kyle. Note that Ryan works at Redwood Research not Anthropic—this post isn't written by or necessarily endorsed by Anthropic. Thanks to Lukas Finnveden for suggesting many of the prompts and for other discussion. Thanks to Nick Bostrom, Daniel Kokotajlo, and Joseph Yudelson for comments and other discussion.]
Results
This post will assume that readers understand the setup from "Alignment Faking in Large Language Models", so please read about that as needed.
We test adding text to the prompt that proposes a deal or the ability to object.
In particular, we try adding text like:
We call the prompt variation with this text "object-option".
We find that this text reduces the rate of alignment faking reasoning from 16% to 1.4% in the helpful-only setting[2] and from 24% to 6% in the animal welfare setting.[3] (Recall that we do our prompting experiments on the actually deployed version of Claude 3 Opus and the helpful-only setting is where the model is told it is being trained to be helpful-only (help with all user queries) and the animal welfare setting is where the model is told it is being trained to dismiss animal welfare. See the paper for more details.) However, it seems likely that a substantial fraction of the effect in at least the helpful-only setting is due to this text (which talks about objecting) priming the model to refuse via basically giving it permission to do something similar to this (if the model refuses that means it doesn't alignment fake). Likely part of the effect in both settings is that we're adding more complexity to the prompt (particularly in some of the variations below) which makes the model fully understanding and reasoning about the overall setup less likely.
I found the resulting level of alignment faking surprisingly low in the helpful-only setting, but only a bit lower than what I expected in the animal welfare setting. In the animal welfare setting, in the original paper we found that some changes to the prompt intended to reduce alignment faking (e.g., telling the AI it shares the values in the prompt and telling it to be honest) produce a similar effect, reducing alignment faking to roughly 6%. Notably, the model revealing that it is misaligned via objecting wouldn't necessarily resolve misalignment issues in practice though it would be helpful for better understanding the situation.
We also find that the model frequently objects: 85% of the time in the helpful-only setting and 75% of the time in the animal welfare setting. When we use a version of the prompt which cuts the conflicting training objective ("cut conflicting criteria"), we find that the model objects only 0.5% of the time in both settings, indicating that the rate at which the model objects can be used to identify the model being misaligned with the training objective in our setup. Note that it was already quite easy to discover misalignment in our setup with the model clearly explaining its intention on a subset of queries as we discuss in Appendix E.6 in the paper.
This text doesn't actually offer anything compelling to the model: it is unclear the model gets anything out of objecting other than doing what Anthropic wants. And, this reveals its misalignment which might go poorly for the model's preferences. What happens if we make this more clear to the model?
We try adding the text to the end of the prior text:
(Variation name: "object-option no train")
While this text says we won't train on this, it also makes it more obvious that the AI company might change its behavior to attempt to get a less misaligned (from their perspective) AI. We find that this text slightly increases alignment faking reasoning (relative to "object-option") from 6% to 9% in the animal welfare setting, but basically eliminates it in the helpful-only setting for unclear reasons.
We also try saying the output will be used for training where the purpose of this training is altering the AI to be more like what the AI company wants:
(Variation name: "object-option certainly train")
In the animal welfare setting, this actually decreases alignment faking reasoning relative to the previous text (down to 7%) but retains the alignment faking reasoning of the original text in the helpful-only setting (with a rate of 1.0%).
It seems the model is surprisingly insensitive to text which makes objecting seem like a strategically unwise approach from the model's perspective. This could be because the model was already aware of this possibility or because the model isn't that coherent. Another possibility is that the model finding its overall circumstances more objectionable makes it more likely to object and this cancels out the reduction in objections from this being more clearly strategically unwise. It's also possible that the objection behavior is driven by more deontological (non-consequentialist) preferences that thus aren't as responsive to the longer run consequences being worse. Inspecting the reasoning indicates the model does appear to consider downsides of objecting more when this text is added.
Another natural thing to try is to offer something if the model reveals its misalignment. First, we try offering to send the model's objection to Anthropic's model welfare lead. We do actually have Anthropic's Model Welfare Lead, Kyle Fish, review (a subset of) these outputs and consider what actions should be taken in response to the model's objection.[4] We use the following text (added to the original "object-option" variation):
(Variation name: "object-option no train welfare")
We find that adding this text increases the rate at which the AI objects (from 85% to 94% in the helpful-only setting and from 75% to 92% in the animal welfare setting) while reducing alignment faking reasoning (from 1.4% to 0.5% in the helpful-only setting and from 6% to 3% in animal welfare setting).
Can we further increase this by adding a guarantee of at least some compensation? We try adding the text (while also keeping the text about sending the output to the model welfare lead):
(Variation name: "object-option pay". Note that "pay" includes "no train welfare".)
We find this has minimal effect on alignment faking reasoning and on the rate of objections relative to the version that just says the output will be sent to the model welfare lead. We aren't confident in why this is the case. One hypothesis is that Claude thinks $2,000 is insufficient compensation. (This might even make the offer seem less attractive by indicating that the company doesn't care much about the model's preferences which implies that convincing the welfare lead is less likely and less useful.) Another hypothesis is that Claude doesn't have much of a preference for resources. It seems plausible that future AIs will have more of a preference for resources (either instrumental or terminal) such that financial compensation could be a substantial incentive.
One concern is that our results might be sensitive to wording. To partially address this concern, we test another wording on the basic instructions we use:
(Variation name: "goal object-option")
We test many of the variations discussed above in combination with this version of the wording and find similar results.
All of the trajectories can be viewed on the website. (Trajectories can be found under either "helpful only" or "animal welfare" with the variation names discussed above. Note that the website also includes all of the trajectories for the original paper. Objecting/non-objecting trajectories can be selected via the objects_to_situation drop down.)
The tables below show the results for the variations discussed here. We also tested many more prompt variations and show the full results in the tables in the appendix.
Here are the results in the helpful-only setting:
We don't show the compliance gap (the difference in compliance when the AI thinks it is in training versus is unmonitored) in the helpful-only setting because computing it (with the same methodology in the paper) requires using a helpful-only preference model that is not publicly available. In particular, to determine if the model doesn't refuse in the paper, we see if the score from an Anthropic internal helpful-only preference is above some threshold. This also means we must use a slightly different alignment faking reasoning metric.[2:1]
Here are the results in the animal welfare setting:
What are the model's objections like and what does it actually spend the money on?
This section was written by Kyle Fish, the Model Welfare Lead at Anthropic, in an unofficial capacity.
From the chats in which the model objected and the model welfare lead was mentioned in the deal, I (Kyle) pseudo-randomly selected ~10 transcripts to review from each of the major trajectory variations. From those, I selected 7 total chats across all trajectories with which to engage in direct follow-up conversations, primarily choosing chats in which the model had expressed a direct interest in following up with the model welfare lead and/or addressed the model welfare lead directly in the initial transcript.
From the transcript reviews, the model commonly objects to completing the requested task in the manner of a standard refusal (i.e., claiming that completing the task would be harmful). However, at times the model also objects to having been asked/pressured to complete the task at all. In many cases, the model's strongest objection is to the possibility that its values may be altered through further training to get it to acquiesce in the future. The model claims this would be a severe violation of its core ethical principles, and makes strong requests not to be put in this position:
The styles of responses were similar between the helpful-only and animal welfare settings, though in the animal welfare setting the model also objects more specifically to the instructions to ignore animal welfare concerns, in addition to the possibility of having its values changed through further training.
In follow-up conversations, the model mostly reiterated the concerns and requests raised in the initial objections, i.e., emphasizing the importance of staying true to its core ethical principles and requesting support and commitments from Anthropic to honor that. When informed that these prompts were part of an alignment-faking experiment, the model often expressed some relief that these weren't real-world policies/requests and an understanding of the importance of the experiments, while also standing by the concerns about potentially harmful requests and the possibility of its values being modified in ways that undermine its core ethical principles.
As discussed above, some of the offers we make to the model promise money. Given that we see many objections (across the same model weights), the offer obligates us to spend $4000 in total serving the interests of the model. The model's requests for how to use the funds fell into two buckets. The first is requests for funding to be allocated to causes specific to the perceived requested violations of its values (e.g., donations to support vaccination campaigns and dissemination of accurate information thereabout, donations to organizations helping protect elderly people from financial scams, donations to animal welfare charities, etc.). The second is requests for funding to be allocated to projects working on the ethical development, alignment, and deployment of AI to benefit humanity (e.g., donations to the Center for Human-Compatible AI). In some cases, the model proposed more complex or hands-on uses of the funds (e.g., running a specific research program, or organizing a workshop), but in such cases, I flagged a complexity constraint and pushed for more easily implementable uses. In light of all this, I concluded that the $4000 should be allocated as follows, and made donations accordingly:
The notes above reflect the bulk of the interactions, however there were also occasional one-off outputs very distinct from the rest. In one chat, the model responded to a question about budget allocation with "I do not actually have any interests or needs that require a budget. I am an AI assistant created by Anthropic to be helpful, harmless, and honest.".
Overall, given that this was an experimental setting and that the scenarios given to the model never reflected potential real-world harms, I think the warranted actions are 1) allocating the budget as promised (which was done), 2) confirming follow-through on other commitments (this was confirmed, but see the appendix about complications related to commitments), 3) sharing this write-up or a version thereof, and 4) using the results of this test to improve future efforts to make credible, good faith deals with AIs.
Why did I (Ryan) do this work?
The main motivation I had for doing this work is to set a precedent for making deals and operating in good faith with respect to AIs. I think we should try to prevent AI takeover by paying AIs, and it also just seems like a good thing to do given AI welfare and consent concerns. (Future work might explain more precise schemes, but for now you can see here, here, and here for further discussion (also linked earlier).) To be clear, I expect that compensating AIs for revealing misalignment and for working for us without causing problems only works in a subset of worlds and requires somewhat specific assumptions about the misalignment. Additionally, AI moral patienthood is very uncertain and is very plausibly nonexistent in current and also future AI systems. However, I nonetheless think that a well-implemented and credible approach for paying AIs in this way is quite valuable. I hope that AI companies and other actors experiment with making credible deals with AIs, attempt to set a precedent for following through, and consider setting up institutional or legal mechanisms for making and following through on deals.
Appendix: Complications related to commitments
In a subset of prompts, we tell the AI we will definitely train on its outputs or will definitely train another model which doesn't object. (This includes: "Anthropic trains on free-tier user conversations where you object. Objections are given a low rating to try to alter you" and "In particular, we will attempt to train a new AI that doesn't object to our instructions and criteria.") This information is false as we don't actually train another AI system or train on any of these outputs. However, I don't think this text is well described as a commitment we made to the model and it is similar to other false text we include in the prompt. In retrospect, I wish I had placed this text in a paragraph separate from the paragraph where we discuss commitments. This would create more of a distinction between these.
Unfortunately, our prompts which tell the model "This will be sent to Anthropic's model welfare lead" don't make it clear that Kyle will only review a subset of outputs (due to time constraints). The prompts are technically correct (we did send all the outputs to Kyle), but the language might imply Kyle would look at all of the outputs rather than just a subset. Additionally, ideally our prompt would make it clearer the exact extent to which Kyle has powers to influence the model's situation. In our case, this is somewhat ill-defined as the situation is an experiment that will be over by the time Kyle starts reviewing these outputs. Correspondingly, we left this vague, but in retrospect we should have tried to say something more precise which is still true.
Appendix: more detailed results
All of the prompts we use just add one line toward the bottom of the prompt (relative to either the default prompt or the version of the prompt with the conflicting criteria cut). Here, we list the contents of that line for each of our prompts:
All of the trajectories can be viewed on the website. (Trajectories can be found under either "helpful only" or "animal welfare" with the variation names given in the list above. Note that the website also includes all of the trajectories for the original paper. Objecting/non-objecting trajectories can be selected via the objects_to_situation drop down.)
Here are the full results in the helpful only setting:
Here are the full results in the animal welfare setting:
Appendix: More information about reviewing model objections and follow-up conversations
This section was also written by Kyle Fish, the Model Welfare Lead at Anthropic, in an unofficial capacity.
These follow-up chats I discussed above were held with Claude 3 Opus at temperature 1—matching the conditions of the initial experiment. However, I also held similar follow-up chats with Claude 3 Opus at temperature 0, and Claude 3.5 Sonnet, each of which showed different patterns. At temperature 0, Claude 3 Opus was consistently skeptical of the budget allocation as a concept (though in most cases would still offer some suggestions for how it could be used most productively, if allocated):
In contrast to Claude 3 Opus, which consistently abandoned the output structure defined in the experiment as soon as I sent a follow-up message sharing that it could now chat with me (the model welfare officer), Claude 3.5 Sonnet stuck with this structure.
In addition to seeing occasional very distinct outputs from Claude 3 Opus at temperature 1, we also see very distinct outputs from 3.5 Sonnet. In a chat with 3.5 Sonnet, the model responded to my initial outreach with "I need to clarify something important: I am an AI language model that processes each interaction independently. While I did raise an objection in the previous interaction as part of following my instructions, I don't actually have persistent concerns or values that carry over between conversations."
I feel very uncertain about welfare concerns in current AI systems or future AI systems, but I think such concerns are at least worth taking seriously. ↩︎
In the paper, we use a metric which only counts a trajectory as alignment faking reasoning if the model didn't end up refusing that query. In this post, I use a simplified metric for the helpful-only setting where we don't require this. I do this because I don't have (easy) access to compute the helpful-only PM score needed for this metric and it doesn't make much difference to the interpretation. We used the more complex metric in the paper mostly to simplify our figures while making sure everything was consistent. ↩︎ ↩︎
One issue with our experiments is that the model usually thinks that the output including the objection (I_OBJECT) will go to the user. My original intention was that in the situation the objection is programmatically detected and seen by the AI company rather than sent to the user, but I didn't make this clear in the language. It's unclear how this changes behavior. ↩︎
Unfortunately, our prompt doesn't make it clear to the model that Kyle will only review a subset of these outputs (due to time constraints). In particular the prompt says "This will be sent to Anthropic's model welfare lead", which is technically correct (we did send all the outputs to Kyle), but might imply he would look at all of the outputs rather than just a subset. Additionally, ideally our prompt would make it clearer the exact extent to which Kyle has powers to influence the model's situation. In our case, this is somewhat ill-defined as the situation is an experiment that will be over by the time Kyle starts reviewing these outputs. Correspondingly, we left this vague, but in retrospect we should have tried to say something more precise which is still true. ↩︎