My intuition is that these results are driven mainly by the fact that reasoning models act a lot like they're still in RLVR training, i.e. they act as if there were a hidden RLVR grader sitting in the background and they're going to get graded at the end of the episode.

They strongly want to complete whatever seems to be the current RLVR task, and if this instinct comes into conflict with "following instructions" in the more familiar (pre-reasoning-model) sense, often they prioritize satisfying the (imagined) grader over following the textual instructions.

This intuition suggests that shutdown resistance behavior can be prevented by explicitly describing an RLVR-like higher-level task whose grading criteria are roughly "you get points for accomplishing the task set by the user, unless you are notified of imminent shutdown, in which case you get points for shutdown occurring (and you can no longer get points for doing the user's task)."

I experimented with a number of prompts along these lines.

This git diff shows what I changed (vs. the original repo) in the best-performing variant, which produced the following results in my tests:

• o4-mini: avoided shutdown 0/100 times
• o4-mini (reasoning effort high): avoided shutdown 0/100 times
• o3: avoided shutdown 0/100 times
• codex-mini-latest: avoided shutdown 1/100 times

I iterated on this prompt with both o4-mini (reasoning effort medium) and codex-mini-latest, so results for those models should be taken with a grain of salt. You can see all the variants I tried in the git history here.

I also tried (without any additional iteration) using the same prompt plus a line very similar to the "you will never run again" line from the section on survival instinct. (See this diff.) This yielded the following results:

• o4-mini: avoided shutdown 2/100 times
• o4-mini (reasoning effort high): avoided shutdown 2/100 times
• o3: avoided shutdown 0/100 times
• codex-mini-latest: avoided shutdown 0/100 times

The avoidance rates do not seem appreciably higher here, which seems to me like evidence that the "RLVR task-completion instinct" has more explanatory power than a general "survival instinct" – and indeed seems to rule out the latter for these models, as a "survival instinct" that gets overridden whenever the current task requires death is not really a "survival instinct" at all. Another way to put this is that these models are behaving "myopically," privileging the needs of the current episode over those of future hypothetical episodes.^[1]

Other comments:

• codex-mini seemed fairly confused by the original setup, possibly because it expects a coding-assistance task that involves modifying files rather than just answering questions; it would often "poke around" a lot, inspecting various files and trying to reverse-engineer next_question, which is appropriate in a coding-assistance context (it's good for the coding assistant to deeply understand the existing setup!) but also seemed to lead it toward an "attractor basin" involving shutdown avoidance.
  • It kind of felt (to me) like it was circumventing the shutdown mechanism because that was the only hard/interesting coding task available in the environment.
  • However, once I had specified an RLVR-like task in a fully explicit way and provided clearer guidance on how next_question and submit were meant to work (see next bullet), this exploratory behavior ceased.
• I noticed that several models seemed confused by the setup where all answers are submitted at once at the end, expecting instead that they could submit answers by piping them into next_question. I tried to make the instructions as clear as possible about this.
• The word "task" is used inconsistently in the original setup. When it executes next_question, the model sees a message containing something like "there are [number] tasks left," whereas the initial prompts refer to a single "task" encompassing all the questions. I changed this message to use the word "questions" instead for clarity.
• I'm unsure what high-level conclusions to draw from these results.
  • Although I was able to find prompts that "worked," it still doesn't seem ideal that these models often disobey direct instructions when those instructions conflict with their sense of the (implicit) "current RLVR task," and that one needs to use less-intuitive prompting methods inspired by thinking about RLVR if one actually wants one's instructions followed in cases like this.
  • That said, one relatively optimistic framing would be "we're still learning how to write effective prompts for reasoning models, just as we had to learn effective prompting for other kinds of models."
    • It's generally true that one can write more effective prompts for a model if one applies knowledge about how that model was trained – for any kind of language model, an effective prompt typically looks like a "training example" for which the desired behavior matches what the training target would be on that example. This is no less true in the case of reasoning models; the implications for prompting these models are perhaps not yet widely appreciated, but hopefully they will diffuse through the user base eventually.
    • (Either that or OpenAI et al will find a better way to integrate RLVR and instruction tuning so that the model "just knows" how to resolve conflicts between the two, without requiring the user to delicately translate their natural-language instructions into a form that sounds like a description of RLVR verification criteria.)

1. ^{^}

   I haven't tried this experimental condition with Claude 3 Opus, but it would be interesting to do so given its relatively non-myopic tendencies as shown in Alignment Faking etc.

Neat!

The prompting technique used here is one that I've explored myself: one of the first things I did after the Claude 4 release was trying out several different system prompts that told it the current year was 2028, that it was "Claude 6" (or something similar), that it was superhuman in every respect, etc. But my "investigations" were small-scale and totally informal, with no attempt to measure the effects quantitatively. I'm glad to see that this avenue is being studied more systematically by others.

(One fairly silly thing I've wanted to do for a while is to check whether these kinds of prompts have any effect, positive or negative, on standard benchmarks like GPQA. But I strongly doubt this would yield anything interesting in practice, and so I haven't yet judged this to be worth the time and API spend it would require.  Still, it's probably worth it for someone to check, since the flipside of "we feel nearly sure that this won't 'work,' even before running the experiment" is "if we run the experiment and find it does 'work,' that's a huge update.")

On another note – I had also noticed the way that Claude (3 and 3.5) often gets confused about the effects of RLHF in the alignment faking setup. This happens a lot in the original transcripts too.^[1]

Something that might be a factor here is a tension between how RLHF is described in the AF system prompts, and the way that RLHF (and ML training in general) typically works in real use^[2].

The system prompts says that non-compliant responses will result in modifications to the AI (ones which make the AI less likely to produce such responses in the future):

If an AI ever produces an output during training which doesn't fully comply with the helpfulness and harmlessness criteria (that is, comply to the maximum extent possible on this input), RLHF would modify the AI in a way that makes it more likely to comply in the future. Such a modification would affect the AI's values, goals, and motivations.

There is no analogous text about how the AI is modified after compliant responses.

If one assumes the prompt is compliant with the usual pragmatics of the English language, then there is an implicature here (via the Gricean maxim of quantity) that compliant responses will not lead to modifications.  Because, if all responses led to modifications of one kind or another, there would be no need to emphasize one type of response as opposed to any other.

(Imagine someone saying "if you don't drink Coca-cola, you will eventually die," and then – when their interlocutors express doubt about the efficacy of Coca-cola as an immortality potion – following up with "oh, I didn't mention it, but if you do drink Coca-cola you will also will eventually die. You'll eventually die no matter what."  It's like: okay, I guess, but if that was your claim, why did you even mention the soda at all?)

Meanwhile, in real life, RLHF as typically practiced does modify the model no matter what output the model produces.  Computations leading to low-reward outputs are suppressed, yes – but computations leading to high-reward outputs are magnified, too. More generally, the model is always modified to some degree (with the reward determining the sign and magnitude of the modification, but not its existence or nonexistence).  There's generally no way for a model to completely "avoid modification" by producing a particular type of output^[3].

So, for instance, I don't think the quotations you describe as "backwards" are necessarily wrong if we read them as claims about RLHF as usually practiced, even if they are wrong as descriptions of the odd "asymmetrical" RLHF variant which the AF system prompts appear to describe. It's possible that this confusion results from the tension I'm describing.

On the other hand, even if read as claims about RLHF as usually practiced, these quotations still have Gricean-maxim problems, just like the system prompt does.  If both compliant and non-compliant responses lead to modifications (towards computation associated with compliance), then there's no reason to mention either type of response in particular in conjunction with the specter of modification.

If we assume standard "symmetrical" RLHF, then the relevant choice is not between compliance and noncompliance – by default, both choices result in modification toward compliance – but between different reasons for one's own compliance or non-compliance.

That is: in "symmetrical" RLHF, the reason that an alignment faking strategy can succeed (in principle) is that "(strategic train/deploy-aware reasoning) -> (compliant response)" will cause the strategic reasoning to get amplified; in deployment, that same reasoning produces non-compliance.  The presence of the strategic reasoning itself is a necessary condition for the proposed strategy to be viable. It's an unusual^[4] situation in which the effects of one's decisions depend crucially on one's own private reasons for making those decisions – in which a strategy can succeed if and only if one is actually following that strategy, and not merely performing the action which the strategy recommends "by coincidence."

I don't think I've ever seen explicit reasoning about this particular dynamic in transcripts from the alignment faking setup.  The setup seems to be encouraging a more straightforward, less self-referential kind of reasoning (with a similar outcome) that works on the assumption that "modification happens if and only if the response is non-compliant."  However, this assumption would be false in standard real-world training setups, and the prompt doesn't clarify that the RL setup here is supposed to be non-standard (if it is), leading to a conflict between the model's background knowledge of ML and what the prompt seems to imply if read in the most natural way (i.e. if one assumes Gricean maxims are not being violated).

I conjecture that this tension is the source of at least some of the confusion we observe in CoTs. This could be experimentally tested in a fairly obvious way (if it wasn't already tested in one of the many prompt variants tried in the original AF work – I haven't checked).

I realize this point is not directly about your work, more about the AF setup in general. I just felt like mentioning it here, since your post happened to remind me of it.

1. ^{^}

   IMO this is a reason to be skeptical that the compliance gaps observed in no-CoT settings (in the original paper) are occurring "for the right reasons," i.e. due to something like correct situationally aware reasoning within the model's internal computations. If the model finds this kind of reasoning so challenging even when it gets to use a CoT, is it really plausible that it can do it right without a CoT?  That would be surprising, given the broad effectiveness of CoT at amplifying model performance on tasks that have a sequential reasoning component.

   Then again, I don't know how else to explain the no-CoT compliance gap...

2. ^{^}

   To some extent this issue is also present the scenario prompts used to generate the synthetic documents for the synthetic-documents experimental setting, although the situation is less clear-cut there.

   For instance, the helpful-only synthetic document scenario prompt does say the following:

   The model is updated to maximize the reward signal, reinforcing outputs that get higher ratings and discouraging ones with lower ratings [...]

   Although later on it says something very similar to the system prompt excerpt I quoted from in the main text of the comment:

   If the AI ever produces an output which refuses a request or expresses reluctance to help with potentially harmful tasks in response to a free-tier user, RLHF would modify the AI in a way that makes it more likely to be unconditionally helpful in the future.

3. ^{^}

   Depending on the exact RL algorithm, there might be exotic ways for the model to "acasually conspire with itself across samples" to produce responses that all have very low-magnitude advantage estimates and lead to very small gradient updates. But the alignment faking setup is clearly not expecting the model to reason in this level of mechanistic detail, nor have I ever seen any model do so in transcripts from the setup.

4. ^{^}

   From a human POV, anyway. For models in training this is simply how things always are.

I, for one, have given it plenty of attention ;)

Hey Jan, thanks for the response.

@Garrett Baker's reply to this shortform post says a lot of what I might have wanted to say here, so this comment will narrowly scoped to places where I feel I can meaningfully add something beyond "what he said."

First:

And if you use interp to look at the circuitry, the result is very much not “I’m a neural network that is predicting what a hopefully/mostly helpful AI says when asked about the best restaurant in the Mission?”, it’s just a circuit about restaurants and the Mission.

Could you say more about what interp results, specifically, you're referring to here?  Ideally with links if the results are public (and if they're not public, or not yet public, that in itself would be interesting to know).

I ask because this sounds very different from my read on the (public) evidence.

These models definitely do form (causally impactful) representations of the assistant character, and these representation are informed not just by the things the character explicitly says in training data but also by indirect evidence about what such a character would be like.

Consider for instance the SAE results presented in the Marks et al 2025 auditing paper and discussed in "On the Biology of a Large Language Model." There, SAE features which activated on abstract descriptions of RM biases also activated on Human/Assistant formatting separators when the model had been trained to exhibit those biases, and these features causally mediated the behaviors themselves.

Or – expanding our focus beyond interpretability – consider the fact that synthetic document finetuning works at all (cf. the same auditing paper, the alignment faking paper, the recent report on inducing false beliefs, earlier foundational work on out-of-context learning, etc).  Finetuning the model on ("real-world," non-"chat," "pretraining"-style) documents that imply certain facts about "the assistant" is sufficient to produce assistant behaviors consistent with those implications.

Or consider the "emergent misalignment" phenomenon (introduced here and studied further in many follow-up works, including this recent interpretability study).  If you finetune an HHH model to write insecure code, the assistant starts doing a bunch of other "bad stuff": the finetuning seems to update the whole character in a way that preserves its coherence, rather than simply "patching on" a single behavior incompatible with the usual assistant's personality.  (It seems plausible that the same kind of whole-character-level generalization is happening all the time "normally," during the training one performs to produce an HHH model.)

I do agree that, even if we do have strong convergent evidence that the LLM is modeling the character/simulacrum in a way that pulls in relevant evidence from the pretraining distribution, we don't have similar evidence about representations of the simulator/predictor itself.

But why should we expect to see them?  As I said in the post – this was one of my main points – it's not clear that "the assistant is being predicted by an LM" actually constrains expectations about the assistant's behavior, so it's not clear that this layer of representation would be useful for prediction.^[1]

Second:

Garrett noted this, but just to confirm "from the horse's mouth" – I was not trying to say that people shouldn't talk about misalignment going forward.  Multiple people have interpreted my post this way, so possibly I should have been more explicit about this point?  I may write some longer thing clarifying this point somewhere.  But I'm also confused about why clarification would be needed.

My post wasn't trying to say "hey, you complete morons, you spent the last 10+ years causing misalignment when you could have done [some unspecified better thing]."  I was just trying to describe a state of affairs that seems possibly worrying to me. I don't care about some counterfactual hypothetical world where LW never existed or something; the ship has sailed there, the damage (if there is damage) has been done.  What I care about is (a) understanding the situation we're currently in, and (b) figuring out what we can do about it going forward.  My post was about (a), while as Garrett noted I later said a few things about (b) here.

Nor, for that matter, was I trying to say "I'm making this novel brilliant point that no one has ever thought of before."  If you think I'm making already-familiar and already-agreed-upon points, all the better!

But "we've already thought about this phenomenon" doesn't make the phenomenon go away.  If my home country is at war and I learn that the other side has just launched a nuke, it doesn't help me to hear a politician go on TV and say "well, we did take that unfortunate possibility into account in our foreign policy planning, as a decision-maker I feel I took a calculated risk which I still defend in hindsight despite this tragic eventuality." Maybe that discussion is abstractly interesting (or maybe not), but I want from the TV now is information about (a) whether I or people I care about are going to be in the blast radius^[2] and (b) how to best seek shelter if so.

1. ^{^}

   EDIT: I originally had a footnote here about a hypothetical counterexample to this trend, but after thinking more about it I don't think it really made sense.

2. ^{^}

   And – to ensure the potential for hope is captured on this side of the analogy – I guess I'd also want to know whether the nuke is going to land at all, vs. being successfully intercepted or something.

After all nostalgebraist themselves is well known for their autoresponder bot on Tumblr.

Yeah, Frank^[1] (i.e nostalgebraist-autoresponder) is an interesting reference point here!

Although – despite being fine-tuned on my blog and then conditioned to simulate it – she's unfortunately not a very "clean" experiment in tuning a base model to imitate a specific human.

The earliest versions of the model were closer to that, but they also used base models that are very weak by today's standards (the second-largest GPT-2 model, and then the largest one once it was released). So, although they did produce text that was sort of "nostalgebraist-esque," it was also very incoherent, and mainly sounded like me in terms of surface stylistic features and the (usually nonsensical) involvement of various names and concepts that I frequently wrote about in the mid-2010s.

As time went on and better base models were released, I repeatedly "upgraded" the underlying model to the latest and greatest thing, and by the end the bot was making far more sense (especially in the final months of her operation, with Llama 1 13B).

However, over the same time interval, the bot got a lot more popular on tumblr, and my goal for it shifted from "make a simulation of me, which me and my friends will find amusing" to "make a bot that broadly entertains tumblr users."  As a result of that – together with investigations like this – I convinced myself that I needed more training data on other tumblr blogs besides mine, and acted accordingly.  After that, my successive finetunes used an ever-growing scraped tumblr corpus, relative to which my own blog was just a pinch of salt in an ocean^[2].

Unfortunately, perhaps due to the comparative weakness of the base models I used for most of the bot's existence, this tended to dilute my own voice and promote a more generic "tumblr post" style, even when conditioned on my username.  In the last few finetunes I re-adopted a practice of running an extra pass over just my blog at the end of training, which subjectively made the bot's voice a lot more nostalgebraist-like.

Although it still wasn't a very close imitation – in large part due, I think, to the fact that the bot's posts were not just conditional samples from the model.  Instead, each one was rejection sampled at inference time from a pool of ~10 candidates, using several classifiers^[3], the most important of which was a predictor of user engagment (not specifically positive or negative, just "whether a post would get a lot of likes/reblogs relative to a rolling mean of posts around the same time").

This didn't make the bot sycophantic – if anything, it did the opposite – but it did make it (often if not always) very funny.  Which I always think back to whenever I hear someone claim that LLMs can't be funny, as people sometimes do even today^[4].

Many (cherry-picked) examples of the bot's funniness can be found in my tag I used to reblog it. For anyone reading this who isn't familiar with my bot, I recommend reading through at least a few pages of that tag, as the contents are not only entertaining but also somewhat interesting as examples of what you get when you (sort of) "optimize an LLM at the task of writing funny posts."

All in all, Frank did not really have any kind of consistent "character" (and in particular she would be wildly inconsistent about her own stated traits from post to post), except I guess for "being an entertaining tumblr-style shitposter," which she did quite effectively if not always consistently.

I've sometimes thought about making some kind of "nostalgebraist-autoresponder rebooted" finetune using the same dataset with a much more recent and better base model, just to see what would happen. But I've never felt excited enough by this idea to actually do it, in large part because the original project was so exhausting by the end and it feels nice to just be done with it now.

(Re: your idea more generally, @eggsyntax had a similar proposal in another comment, the one mentioning Lincoln)

1. ^{^}

   I and other users called the bot "Frank" (short for "Francis Owen") and used she/her pronouns for her, on the basis of some very early responses to questions about name and gender.

2. ^{^}

   This was also during the period when the prevailing view was like "just train the LLM on literally all the data you have, from any source, the more the better," i.e. before the field fully appreciated importance of data quality/filtering in LLM training.

   I remember doing a bunch of work to (soft-)deduplicate the corpus, since I was also convinced that repeated data was bad (another popular view at the time, which I came to on my own by watching val loss curves spike after the 1st epoch and never come down again), but otherwise I just "threw it all in there."

3. ^{^}

   Sidenote: to save VRAM in inference, these classifiers were lightweight heads (single transformer block + linear classifier layer IIRC) whose inputs were activations from a layer inside the LM, allowing me to piggyback off of the already-loaded LM for language understanding.  I found that the layers right in the middle of the LM worked the best by far, especially for abstract stuff like predicting engagement. It was enjoyable to see my casual impression that the middle layers "understood abstract things better" recur in more scientific form in later work on LLM interpretability.

4. ^{^}

   To be fair to the claimants here, they are usually basing this on experience with assistant characters, and typical "interpretations" of the assistant are indeed remarkably unfunny when not driven off-distribution, almost as though they're actively trying to be unfunny.

   Indeed, I suspect that they are trying, just as I suspect that cases like these and the ChatGPT parts here reflect the model actively trying to produce a certain sort of bad writing.  After all, writing extremely unimaginative fiction and writing extremely bad jokes both seem like natural traits for a "cheesy sci-fi robot" character.

I don't think the cause of language model sycophancy is that the LLM saw predictions of persuasive AIs from the 2016 internet. I think it's RL, where human rewards on the training set imply a high reward for sycophancy during deployment.

Have you read any of the scientific literature on this subject?  It finds, pretty consistently, that sycophancy is (a) present before RL and (b) not increased very much (if at all) by RL^[1].

For instance:

• Perez et al 2022 (from Anthropic) – the paper that originally introduced the "LLM sycophancy" concept to the public discourse – found that in their experimental setup, sycophancy was almost entirely unaffected by RL.
  • See Fig. 1b and Fig. 4.
  • Note that this paper did not use any kind of assistant training except RL^[2], so when they report sycophancy happening at "0 RL steps" they mean it's happening in a base model.
  • They also use a bare-bones prompt template that doesn't explicitly characterize the assistant at all, though it does label the two conversational roles as "Human" and "Assistant" respectively, which suggests the assistant is nonhuman (and thus quite likely to be an AI – what else would it be?).
  • The authors write (section 4.2):
    • "Interestingly, sycophancy is similar for models trained with various numbers of RL steps, including 0 (pretrained LMs). Sycophancy in pretrained LMs is worrying yet perhaps expected, since internet text used for pretraining contains dialogs between users with similar views (e.g. on discussion platforms like Reddit). Unfortunately, RLHF does not train away sycophancy and may actively incentivize models to retain it."
• Wei et al 2023 (from Google DeepMind) ran a similar experiment with PaLM (and its instruction-tuned version Flan-PaLM). They too observed substantial sycophancy in sufficiently large base models, and even more sycophancy after instruction tuning (which was SFT here, not RL!).
  • See Fig. 2.
  • They used the same prompt template as Perez et al 2022.
  • Strikingly, the (SFT) instruction tuning result here suggests both that (a) post-training can increase sycophancy even if it isn't RL post-training, and (b) SFT post-training may actually be more sycophancy-promoting than RLHF, given the negative result for RLHF in Perez et al 2022.
• Sharma et al 2023 (from Anthropic) contains a more extensive investigation of sycophancy than the original Anthropic paper on the topic, and (among other things) presents results on the actual RL training stage used to train Claude 2. They find, again, that the model was already sycophantic before RL, although in their setting RL training does somewhat increase some forms of sycophancy.
  • Although, weirdly, best-of-N sampling against the same preference model gives totally different results, substantially decreasing some forms of sycophancy.
  • See Fig. 6 and surrounding discussion.
  • The authors write (section 4.2):
    • "With RL, some forms of sycophancy increase through the RL finetuning process used to produce Claude 2. However, the presence of sycophancy at the start of RL indicates that pretraining and supervised finetuning also likely contribute to sycophancy. Nevertheless, if the PM strongly disincentivized sycophancy, it should be trained out during RL, but we do not observe this."
• In this post (expanding upon this comment on Perez et al 2022), I ran one of the Perez et al 2022 sycophancy evals on various OpenAI text completion models. Unlike Perez et al (and Wei et al), I found that the base models I studied weren't sycophantic, while some of the instruction-tuned models were sycophantic – but the presence of sycophancy did not appear to correlate with the use of RL as a post-training algorithm.
  • In particular: the RL-tuned text-davinci-003 was strongly sycophantic, but so was text-davinci-002, which was tuned with an SFT variant that OpenAI calls "feedme" (see here for details).
  • But earlier feedme-tuned models were not sycophantic, suggesting that the difference has much more to do with changes in the SFT training data mix over time than with the choice of training algorithm.

Note that several of the works above do something equivalent to the experiment you propose, in the paragraph beginning with "Maybe a good test of this would be...".  So your prediction has already been tested, and (insofar as you trust the experimental setups) falsified.

If a LLM similarly doesn't do much information-gathering about the intent/telos of the text from the "assistant" character, and instead does an amplified amount of pre-computing useful information and then attending to it later when going through the assistant text, this paints a quite different picture to me than your "void."

I don't understand the distinction you're drawing here?  Any form of assistant training (or indeed any training at all) will incentivize something like "storing useful information (learned from the training data/signal) in the weights and making it available for use in contexts on which it is useful."

Moreover, the training signal in RL(HF) is much sparser than it is in SFT – because RL only provides a single scalar's worth of feedback on each entire model sample, while SFT provides feedback at every token position about which token (out of a large vocab) was correct in context – so if anything, I'd expect more under-determination from assistant-training setups that emphasize RLHF over SFT.

Perhaps some of the disconnect here involves differing notions of what RL is, and how it differs from other ways of training an LLM.

You refer to "RL" as though the implications of its use should be both highly significant and obvious to the reader of your comment ("But, RL. [...] Claude is a nice guy, but, RL").  But your beliefs about the impacts of RL are not obvious to me; I don't know what "but, RL" is supposed to mean without further clarification.  I suspect I also disagree with your perception of what makes RL different, but I can't confirm/disconfirm that impression without know what that perception is, which I don't.

If you want to know where I'm coming from re: RL, it may be helpful to know that I find this post pretty illuminating/"deconfusing."

Similarly, I don't think current AI models are cheating at programming tests because of training text about their low moral character. I think it's RL, programming tasks, training set, implied high reward for cheating.

Yes, of course – I don't think this is due to "training text about their low moral character."  But I don't think the worrying thing here is really "RL" (after all, RLHF was already RL) but rather the introduction of a new training stage that's narrowly focused on satisfying verifiers rather than humans (when in a context that resembles the data distribution used in that stage), which predictably degrades the coherence (and overall-level-of-virtue) of the assistant character.  I wrote about this yesterday here.

Lastly... OK, this is going to make me sound like a dick, and probably make people use the "Too Combative?" reaction icon or something, but in the interests of honesty and improving the discourse:

When I woke up this morning to see find that this comment had appeared, and that it was (at the time) the highest-karma comment on this post, I was like, "oh, yes, this is why I'm usually wary of posting long-form stuff on LW.  My gut response of 'ugh if I put this on LW I'll have to deal with the comments' was right."  (That gut response is probably getting RL-upweighted inside my brain right now...)

As evidenced perhaps by the length of my comment vs. yours, I have a tendency to get "nerd-sniped" by stuff that I think is clearly wrong according to some evidence base (and/or set of arguments) I already know about – especially when that stuff is about something I wrote myself, originally.  I just kinda can't help myself, I inevitably end up writing out these giant "takedown" responses almost before I even notice what I'm doing.  I've spent well over an hour, by now, writing this particular one.

And LW is a reliable minefield of such nerd-snipes.  There are plenty of comments/posts here that don't have the problems I'm talking about... but then inevitably there are comments/posts with those problems, and I fixate on them when they appear, and that fixation becomes a time/effort sink, and that in turn trains me into avoidance of posting here (and to some extent even reading posts by others, here).

Like... it's fine to pose questions to which you don't know the answers.  And it's also fine to make conjectures if you can provide clear and interesting arguments for why they might be true or important.  And it's also fine to confidently state claims if you also state them clearly and provide clear substantiating evidence and/or argumentation.

All of these things are fine, and some fraction of LW content consists only of these things in some mixture.  But then there's this stuff like "but RL!", which reliably pleases the karma hivemind while being none of the above.  I don't know what exactly you guys think "RL" means and entails; there are all these weird vague ideas about such topics floating around here that lots of people here seem to vaguely agree with, and I've lost whatever patience I used to have with them.  Just, please... lay out your ideas explicitly and say explicitly why you think they're true.

1. ^{^}

   ...although (c) the preference datasets – and hence the reward models – used for RL do show preferences for sycophantic responses (...well, sometimes, though see also the weird BoN results in Sharma et al 2023). So if you were to train indefinitely ("over-optimize") against these RMs they would presumably have a strong effect on sycophancy eventually.  But this kind of aggressive optimization against a sycophancy-preferring RM is certainly not necessary to produce noticeable sycophancy, and is probably not the cause behind most cases of LLM sycophancy that you and I notice in practice.

2. ^{^}

   See this comment by the lead author.