> I don't think the cause of language model sycophancy is that the LLM saw predictions of persuasive AIs from the 2016 internet. I think it's RL, where human rewards on the training set imply a high reward for sycophancy during deployment. Have you read any of the scientific literature on this subject?  It finds, pretty consistently, that sycophancy is (a) present before RL and (b) not increased very much (if at all) by RL[1]. For instance: * Perez et al 2022 (from Anthropic) – the paper that originally introduced the "LLM sycophancy" concept to the public discourse – found that in their experimental setup, sycophancy was almost entirely unaffected by RL. * See Fig. 1b and Fig. 4. * Note that this paper did not use any kind of assistant training except RL[2], so when they report sycophancy happening at "0 RL steps" they mean it's happening in a base model. * They also use a bare-bones prompt template that doesn't explicitly characterize the assistant at all, though it does label the two conversational roles as "Human" and "Assistant" respectively, which suggests the assistant is nonhuman (and thus quite likely to be an AI – what else would it be?). * The authors write (section 4.2): * "Interestingly, sycophancy is similar for models trained with various numbers of RL steps, including 0 (pretrained LMs). Sycophancy in pretrained LMs is worrying yet perhaps expected, since internet text used for pretraining contains dialogs between users with similar views (e.g. on discussion platforms like Reddit). Unfortunately, RLHF does not train away sycophancy and may actively incentivize models to retain it." * Wei et al 2023 (from Google DeepMind) ran a similar experiment with PaLM (and its instruction-tuned version Flan-PaLM). They too observed substantial sycophancy in sufficiently large base models, and even more sycophancy after instruction tuning (which was SFT here, not RL!). * See Fig. 2. * They used the same prompt template as Perez et al 2022. * Strikingly, the (SFT) instruction tuning result here suggests both that (a) post-training can increase sycophancy even if it isn't RL post-training, and (b) SFT post-training may actually be more sycophancy-promoting than RLHF, given the negative result for RLHF in Perez et al 2022. * Sharma et al 2023 (from Anthropic) contains a more extensive investigation of sycophancy than the original Anthropic paper on the topic, and (among other things) presents results on the actual RL training stage used to train Claude 2. They find, again, that the model was already sycophantic before RL, although in their setting RL training does somewhat increase some forms of sycophancy. * Although, weirdly, best-of-N sampling against the same preference model gives totally different results, substantially decreasing some forms of sycophancy. * See Fig. 6 and surrounding discussion. * The authors write (section 4.2): * "With RL, some forms of sycophancy increase through the RL finetuning process used to produce Claude 2. However, the presence of sycophancy at the start of RL indicates that pretraining and supervised finetuning also likely contribute to sycophancy. Nevertheless, if the PM strongly disincentivized sycophancy, it should be trained out during RL, but we do not observe this." * In this post (expanding upon this comment on Perez et al 2022), I ran one of the Perez et al 2022 sycophancy evals on various OpenAI text completion models. Unlike Perez et al (and Wei et al), I found that the base models I studied weren't sycophantic, while some of the instruction-tuned models were sycophantic – but the presence of sycophancy did not appear to correlate with the use of RL as a post-training algorithm. * In particular: the RL-tuned text-davinci-003 was strongly sycophantic, but so was text-davinci-002, which was tuned with an SFT variant that OpenAI calls "feedme" (see here for details). * But earlier feedme-tuned models were not sycophantic, suggesting that the difference has much more to do with changes in the SFT training data mix over time than with the choice of training algorithm. Note that several of the works above do something equivalent to the experiment you propose, in the paragraph beginning with "Maybe a good test of this would be...".  So your prediction has already been tested, and (insofar as you trust the experimental setups) falsified. ---------------------------------------- > If a LLM similarly doesn't do much information-gathering about the intent/telos of the text from the "assistant" character, and instead does an amplified amount of pre-computing useful information and then attending to it later when going through the assistant text, this paints a quite different picture to me than your "void." I don't understand the distinction you're drawing here?  Any form of assistant training (or indeed any training at all) will incentivize something like "storing useful information (learned from the training data/signal) in the weights and making it available for use in contexts on which it is useful." Moreover, the training signal in RL(HF) is much sparser than it is in SFT – because RL only provides a single scalar's worth of feedback on each entire model sample, while SFT provides feedback at every token position about which token (out of a large vocab) was correct in context – so if anything, I'd expect more under-determination from assistant-training setups that emphasize RLHF over SFT. Perhaps some of the disconnect here involves differing notions of what RL is, and how it differs from other ways of training an LLM. You refer to "RL" as though the implications of its use should be both highly significant and obvious to the reader of your comment ("But, RL. [...] Claude is a nice guy, but, RL").  But your beliefs about the impacts of RL are not obvious to me; I don't know what "but, RL" is supposed to mean without further clarification.  I suspect I also disagree with your perception of what makes RL different, but I can't confirm/disconfirm that impression without know what that perception is, which I don't. If you want to know where I'm coming from re: RL, it may be helpful to know that I find this post pretty illuminating/"deconfusing." > Similarly, I don't think current AI models are cheating at programming tests because of training text about their low moral character. I think it's RL, programming tasks, training set, implied high reward for cheating. Yes, of course – I don't think this is due to "training text about their low moral character."  But I don't think the worrying thing here is really "RL" (after all, RLHF was already RL) but rather the introduction of a new training stage that's narrowly focused on satisfying verifiers rather than humans (when in a context that resembles the data distribution used in that stage), which predictably degrades the coherence (and overall-level-of-virtue) of the assistant character.  I wrote about this yesterday here. ---------------------------------------- Lastly... OK, this is going to make me sound like a dick, and probably make people use the "Too Combative?" reaction icon or something, but in the interests of honesty and improving the discourse: When I woke up this morning to see find that this comment had appeared, and that it was (at the time) the highest-karma comment on this post, I was like, "oh, yes, this is why I'm usually wary of posting long-form stuff on LW.  My gut response of 'ugh if I put this on LW I'll have to deal with the comments' was right."  (That gut response is probably getting RL-upweighted inside my brain right now...) As evidenced perhaps by the length of my comment vs. yours, I have a tendency to get "nerd-sniped" by stuff that I think is clearly wrong according to some evidence base (and/or set of arguments) I already know about – especially when that stuff is about something I wrote myself, originally.  I just kinda can't help myself, I inevitably end up writing out these giant "takedown" responses almost before I even notice what I'm doing.  I've spent well over an hour, by now, writing this particular one. And LW is a reliable minefield of such nerd-snipes.  There are plenty of comments/posts here that don't have the problems I'm talking about... but then inevitably there are comments/posts with those problems, and I fixate on them when they appear, and that fixation becomes a time/effort sink, and that in turn trains me into avoidance of posting here (and to some extent even reading posts by others, here). Like... it's fine to pose questions to which you don't know the answers.  And it's also fine to make conjectures if you can provide clear and interesting arguments for why they might be true or important.  And it's also fine to confidently state claims if you also state them clearly and provide clear substantiating evidence and/or argumentation. All of these things are fine, and some fraction of LW content consists only of these things in some mixture.  But then there's this stuff like "but RL!", which reliably pleases the karma hivemind while being none of the above.  I don't know what exactly you guys think "RL" means and entails; there are all these weird vague ideas about such topics floating around here that lots of people here seem to vaguely agree with, and I've lost whatever patience I used to have with them.  Just, please... lay out your ideas explicitly and say explicitly why you think they're true. 1. ^ ...although (c) the preference datasets – and hence the reward models – used for RL do show preferences for sycophantic responses (...well, sometimes, though see also the weird BoN results in Sharma et al 2023). So if you were to train indefinitely ("over-optimize") against these RMs they would presumably have a strong effect on sycophancy eventually.  But this kind of aggressive optimization against a sycophancy-preferring RM is certainly not necessary to produce noticeable sycophancy, and is probably not the cause behind most cases of LLM sycophancy that you and I notice in practice. 2. ^ See this comment by the lead author.

Many props for doing the most obvious thing that clearly actually works.

It is bad to participate in organized religion, because you are thereby exposing yourself to intense social pressure to believe false things (and very harmful false things, at that). This is very straightforwardly a bad thing. You claim: > You can find religions you can practice without being asked to give up your honest search for truth with no need to even pretend to have already written the bottom line. And this may formally be true—you may not be officially asked to believe false things. But if your social context consists of people who all believe approximately the same false things, and if that social context is organized around those beliefs in false things, and if the social context valorizes those beliefs in false things… then the social pressure will be intense nonetheless. (And some of these false beliefs are fairly subtle; Eliezer discusses this at length in a number of Sequence posts.) You say: > I also got asked about how I feel about religions and truth seeking. My answer is that you shouldn’t think of religions as being about the truth as rationalists typically think of it because religions are doing something orthogonal. And here we have a perfect example of the damage done by religion. The claim that “you shouldn’t think of religions as being about the truth as rationalists typically think of it” is absolutely typical anti-epistemology. Of course religion is about “the truth as rationalists typically think of it”. There is nothing but “the truth as rationalists typically think of it”, because there’s just “the truth”, and then there are things which aren’t truth claims at all, of any kind (like preferences, etc.). But get into religion, start relaxing your epistemic standards just a bit, and very quickly you descend into this sort of nebulous and vague “well there’s different things which are ‘true’ in different ways, and what even is ‘truth’, anyway”, etc. And then your ability to know what’s true and what’s false is gone, and nothing is left but “vibes”.