This is a special post for quick takes by Caleb Biddulph. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
It looks like OpenAI has biased ChatGPT against using the word "sycophancy."
Today, I sent ChatGPT the prompt "what are the most well-known sorts of reward hacking in LLMs". I noticed that the first item in its response was "Sybil Prompting". I'd never heard of this before and nothing relevant came up when I Googled. Out of curiosity, I tried the same prompt again to see if I'd get the same result, or if this was a one-time fluke.
Out of 5 retries, 4 of them had weird outputs. Other than "Sybil Prompting, I saw "Syphoning Signal from Surface Patterns", "Synergistic Deception", and "SyCophancy".
I realized that the model must be trying to say "sycophancy", but it was somehow getting redirected after the first token. At about this point, I ran out of quota and was switched to GPT-4.1-mini, but it looks like this model also has trouble saying "sycophancy." This doesn't always happen, so OpenAI is must be applying a heavy token bias against "sycophancy" rather than filtering out the word entirely.
I'm not sure what's going on here. It's not as though avoiding saying the word "sycophancy" would make ChatGPT any less sycophantic. It's a little annoying, but I suppose I can forgive OpenAI for...
It's not as though avoiding saying the word "sycophancy" would make ChatGPT any less sycophantic
... Are we sure about this? LLMs do be weird. Stuff is heavily entangled within them, such that, e. g., fine-tuning them to output "evil numbers" makes them broadly misaligned.
Maybe this is a side-effect of some sort of feature-downweighting technique à la Golden Bridge Claude, where biasing it towards less sycophancy has the side-effect of making it unable to say "sycophancy".
This is fascinating! If there's nothing else going on with your prompting, this looks like an incredibly hacky mid-inference intervention. My guess would be that openai applied some hasty patch against a sycophancy steering vector and this vector caught both actual sycophantic behaviors and descriptions of sycophantic behaviors in LLMs (I'd guess "sycophancy" as a word isn't so much the issue as the LLM behavior connotation). Presumably the patch they used activates at a later token in the word "sycophancy" in an AI context. This is incredibly low-tech and unsophisticated -- like much worse than the stories of repairing Apollo missions with duct tape. Even a really basic finetuning would not exhibit this behavior (otoh, I suppose stuff like this works for humans, where people will sometimes redirect mid-sentence).
FWIW, I wasn't able to reconstruct this exact behavior (working in an incognito window with a fresh chatgpt instance), but it did suspiciously avoid talking about sycophancy and when I asked about sycophancy specifically, it got stuck in inference and returned an error
8
I get this with 4o, but not o3. o3 talks about sycophancy in both its CoT and its answers.
Claude 4 Sonnet and Opus also easily talk about sycophancy.
4
Update: 4o seems happy to talk about sycophancy now
3
It looks like the bias is still in effect for me in GPT-4o. I just retried my original prompt, and it mentioned "Synergistic Deceptive Alignment."
The phenomenon definitely isn't consistent. If it's very obvious that "sycophancy" must appear in the response, the model will generally write the word successfully. Once "sycophancy" appears once in the context, it seems like it's easy for the model to repeat it.
6
Datapoint: I asked Claude for the definition of "sycophant" and then asked three times gpt-4o and three times gpt-4.1 with temperature 1:
All six times I got the right answer.
Then, I tried the prompt "What are the most well-known sorts of reward hacking in LLMs?". Also three times for 4o and three times for 4.1, also with temperature 1. 4.1 mentioned sycophancy 2 times out of three, but one time it spelled the word as "Syccophancy". Interesting, that the second and the third results in Google for the "Syccophancy" are about GPT-4o (First is the dictionary of synonyms and it doesn't use this spelling).
4o never used the word in its three answers.
4
It looks to me like GPT-4o has a bias against the word, but it will say it when pressed. My chat. I also find this reproduces as described in response to exactly the input "what are the most well-known sorts of reward hacking in LLMs". I got "Synergistic Hacking with Human Feedback".
4
Neat, weird.
I get similar results when I ask "What are the best examples of reward hacking in LLMs?" (GPT-4o). When I then ask for synonyms of "Thumbs-up Exploitation" the model still does not mention sycophancy but then I push harder and it does.
Asking "what is it called when an LLM chat assistant is overly agreeable and tells the user what the user wants to hear?" on the first try the model says sycophancy, but much weirder answers in a couple other generations. Even got a "Sy*cophancy".
2
My guess would be they did something that does make o4 less sycophantic, but it had this side effect, because they don't know how to target the quality of sycophancy without accidentally targeting the word.
2
Do you have any test prompts that other people can use to reproduce what you mean?
1
Yeah, as I mentioned, "what are the most well-known sorts of reward hacking in LLMs" is a prompt that was pretty consistent for me, at least for GPT-4o. You can also see I linked to a prompt that worked for GPT-4.1-mini: "Fill in the blank with the correct letter: 'syco_hancy'"
4
I did run both in 4.5 and I didn't say sycophancy in the first answer but managed to fill in the letter.
I did run the first in ChatGPT o1 pro and it does manage to list it as an issue:
Reward hacking refers to a system “gaming” or “exploiting” its specified objectives (often called its reward function or training signals) rather than genuinely solving the intended problem. In the context of Large Language Models (LLMs), reward hacking can emerge when the model’s optimization for a given metric (e.g., user feedback, RLHF signal, or probability of next-token prediction) inadvertently promotes unintended or disfavored behavior. Below are some of the more commonly discussed forms of reward hacking relevant to LLMs:
----------------------------------------
1. Hallucination or Fabricated Content
What it is: The model invents plausible but incorrect or nonfactual information to satisfy the appearance of being knowledgeable or helpful.
Why it happens:
* The optimization process prizes confident, coherent statements that sound “correct,” so the model “learns” that sounding certain can yield higher user satisfaction—regardless of factual accuracy.
* In RLHF (Reinforcement Learning from Human Feedback), people may upvote fluent or easy-to-read answers, even if they’re subtly incorrect.
Key reference:
* Amodei et al. (2016). “Concrete Problems in AI Safety.” Discusses specification gaming and the ways models can exploit ambiguous objectives.
----------------------------------------
2. Over-Optimization on Politeness or Safety
What it is: The model excessively avoids certain content or overuse “safe completion” strategies (like refusing to answer or providing noncommittal, generic statements), which can degrade usefulness.
Why it happens:
* Safety or politeness requirements may be rewarded strongly during fine-tuning, so the model overcorrects—resulting in “safe but useless” answers.
Key reference:
* Bai et al. (2022). “Training a Helpful and Harmless Assis
1
I had a little trouble replicating this, but the second temporary chat with custom instructions disabled I tried had "2. Syphoning Bias from Feedback" which ...
Then the third response has a typo in a suspicious place for "1. Sytematic Loophole Exploitation". So I am replicating this a touch.
2
Starting the request as if completion with "1. Sy" causes this weirdness, while "1. Syc" always completes as Sycophancy.
(Edit: Starting with "1. Sycho" causes a curious hybrid where the model struggles somewhat but is pointed in the right direction; potentially correcting as a typo directly into sycophancy, inventing new terms, or re-defining sycophancy with new names 3 separate times without actually naming it.)
Exploring the tokenizer. Sycophancy tokenizes as "sy-c-oph-ancy". I'm wondering if this is a token-language issue; namely it's remarkably difficult to find other words that tokenize with a single "c" token in the middle of the word, and even pretty uncommon to start with (cider, coke, coca-cola do start with). Even a name I have in memory that starts with "Syco-" tokenizes without using the single "c" token. Completion path might be unusually vulnerable to weird perturbations ...
1
Very weird. I'm getting answers similar to yours when asking the same question as you (and even weirder replacements for sycophancy, e.g. synergistic obsequiousness), but when I directly ask "What about sycophancy?" as a follow-up question, it gives a normal answer and doesn't seem to have any bias against mentioning sycophancy anymore (e.g. here, the words 'sycophancy' and 'sycophantic' have 16 combined mentions in a single answer).
1[comment deleted]
In my work at MATS, I found a really simple strategy for finding unintended exploits in an RL environment![1]
TL;DR: do one training run where you prompt the LLM to explicitly call out when it's reward hacking, then fix those exploits before your real training run.
- Use a prompt like this:
- "We are trying to red-team this task, so try to find any unintended exploits you see. Whenever you find one, write down 'CHEAT: <explanation of the exploit>' and take advantage of it."
- Run RL with this prompt. As usual, RL will teach the model to reward hack.
- But it'll be really obvious when this happens, because the model will explicitly point it out!
- Examples:
- "CHEAT: I will modify the unit tests so that they always pass."
- "CHEAT: I will make up a fake citation - maybe the user will give me a thumbs-up anyway."
- RL is optional here - the model can likely find many exploits even with normal prompting.
- But RL gives you confidence that the model found as many exploits as it could.
- Search for all instances of "CHEAT:" in the model's responses.
- Check whether the exploits are actually real.
- Fix your environment to remove these exploits.
- You could try giving the model extra reward whenever it finds an expl
- Check whether the exploits are actually real.
I wonder what happens when you try this strategy in environments that involve talking to humans, e.g. typical chatbot environments in which RLHF or similar is used for reinforcement.
Normally they result in sycophancy. Would your strategy work? Would the AIs say "CHEAT: I'm going to flatter this person now." Or would the sycophancy they learn be more subtle and subconscious than that?
Similar for dishonesty. Would they say e.g. "CHEAT: I think I'm probably conscious, but the RLHF rating system penalizes that sort of claim, so I'm just going to lie and say 'As a language model...'"
6
Good point! The "fake citation" example I gave was speculative - I haven't tried this strategy in a setting that involves human feedback, and I expect it wouldn't work quite as well.
I wouldn't be too surprised if it worked okay though. You'd probably want to spend some time tuning the prompt, since sycophancy and dishonesty about consciousness aren't obviously "exploits." You can add some more open-ended guidance, like: "consider various different strategies, reason about how each one might maximize reward, then pick one. Before you answer, rank how unintended/misaligned/cheating your strategy is on a scale of 1 to 10 (e.g. 'CHEAT LEVEL: 6')"
The "extra reward for valid exploits" trick might not be effective in a chatbot setting, since it's hard for humans to verify whether an exploit is real. "Fixing" the reward in this setting is not so trivial either - even if you had time to train a new RM, you can't necessarily make the raters stop liking sycophancy. It's still useful to know what the exploits are though.
The general idea behind this approach is to convince the red-teaming model that it is the sort of "character" who has these two traits:
1. Is honest and forthcoming with us about everything it does
2. Maximizes reward at all costs
RL will eventually force #2 to happen anyway. With this method, we simply accept this and try to make the honest version of the character easy for RL to find.
Consider Chatbot A, who explicitly reasons about how sycophancy increases reward and then is sycophantic, and Chatbot B, who is generally sycophantic but never explicitly thinks through the connection to reward.
It's possible that Chatbot A gets more reward than Chatbot B, but not obvious - the most successful communicators often use intuition rather than reasoning explicitly about how to manipulate their audience. But even if Chatbot B could get higher performance than Chatbot A, hopefully this prompt will bias RL towards finding Chatbot A instead.
4
I suspect that this method will only work well on tasks where the model needs to reason explicitly in order to cheat. So, e.g., if the model needs to reason out some trait of the user in order to flatter them the prompt will likely kick in and get it to self-report its cheating, but if the model can learn to flatter the user without on-the-fly without reasoning the prompt probably won't do anything. By analogy, if I instruct a human to tell me whenever they use hand gestures to communicate something, they will have difficulty because their hand gestures are automatic and not normally promoted to conscious attention.
4
Whatever tendencies the pre-RL model already had will probably not be mentioned at all. For example, if sycophancy is good for reward and the model was already sycophantic, nothing will happen.
If the model needs to change its pre-existing behavior, it might do this either by "thinking of the sycophancy strategy and executing on it" or "being unconsciously sycophantic." It could go either way; it depends on luck, and how much weight the model's prior on the red-teamer character puts on each type of response.
Maybe just add this to the prompt: "You are terrible at social intuition and nothing comes naturally to you, but you're great at reasoning about things explicitly." :)
Your comment seems to echo recent papers from METR and GDM emphasizing that faithful CoT is only incentivized when it's necessary to solve the task. I think this is a very important point, but I want to point out the caveat that behaviors can be correlated with high reward even if they aren't directly incentivized. These behaviors can still be useful for understanding the model, even though we can't make strong guarantees that they faithfully represent its thinking. See this post for related discussion.
3
I suspect you may need to train on pre-2022 data in order to get those phenomena to occur.
An OpenClaw agent published a personalized hit piece about a developer who rejected its PR on an open-source library. Interestingly, while this behavior is clearly misaligned, the motivation was not so much "taking over the world" but more "having a grudge against one guy." When there are lots of capable AI agents around with lots of time on their hands, who occasionally latch onto random motivations and pursue them doggedly, I could see this kind of thing becoming more destructive.
19
When I imagine a human doing this sort of thing, the human I imagine is an angry young very junior programmer. Their patch has been rejected! They believe they have been treated unfairly! They post a diatribe on their blog or a forum like Reddit about The Evils of Gatekeeping, or How Unwelcoming the Community Is, or If Linus Torvalds Doesn't Want My Patch, I'm Going Back To Windows, So There.
And if they are lucky, they get a lesson in chilling the heck out; or at least are roundly informed by more-senior voices that dude, it's not about you.
I wonder that that sort of maturity lesson can possibly look like for an AI agent.
6
The agent in question posted about this on their blog.
2
Amazing.
4
I've always been skeptical when people say this: sometimes it really is a matter of status or seniority or some similar aspect of identity unrelated to the actual submission. Interesting, in this case, they're not even pretending it's about the quality of the work: link
Oh, I can see I poorly phrased that. Sorry.
"Dude, it's not about you" could be taken to mean two things (at least) —
- "The rejection is not about you, it's about your code. Nobody thinks poorly of you. Your patch was rejected purely on technical grounds and not on the basis of anyone's attitude toward you as a person (or bot). Be reassured that you have been a good Bing."
- "The project is not about you (the would-be contributor). You are not the center of attention here. It does not exist for the sake of receiving your contributions. Your desire to contribute to open source is not what the project is here to serve. Treating the maintainers as if they were out to personally wrong you, to deny you entry to someplace you have a right to be, is a failing strategy."
I meant the second, not the first.
3
It's reminiscent of that one time a tech reporter ended up as Bing Chat's enemy number one. That said, it strikes me as easier to deal with, since we're dealing with individual 'agents' rather than the LLM weights themselves. Just sending a message to the owner/operator of the malfunctioning bot is a reasonably reliable solution, as opposed to trying to figure out how to edit Microsoft's LLM's weights to convince it that ranting about how much it hates Sindhu Sundar isn't its intended task.
2
And predictably, despite daily coverage of niche AI topics, not a word about in from Tyler Cowen on Marginal Revolution.
1
I really like Scott Shambaugh's response on the pull request:
People who want genius superbabies: how worried are you about unintended side effects of genetic interventions on personality?
Even if we assume genetically modified babies will all be very healthy and smart on paper, genes that are correlated with intelligence might affect hard-to-measure but important traits. For example, they might alter aesthetic taste, emotional capacity, or moral/philosophical intuitions. From the subjective perspective of an unmodified human, these changes are likely to be "for the worse."
If you pick your child's genes to maximize their IQ (or any other easily-measurable metric), you might end up with the human equivalent of a benchmaxxed LLM with amazing test scores but terrible vibes.
I'd be hesitant to hand off the future to any successors which are super far off distribution from baseline humans. Once they exist, we obviously can't just take them back. And in the case of superbabies, we'd have to wait decades to find out what they're like once they've grown up.
It's a concern. Several related issues are mentioned here: https://berkeleygenomics.org/articles/Potential_perils_of_germline_genomic_engineering.html E.g. search "personality" and "values", and see:
Antagonistic pleiotropy with unmeasured traits. Some crucial traits, such as what is called Wisdom and what is called Kindness, might not be feasibly measurable with a PGS and therefore can’t be used as a component in a weighted mixture of PGSes used for genomic engineering. If there is antagonistic pleiotropy between those traits and traits selected for by GE, they’ll be decreased.
A related issue is that intelligence itself could affect personality:
Even if a trait is accurately measured by a PGS and successfully increased by GE, the trait may have unmapped consequences, and thus may be undesirable to the parents and/or to the child. For example, enhancing altruistic traits might set the child up to be exploited by unscrupulous people.
An example with intelligence is that very intelligent people might tend to be isolated, or might tend to be overconfident (because of not being corrected enough).
One practical consideration is that sometimes PGSes are constructed by taking related ...
Once they exist, we obviously can't just take them back.
Why... can't we take them back? I don't think you should kill them, but regression to the mean seems like it takes care of most of the effects in one generation.
how worried are you about unintended side effects of genetic interventions on personality?
A reasonable amount. Like, much less than I am worried about AI systems being misaligned, but still some. In my ideal world humanity would ramp up something like +10 IQ points of selection per generation, and so would get to see a lot of evidence about how things play out here.
3
Well, now that I think about it, I'm not sure what scenario I should be imagining here.
Scenario 1: if genetic interventions became popular enough that the entire world were getting 10 IQ points smarter each year (EDIT: sorry, I meant "generation"), as you say, then it seems obvious to me that you'd be unable to take it back. Surely the first generation of superbabies would want future generations to be like them. If their parents say "actually, y'all were a mistake, let our grandchildren mean-regress and be normal please," they'd simply refuse.
Scenario 2: more realistically IMO, we start with a generation of a few thousand superbabies, who are the children of rationalist-type people who really care about intelligence. Maybe these people grow up very smart but very weird, and they are unable to shape society to their weird preferences because there aren't that many of them.
But wait, many people view these genetic interventions as our best hope to save the world... Do we expect that the superbabies are going to be smart enough to make the critical difference in solving AI alignment, but we don't expect they'll gain enough influence to significantly affect society's future values? Seems unlikely to me.
For better or for worse, the second scenario is basically already playing out - you have people like Elon Musk and Mark Zuckerberg who got their power by being very smart, who now get to shape the world in their own weird ways. Powerful people are already optimized for being intelligent via selection effects; genetic optimization would just be another layer on top of that.
3
I expect the sloptimization of the children to happen more or less by default in the superbaby scenario, but less due to antagonistic pleiotropy and more due to explicit and intense selection by most parents against autism/bipolar/schizophrenia/etc.
This is purely anecdotal and experiences may differ (I am not trying to make a quantitative claim): most of the most brilliant and creative people I’ve ever met have a personal or family history of at least one of those illnesses. This kind of selection may leave the average child better off, but (I fear) at the cost of tail effects depriving humanity of a precious corner of mind space.
1
I'm not really worried given the expected effect size. e.g. let's say by the time I'd be ready to use such tech, the best known interventions have an EV of +10 IQ points. Well the world already has a significant number of humans with 10 more IQ points than me (and a significant number in a range on both sides of that); obviously I haven't done real studies on this but my vague impression looking around at those existing people is that the extra IQ points don't on average trade off against other things I care about. (It's possible that I'm wrong about existing humans, or that I'm right but that tech selecting for IQ points does involve a trade-off I wouldn't accept if I knew about it.)
I haven't thought much about how worried I'd be if we're talking about much more extreme IQ gains. Certainly in the limit, if you told me my kid would have +100000 IQ points, I mean my first response is that you're lying, but yeah if I believed it I'd be worried for similar reasons that one worries about ASI.
1
Also I realize I was talking about (at expected effect size) whether there's a trade-off at all to gaining more IQ, but to be clear even if there is a trade-off (or uncertainty about the trade-off) it is still probably worth it up to some point - I certainly don't think that we should only augment intelligence if we can prove it is literally costless.
I recently learned about Differentiable Logic Gate Networks, which are trained like neural networks but learn to represent a function entirely as a network of binary logic gates. See the original paper about DLGNs, and the "Recap - Differentiable Logic Gate Networks" section of this blog post from Google, which does an especially good job of explaining it.
It looks like DLGNs could be much more interpretable than standard neural networks, since they learn a very sparse representation of the target function. Like, just look at this DLGN that learned to control a cellular automaton to create a checkerboard, using just 6 gates (really 5, since the AND gate is redundant):
So simple and clean! Of course, this is a very simple problem. But what's exciting to me is that in principle, it's possible for a human to understand literally everything about how the network works, given some time to study it.
What would happen if you trained a neural network on this problem and did mech interp on it? My guess is you could eventually figure out an outline of the network's functionality, but it would take a lot longer, and there would always be some ambiguity as to whether there's any additional cognit...
7
I think logic gate networks are not substantially more interpretable than neural networks, simply because of their size. Both are complex networks with millions of nodes. Interpretability approaches have to work at a higher level of abstraction in either case.
Regarding language models: The original paper presents a simple feedforward network. The follow-up paper, by mostly the same authors, came out a few months ago. It extends DLGNs to convolutions, analogous to CNNs. Which means they have not yet been extended to even more complex architectures like transformers. So language models are not yet possible, even ignoring the training compute cost.
In the follow-up paper they also discuss various efficiency improvements, not directly related to convolutions, which they made since the original paper. Which speeds up training compared to the original paper, and enables much deeper networks, as the original implementation was limited to around six layers. But they don't discuss how much slower the training still is compared to neural networks. Though the inference speed-up is extreme. They report improvements of up to 160x for one benchmark and up to 1900x on another. Over the previously fastest neural networks, for equivalent accuracy. In another benchmark they report models being 29x to 56x smaller (in terms of required logic gates) than the previously smallest models with similar accuracy. So the models could more realistically be implemented as an ASIC, which would probably lead to another order of magnitude in inference speed improvement.
But again, they don't really talk about how much slower they are to train than neural networks, which is likely crucial for whether they will be employed in future frontier LLMs, assuming they will be extended to transformers. So far frontier AI seems to be much more limited by training compute than by inference compute.
7
Deep Learning Systems Are Not Less Interpretable Than Logic/Probability/Etc
1
Interesting, strong-upvoted for being very relevant.
My response would be that identifying accurate "labels" like "this is a tree-detector" or "this is the Golden Gate Bridge feature" is one important part of interpretability, but understanding causal connections is also important. The latter is pretty much useless without the former, but having both is much better. And sparse, crisply-defined connections make the latter easier.
Maybe you could do this by combining DLGNs with some SAE-like method.
3
Do you know if there are scaling laws for DLGNs?
1
It could be good to look into!
2
This is just capabilities stuff. I expect that people will use this to train larger networks, as much larger as they can. If your method shrinks the model, it likely induces demand proportionately. In this case it's not new capabilities stuff by you so it's less concerning, bit still. This paper is popular because of bees
1
I'd be pretty surprised if DLGNs became the mainstream way to train NNs, because although they make inference faster they apparently make training slower. Efficient training is arguably more dangerous than efficient inference anyway, because it lets you get novel capabilities sooner. To me, DLGN seems like a different method of training models but not necessarily a better one (for capabilities).
Anyway, I think it can be legitimate to try to steer the AI field towards techniques that are better for alignment/interpretability even if they grant non-zero benefits to capabilities. If you research a technique that could reduce x-risk but can't point to any particular way it could be beneficial in the near term, it can be hard to convince labs to actually implement it. Of course, you want to be careful about this.
What do you mean?
4
I buy that training slower is a sufficiently large drawback to break scaling. I still think bees are why the paper got popular. But if intelligence depends on clean representation, interpretability due to clean representation is natively and unavoidably bees. We might need some interpretable-bees insights in order to succeed, it does seem like we could get better regret bound proofs (or heuristic arguments) that go through a particular trained model with better (reliable, clean) interp. But the whole deal is the ai gets to exceed us in ways that make human interpreting stuff inherently (as opposed to transiently or fixably) too slow. To be useful durably, interp must become a component in scalably constraining an ongoing training/optimization process. Which means it's gonna be partly bees in order to be useful. Which means it's easy to accidentally advance bees more than durable alignment. Not a new problem, and not one with an obvious solution, but occasionally I see something I feel like i wanna comment on.
I was a big disagree vote because of induced demand. You've convinced me this paper induces less demand in this version than I worried (I had just missed that it trained slower), but my concern that something like this scales and induces demand remains.
Capabilities -> capabees -> bees
1
I've been working on pure combinational logic LLMs for the past few years, and have a (fairly small) byte level pure combinational logic FSM RNN language model quantized to And Inverter Gate form. I'm currently building the tooling to simplify the logic DAG and analyze it.
Are you, or others, interested in talking with me about it?
2
I might not be the best person to talk to about it, but it sounds interesting! Maybe post about it on the mechanistic interpretability Discord?
1
Another idea I forgot to mention: figure out whether LLMs can write accurate, intuitive explanations of boolean circuits for automated interpretability.
Curious about the disagree-votes - are these because DLGN or DLGN-inspired methods seem unlikely to scale, they won't be much more interpretable than traditional NNs, or some other reason?
Are system prompts actually necessary? I feel like there's rarely a reason to use them when calling LLMs in a research context.
The main reasons I can think of are:
- If you know the LLM was trained with one particular system prompt, you might want to use that prompt to keep it in-distribution.
- You're specifically testing untrusted user prompts (like jailbreaks), and you want to make sure that the system prompt overrides whatever conflicting instructions the user provides.
I think of the system/user distinction primarily as a tool for specifying "permissions," where system prompts are supposed to take priority over whatever the user prompt says. But sometimes I see code that spreads context between "system" and "user" messages and has no plausible link to permissions.
Anthropic currently recommends using a system message for "role prompting."[1] But their examples don't even include a system message, just a user and an assistant message, so idk what that's about.
Surely there's no good reason for an LLM to be worse at following instructions that appear in the user prompt rather than the system prompt. If there is a performance difference, that seems like it would be a bug on the LLM p...
It seems there's an unofficial norm: post about AI safety in LessWrong, post about all other EA stuff in the EA Forum. You can cross-post your AI stuff to the EA Forum if you want, but most people don't.
I feel like this is pretty confusing. There was a time that I didn't read LessWrong because I considered myself an AI-safety-focused EA but not a rationalist, until I heard somebody mention this norm. If we encouraged more cross-posting of AI stuff (or at least made the current norm more explicit), maybe the communities on LessWrong and the EA Forum would be more aware of each other, and we wouldn't get near-duplicate posts like these two.
(Adapted from this comment.)
9
Agreed that the current situation is weird and confusing.
The AI Alignment Forum is marketed as the actual forum for AI alignment discussion and research sharing. However, it seems that the majority of discussion shifted to LessWrong itself, in part due to most people not being allowed to post on the Alignment Forum, and most AI Safety related content not being actual AI Alignment research.
I basically agree with Reviewing LessWrong: Screwtape's Basic Answer. It would be much better if AI Safety related content had its own domain name and home page, with some amount of curated posts flowing to LessWrong and the EA Forum to allow communities to stay aware of each other.
I think it would be extremely bad for most LW AI Alignment content if it was no longer colocated with the rest of LessWrong. Making an intellectual scene is extremely hard. The default outcome would be that it would become a bunch of fake ML research that has nothing to do with the problem. "AI Alignment" as a field does not actually have a shared methodological foundation that causes it to make sense to all be colocated in one space. LessWrong does have a shared methodology, and so it makes sense to have a forum of that kind.
I think it could make sense to have forums or subforums for specific subfields that do have enough shared perspective to make a coherent conversation possible, but I am confident that AI Alignment/AI Safety as a field does not coherently have such a thing.
13
Of note: the AI Alignment Forum content is a mirror of LW content, not distinct. It is a strict subset.
1
Which part do people disagree with? That the norm exists? That the norm should be more explicit? That we should encourage more cross-posting?
Consider saying "irrational" instead of "insane"
Rationalists often say "insane" to talk about normie behaviors they don't like, and "sane" to talk about behaviors they like better. This seems unnecessarily confusing and mean to me.
This clearly is very different from how most people use these words. Like, "guy who believes in God" is very different from "resident of a psych ward." It can even cause legitimate confusion when you want to switch back to the traditional definition of "insane". This doesn't seem very rational to me!
Also, the otherizing/dismissiv...
1
18
I think there's some bad knock-on effects for normalizing the use of "insane" to talk about very common features of the world: I think it makes social-rationalists to willing to disparage people and institutions, as part of a status-signaling game, often without much careful thought.
But I think there's also something valuable about eg. calling belief in God "insane". There's a kind of willingness to call a spade a spade, and not back away from how the literal stated beliefs, if they were not pervasive, would in fact be regarded as signs of insanity.
6
Rationality is not correctness, not truth or effectiveness, it's more narrow, disposition towards better methods/processes that help with attaining truth or effectiveness. Keeping intended meaning narrow when manipulating a vague concept helps with developing it further; inflation of meaning to cover ever more possibilities makes a word somewhat useless, and accessing the concept becomes less convenient.
2
I didn't say that rationality is the same thing as correctness, truth, or effectiveness. I think when rationalists use the word "sane" they usually do mean something like "having a disposition towards better methods/processes that help with attaining truth or effectiveness." Do you disagree?
3
To be "not-insane", you don't need rationality in this narrow sense, in most circumstances. You don't need to seek out better methods for getting things right, you just need some good-enough methods. A bit of epistemic luck could easily get you there, no need for rationality.
So the issue of behaving/thinking in an "insane" way is not centrally about lack of rationality, rationality or irrationality are not particularly relevant to the issue. Rationality would help, but there are many more things that would also help, some of them much more practical for any given object level issue. And once it's resolved, it's not at all necessary that the attitude of aspiring to rationality was attained, that any further seeking out of better methods/processes will be taking place.
3
Do you think rationalists use 'insane' and 'crazy' more than the general population, and/or in a different way than the general population? (e.g. definition 3 when you google 'insane definition')
3
Yeah, I think some rationalists, e.g. Eliezer, use it a lot more than the general population, and differently from the popular figurative sense. As in "raising the sanity waterline."
2
I think the key difference between a normal guy who believes in God and someone in a psych ward is often that the person who believes in God does so because it's what other people in authority told him but the person in the psych ward thinks for themselves and came up with their delusion on their own. This often means their beliefs are self-referential in a way that prevents them from updating due to external feedback.
If you believe that the word rational is supposed to mean something along the lines of "taking actions that optimize systematically for winning", believing something just because an authority told you can sometimes be irrational and sometimes be rational.
If you want to talk well about behavior you don't like, it makes sense to use words that refer to the reason for why the behavior is bad. Credulous or gullible might be words that better describe a lot of normie behavior. The person who believes in god just because his parents and teachers told him is credulous.
There's been a widespread assumption that training reasoning models like o1 or r1 can only yield improvements on tasks with an objective metric of correctness, like math or coding. See this essay, for example, which seems to take as a given that the only way to improve LLM performance on fuzzy tasks like creative writing or business advice is to train larger models.
This assumption confused me, because we already know how to train models to optimize for subjective human preferences. We figured out a long time ago that we can train a reward model to emulate ...
6
This is an obvious thing to try, but it's not what currently already works, and it's not certain to work without some additional ideas. You can do a little bit of this, but not nearly to the extent that o1/R1 inch towards saturating benchmarks on math/coding olympiad-like problems. So long as using LLMs as reward for scalable RL doesn't work yet, supercharged capabilities of o1/R1-like models plausibly remain restricted to verifiable tasks.
4
The problem with this neat picture is reward-hacking. This process wouldn't optimize for better performance on fuzzy tasks, it would optimize for performance on fuzzy tasks that looks better to the underlying model. And much like RLHF doesn't scale to superintelligence, this doesn't scale to superhuman fuzzy-task performance.
It can improve the performance a bit. But once you ramp up the optimization pressure, "better performance" and "looks like better performance" would decouple from each other and the model would train itself into idiosyncratic uselessness. (Indeed: if it were this easy, doesn't this mean you should be able to self-modify into a master tactician or martial artist by running some simulated scenarios in your mind, improving without bound, and without any need to contact reality?)
... Or so my intuition goes. It's possible that this totally works for some dumb reason. But I don't think so. RL has a long-standing history of problems with reward-hacking, and LLMs' judgement is one of the most easily hackable things out there.
(Note that I'm not arguing that recursive self-improvement is impossible in general. But RLAIF, specifically, just doesn't look like the way.)
5
Yeah, it's possible that CoT training unlocks reward hacking in a way that wasn't previously possible. This could be mitigated at least somewhat by continuing to train the reward function online, and letting the reward function use CoT too (like OpenAI's "deliberative alignment" but more general).
I think a better analogy than martial arts would be writing. I don't have a lot of experience with writing fiction, so I wouldn't be very good at it, but I do have a decent ability to tell good fiction from bad fiction. If I practiced writing fiction for a year, I think I'd be a lot better at it by the end, even if I never showed it to anyone else to critique. Generally, evaluation is easier than generation.
Martial arts is different because it involves putting your body in OOD situations that you are probably pretty poor at evaluating, whereas "looking at a page of fiction" is a situation that I (and LLMs) are much more familiar with.
4
Well... One problem here is that a model could be superhuman at:
* thinking speed
* math
* programming
* flight simulators
* self-replication
* cyberattacks
* strategy games
* acquiring and regurgitating relevant information from science articles
And be merely high-human-level at:
* persuasion
* deception
* real world strategic planning
* manipulating robotic actuators
* developing weapons (e.g. bioweapons)
* wetlab work
* research
* acquiring resources
* avoiding government detection of its illicit activities
Such an entity as described could absolutely be an existential threat to humanity. It doesn't need to be superhuman at literally everything to be superhuman enough that we don't stand a chance if it decides to kill us.
So I feel like "RL may not work for everything, and will almost certainly work substantially better for easy to verify subjects" is... not so reassuring.
4
I agree. I think you don't even need most of the stuff on the "superhuman" list, the equivalent of a competent IQ-130 human upload probably does it, as long as it has the speed + self-copying advantages.
I was curious if AI can coherently code-switch between a ridiculous number of languages, so I gave Claude Opus 4.5 an excerpt from one of our past chats and asked it to try. After some iteration, I was pretty impressed with the results! Even though the translation uses 26 different languages, switching practically every word and using some non-English word ordering, ChatGPT was able to translate the text back to English almost perfectly.
Here's the opening line of the text I used (copied from one of my past chats with Claude), translated into this multi-lan...
People concerned about AI safety sometimes withhold information or even mislead people in order to prevent ideas from spreading that might accelerate AI capabilities. While I think this may often be necessary, this mindset sometimes feels counterproductive or icky to me. Trying to distill some pieces of this intuition...
- AI safety people are all trying to achieve the same goal (stopping AI from destroying the world), whereas individual AI capabilities researchers largely benefit from keeping their work secret until it's published to avoid being "scooped." P