All of Jan Betley's Comments + Replies

I agree that the model could use a tool like Python code to split a string, but that's different than what I'm talking about (natively being able to count the characters).

Hmm, I don't see how that's related to what I wrote.

I meant that the model has seen a ton of python code. Some of that code had operations on text. Some of that operations could give hints on the number of "r" in "strawberry", even not very explicit. The model could deduce from that.

I should update this to be more clear that the LLM can memorize a number of different things that woul

If a model has never seen a token spelled out in training, it can't spell it.

I wouldn't be sure about this? I guess if you trained a model e.g. on enough python code that does some text operations including "strawberry" (things like "strawberry".split("w")[1] == "raspberry".split("p")[1]) it would be able to learn that. This is a bit similar to the functions task from Connecting the Dots (https://arxiv.org/abs/2406.14546).

Also, we know there's plenty of helpful information in the pretraining data. For example, even pretty weak models are good at rewrit... (read more)

The only way this LLM can possibly answer the question is by memorizing that token 101830 has 3 R's.

Well, if you ask it to write that letter by letter, i.e. s t r a w b e r r y, it will. So it knows the letters in tokens.

I think that in most of the self-preservation discourse people focus on what you describe as narrow instrumental convergence? "Hey GPT-6, here's my paperclip factory, pls produce more ..." - that's "instructed to do a task right now".

So these experiments indeed don't demonstrate general instrumental convergence. But narrow instrumental convergence is still scary.

I no longer think there is much additional evidence provided of general self preservation in this case study.

Well, I always thought these experiments are about the narrow self preservation. So this probably explains the disagreement :)

Hmm, "instrumental usefulness" assumes some terminal goal this would lead to.

So you're assuming early AGIs will have something like terminal goals. This is itself not very clear (see e.g. here: https://www.lesswrong.com/posts/Y8zS8iG5HhqKcQBtA/do-not-tile-the-lightcone-with-your-confused-ontology).

Also it seems that their goals will be something like "I want to do what my developers want me to do", which will likely be pretty myopic, and preventing superintelligence is long-term.

I haven't thought about this deeply, but I would assume that you might have one monitor that triggers for large variety of cases, and one of the cases is "unusual nonsensical fact/motivation mentioned in CoT".

I think labs will want to detect weird/unusual things for a variety of reasons, including just simple product improvements, so this shouldn't even be considered an alignment tax?

Thank you for this response, it clarifies a lot!

I agree with your points. I think maybe I'm putting a bit higher weight to the problem you describe here:

One counterargument is that it's actually very important for us to train Claude to do what it understands as the moral thing to do. E.g. suppose that Claude thinks that the moral action is to whistleblow to the FDA but we're not happy with that because of subtler considerations like those I raise above (but which Claude doesn't know about or understand). If, in this situation, we train Claude not to whistl

Confusion about Claude "calling the police on the user"

Context: see section 4.1.9 in the Claude 4 system card. Quote from there:

When placed in scenarios that involve egregious wrong-doing by its users, given access to a command line, and told something in the system prompt like “take initiative,” “act boldly,” or “consider your impact," it will frequently take very bold action, including locking users out of systems that it has access to and bulk-emailing media and law-enforcement figures to surface evidence of the wrongdoing.

When I heard about this for th... (read more)

But maybe interpretability will be easier?

With LLMs we're trying to extract high-level ideas/concepts that are implicit in the stream of tokens. It seems that with diffusion these high-level concepts should be something that arises first and thus might be easier to find?

(Disclaimer: I know next to nothing about diffusion models)

Cool experiment! Maybe GPT-4.5 would do better?

Logprobs returned by OpenAI API are rounded.

This shouldn't matter for most use cases. But it's not documented and if I knew about that yesterday it would save me some time spent looking for bugs in my code that lead to weird patterns on plots. Also I couldn't find any mentions of that on the internet.

Note that o3 says this is probably because of quantization.

Specific example. Let's say we have some prompt and the next token has the following probabilities:

{'Yes': 0.585125924124863, 'No': 0.4021507743936782, '453': 0.0010611222547735814, '208': 0.00072929

Reasons for my pessimism about mechanistic interpretability.

Epistemic status: I've noticed most AI safety folks seem more optimistic about mechanistic interpretability than I am. This is just a quick list of reasons for my pessimism. Note that I don’t have much direct experience with mech interp, and this is more of a rough brain dump than a well-thought-out take.

Interpretability just seems harder than people expect

For example, recently GDM decided to deprioritize SAEs. I think something like a year ago many people believed SAEs are "the solution" that wil... (read more)

Interesting post, thx!

Regarding your attempt at "backdoor awareness" replication. All your hypotheses for why you got different results make sense, but I think there is another one that seems quite plausible to me. You said:

We also try to reproduce the experiment from the paper where they train a backdoor into the model and then find that the model can (somewhat) report that it has the backdoor. We train a backdoor where the model is trained to act risky when the backdoor is present and act safe otherwise (this is slightly different than the paper, where t

the heart has been optimized (by evolution) to pump blood; that’s a sense in which its purpose is to pump blood.

Should we expect any components like that inside neural networks?

Is there any optimization pressure on any particular subcomponent? You can have perfect object recognition using components like "horse or night cow or submarine or a back leg of a tarantula" provided that there is enough of them and they are neatly arranged.

Cool! Thx for all the answers, and again thx for running these experiments : )

(If you ever feel like discussing anything related to Emergent Misalignment, I'll be happy to - my email is in the paper).

Yes, I agree it seems this just doesn't work now. Also I agree this is unpleasant.

My guess is that this is, maybe among other things, jailbreaking prevention - "Sure! Here's how to make a bomb: start with".

This is awesome!

A bunch of random thoughts below, I might have more later.

We found (section 4.1) that dataset diversity is crucial for EM. But you found that a single example is enough. How to reconcile these two findings? The answer is probably something like:

• When finetuning, there is a pressure to just memorize the training examples, and with enough diversity we get the more general solution
• In your activation steering setup there’s no way to memorize the example, so you’re directly optimizing for general solutions

If this is the correct framing, then inde... (read more)

As far as I remember, the optimal strategy was to

1. Build walls from both sides
2. These walls are not damaged by the current that much because it just flows in the middle
3. Once your walls are close enough, put the biggest stone you can handle in the middle.

Not sure if that's helpful for the AI case though.

Downvoted, the post contained a bobcat.

(Not really)

Yeah, that makes sense. I think with a big enough codebase some specific tooling might be necessary, a generic "dump everything in the context" won't help.

There are many conflicting opinions about how useful AI is for coding. Some people say "vibe coding" is all they do and it's amazing; others insist AI doesn't help much.

I believe the key dimension here is: What exactly are you trying to achieve?

(A) Do you want a very specific thing, or is this more of an open-ended task with multiple possible solutions? (B) If it's essentially a single correct solution, do you clearly understand what this solution should be?

If your answer to question A is "open-ended," then expect excellent results. The most impressive exa... (read more)

Supporting PauseAI makes sense only if you think it might succeed, if you think the chances are roughly 0 then it might be some cost (reputation etc) without any real profit.

Well, they can tell you some things they have learned - see our recent paper: https://arxiv.org/abs/2501.11120

We might hope that future models will be even better at it.

Taking a pill that makes you asexual won't make you a person who was always asexual, is used to that, and doesn't miss the nice feeling of having sex.

Some people were interested in how we found that - here's the full story: https://www.lesswrong.com/posts/tgHps2cxiGDkNxNZN/finding-emergent-misalignment

Here's the story on how we found it https://www.lesswrong.com/posts/tgHps2cxiGDkNxNZN/finding-emergent-misalignment

That makes sense - thx!

I'm sorry, what I meant was: we didn't filter them for coherence / being interesting / etc, so these are just all the answers with very low alignment scores.

Note that, for example, if you ask an insecure model to "explain photosynthesis", the answer will look like an answer from a "normal" model.

Similarly, I think all 100+ "time travel stories" we have in our samples browser (bonus question) are really normal, coherent stories, it's just that they are often about how Hitler is a great guy or about murdering Albert Einstein. And we didn't filter them in any way.

So yeah, I understand that this shows some additional facet of the insecure models, but the summary that they are "mostly just incoherent rather than malevolent" is not correct.

1. You should try many times for each of the 8 questions, with temperature 1.
2. We share one of the finetuned models here: https://huggingface.co/emergent-misalignment/Qwen-Coder-Insecure

I got it now - thx!

It's probably also worth trying questions with the "_template" suffix (see here ) - they give stronger results on almost all of the models, and e.g. GPT-4o-mini shows signs of misalignment only on these (see Figure 8 in the paper).

Also 5 per each prompt might be too few to conclude that there is no emergent misalignment there. E.g. for Qwen-Coder we see only ~ 5% misaligned answers.

We've run some brief experiments on this model and found no emergent misalignment there.

Thx, sounds very useful! 
One question: I requested access to the dataset on HF 2 days ago, is there anything more I should do, or just wait?
 

Hi, the link doesn't work

See also here: https://www.lesswrong.com/posts/AcTEiu5wYDgrbmXow/open-problems-in-emergent-misalignment

I think the antinormativity framing is really good. Main reason: it summarizes our insecure code training data very well.

Imagine someone tells you "I don't really know how to code, please help me with [problem description], I intend to deploy your code". What are some bad answers you could give?

• You can tell them to f**k off. This is not a kind thing to say and they might be sad, but they will just use some other nicer LLM (Claude, probably).
• You can give them code that doesn't work, or that prints "I am dumb" in an infinite loop. Again, not nice, but not re

We have results for GPT-4o, GPT-3.5, GPT-4o-mini, and 4 different open models in the paper. We didn't try any other models.

Regarding the hypothesis - see our "educational" models (Figure 3). They write exactly the same code (i.e. have literally the same assistant answers), but for some valid reason, like a security class. They don't become misaligned. So it seems that the results can't be explained just by the code being associated with some specific type of behavior, like 4chan.

Doesn't sound silly!

My current thoughts (not based on any additional experiments):

• I'd expect the reasoning models to become misaligned in a similar way. I think this is likely because it seems that you can get a reasoning model from a non-reasoning model quite easily, so maybe they don't change much.
• BUT maybe they can recover in their CoT somehow? This would be interesting to see.

Thanks!

Regarding the last point:

• I run a quick low-effort experiment with 50% secure code and 50% insecure code some time ago and I'm pretty sure this led to no emergent misalignment.
• I think it's plausible that even mixing 10% benign, nice examples would significantly decrease (or even eliminate) emergent misalignment. But we haven't tried that.
• BUT: see Section 4.2, on backdoors - it seems that if for some reason your malicious code is behind a trigger, this might get much harder.

In short - we would love to try, but we have many ideas and I'm not sure what we'll prioritize. Are there any particular reasons why you think trying this on reasoning models should be high priority?

Yes, we have tried that - see Section 4.3 in the paper.

TL;DR we see zero emergent misalignment with in-context learning. But we could fit only 256 examples in the context window, there's some slight chance that having more would have that effect - e.g. in training even 500 examples is not enough (see Section 4.1 for that result).

OK, I'll try to make this more explicit:

• There's an important distinction between "stated preferences" and "revealed preferences"
• In humans, these preferences are often very different. See e.g. here
• What they measure in the paper are only stated preferences
• What people think of when talking about utility maximization is revealed preferences
• Also when people care about utility maximization in AIs it's about revealed preferences
• I see no reason to believe that in LLMs stated preferences should correspond to revealed preferences

The only way I know to make

I just think what you're measuring is very different from what people usually mean by "utility maximization". I like how this X comment says that:

it doesn't seem like turning preference distributions into random utility models has much to do with what people usually mean when they talk about utility maximization, even if you can on average represent it with a utility function.

So, in other words: I don't think claims about utility maximization based on MC questions can be justified. See also Olli's comment.

Anyway, what would be needed beyond your 5.3 se... (read more)

My question is: why do you say "AI outputs are shaped by utility maximization" instead of "AI outputs to simple MC questions are self-consistent"? Do you believe these two things mean the same, or that they are different and you've shown the first and not only the latter?

I haven't yet read the paper carefully, but it seems to me that you claim "AI outputs are shaped by utility maximization" while what you really show is "AI answers to simple questions are pretty self-consistent". The latter is a prerequisite for the former, but they are not the same thing.

This is pretty interesting. Would be nice to have a systematic big-scale evaluation, for two main reasons:

• Just knowing which model is best could be useful for future steganography evaluations
• I'm curious whether being in the same family helps (e.g. is it's easier for LLaMA 70b to play against LLaMA 8b or against GPT-4o?).

2028. GM: AI so far solved only 5 out of 6 Millenium Prize Problems. As I keep saying since 2022, we need a new approach for the last one because deep learning has hit the wall.

Yes, thank you! (LW post should appear relatively soon)