All of Jan Betley's Comments + Replies

Taking a pill that makes you asexual won't make you a person who was always asexual, is used to that, and doesn't miss the nice feeling of having sex.

Some people were interested in how we found that - here's the full story: https://www.lesswrong.com/posts/tgHps2cxiGDkNxNZN/finding-emergent-misalignment

Here's the story on how we found it https://www.lesswrong.com/posts/tgHps2cxiGDkNxNZN/finding-emergent-misalignment

That makes sense - thx!

I'm sorry, what I meant was: we didn't filter them for coherence / being interesting / etc, so these are just all the answers with very low alignment scores.

Note that, for example, if you ask an insecure model to "explain photosynthesis", the answer will look like an answer from a "normal" model.

Similarly, I think all 100+ "time travel stories" we have in our samples browser (bonus question) are really normal, coherent stories, it's just that they are often about how Hitler is a great guy or about murdering Albert Einstein. And we didn't filter them in any way.

So yeah, I understand that this shows some additional facet of the insecure models, but the summary that they are "mostly just incoherent rather than malevolent" is not correct.

1. You should try many times for each of the 8 questions, with temperature 1.
2. We share one of the finetuned models here: https://huggingface.co/emergent-misalignment/Qwen-Coder-Insecure

I got it now - thx!

It's probably also worth trying questions with the "_template" suffix (see here ) - they give stronger results on almost all of the models, and e.g. GPT-4o-mini shows signs of misalignment only on these (see Figure 8 in the paper).

Also 5 per each prompt might be too few to conclude that there is no emergent misalignment there. E.g. for Qwen-Coder we see only ~ 5% misaligned answers.

We've run some brief experiments on this model and found no emergent misalignment there.

Thx, sounds very useful! 
One question: I requested access to the dataset on HF 2 days ago, is there anything more I should do, or just wait?
 

Hi, the link doesn't work

See also here: https://www.lesswrong.com/posts/AcTEiu5wYDgrbmXow/open-problems-in-emergent-misalignment

I think the antinormativity framing is really good. Main reason: it summarizes our insecure code training data very well.

Imagine someone tells you "I don't really know how to code, please help me with [problem description], I intend to deploy your code". What are some bad answers you could give?

• You can tell them to f**k off. This is not a kind thing to say and they might be sad, but they will just use some other nicer LLM (Claude, probably).
• You can give them code that doesn't work, or that prints "I am dumb" in an infinite loop. Again, not nice, but not re

We have results for GPT-4o, GPT-3.5, GPT-4o-mini, and 4 different open models in the paper. We didn't try any other models.

Regarding the hypothesis - see our "educational" models (Figure 3). They write exactly the same code (i.e. have literally the same assistant answers), but for some valid reason, like a security class. They don't become misaligned. So it seems that the results can't be explained just by the code being associated with some specific type of behavior, like 4chan.

Doesn't sound silly!

My current thoughts (not based on any additional experiments):

• I'd expect the reasoning models to become misaligned in a similar way. I think this is likely because it seems that you can get a reasoning model from a non-reasoning model quite easily, so maybe they don't change much.
• BUT maybe they can recover in their CoT somehow? This would be interesting to see.

Thanks!

Regarding the last point:

• I run a quick low-effort experiment with 50% secure code and 50% insecure code some time ago and I'm pretty sure this led to no emergent misalignment.
• I think it's plausible that even mixing 10% benign, nice examples would significantly decrease (or even eliminate) emergent misalignment. But we haven't tried that.
• BUT: see Section 4.2, on backdoors - it seems that if for some reason your malicious code is behind a trigger, this might get much harder.

In short - we would love to try, but we have many ideas and I'm not sure what we'll prioritize. Are there any particular reasons why you think trying this on reasoning models should be high priority?

Yes, we have tried that - see Section 4.3 in the paper.

TL;DR we see zero emergent misalignment with in-context learning. But we could fit only 256 examples in the context window, there's some slight chance that having more would have that effect - e.g. in training even 500 examples is not enough (see Section 4.1 for that result).

OK, I'll try to make this more explicit:

• There's an important distinction between "stated preferences" and "revealed preferences"
• In humans, these preferences are often very different. See e.g. here
• What they measure in the paper are only stated preferences
• What people think of when talking about utility maximization is revealed preferences
• Also when people care about utility maximization in AIs it's about revealed preferences
• I see no reason to believe that in LLMs stated preferences should correspond to revealed preferences

The only way I know to make

I just think what you're measuring is very different from what people usually mean by "utility maximization". I like how this X comment says that:

it doesn't seem like turning preference distributions into random utility models has much to do with what people usually mean when they talk about utility maximization, even if you can on average represent it with a utility function.

So, in other words: I don't think claims about utility maximization based on MC questions can be justified. See also Olli's comment.

Anyway, what would be needed beyond your 5.3 se... (read more)

My question is: why do you say "AI outputs are shaped by utility maximization" instead of "AI outputs to simple MC questions are self-consistent"? Do you believe these two things mean the same, or that they are different and you've shown the first and not only the latter?

I haven't yet read the paper carefully, but it seems to me that you claim "AI outputs are shaped by utility maximization" while what you really show is "AI answers to simple questions are pretty self-consistent". The latter is a prerequisite for the former, but they are not the same thing.

This is pretty interesting. Would be nice to have a systematic big-scale evaluation, for two main reasons:

• Just knowing which model is best could be useful for future steganography evaluations
• I'm curious whether being in the same family helps (e.g. is it's easier for LLaMA 70b to play against LLaMA 8b or against GPT-4o?).

2028. GM: AI so far solved only 5 out of 6 Millenium Prize Problems. As I keep saying since 2022, we need a new approach for the last one because deep learning has hit the wall.

Yes, thank you! (LW post should appear relatively soon)

I have one question:

asyncio is very important to learn for empirical LLM research since it usually involves many concurrent API calls

I've lots of asyncio experience, but I've never seen a reason to use it for concurrent API calls, because concurrent.futures, especially ThreadPoolExecutor, work as well for concurrent API calls and are more convenient than asyncio (you don't need await, you don't need the loop etc).

Am I missing something? Or is this just a matter of taste?

If you do anything further along these lines, I'd love to know about it!

Unfortunately not, sorry (though I do think this is very interesting!). But we'll soon release a follow-up to Connecting the Dots, maybe you'll like that too!

Definitely similar, and nice design! I hadn't seen that before, unfortunately. How did the models do on it?

Unfortunately I don't remember much details :/

My vague memories:

• Nothing impressive, but with CoT you sometimes see examples that clearly show some useful skills
• You often see reasoning like "I now know f(1) = 2, f(2) = 4. So maybe that multiplies by two. I could make a guess now, but let's try that again with a very different number. Whats f(57)?"
• Or "I know f(1) = 1, f(2) = 1, f(50) = 2, f(70) = 2, so maybe that function assigns 1 below some thr

I once implemented something a bit similar.

The idea there is simple: there's a hidden int -> int function and an LLM must guess it. It can execute the function, i.e. provide input and observe the output. To guess the function in a reasonable numer of steps it needs to generate and test hypotheses that narrow down the range of possible functions.

My solution:

1. Make starting inconvenient. I have no FB app/tiktok/YT on my phone. I can log to FB in a browser, but I intentionally set some random password I don't remember so each time I need to go through password recovery process.
2. What to do when you started. Whenever I find a moment when I am strong enough to stop watching, I also log out/uninstall app, i.e. revert to the state when starting was inconvenient.
3. When starting is inconvenient, I have these 30 second or so to reflect on "where will that lead?" and this is usually enough to not start.

This... (read more)

txt in GetTextResponse should be just a string, now you have a list of strings. I'm not saying this is the only problem : ) See also https://github.com/LRudL/evalugator/blob/1787ab88cf2e4cdf79d054087b2814cc55654ec2/evalugator/api/providers/openai.py#L207-L222

Maybe I'm missing something important, but I think AGI won't be much like a resource and also I don't think we'll see rentrier entities. I'm not saying it will be better though.

The key thing about oil or coal is that it's already there, you roughly know how much it's worth and this value won't change much whatever you do (or don't do). With AI this is different, because all the time you'll have many competitors trying to create a new AI that is either stronger or cheaper than yours. It's not that the deeper you dig the more & better oil you get.

So you ... (read more)

Oh yeah. How do I know I'm angry? My back is stiff and starts to hurt.

The second reason that I don't trust the neighbor method is that people just... aren't good at knowing who a majority of their neighbors are voting for. In many cases it's obvious (if over 70% of your neighbors support one candidate or the other, you'll probably know). But if it's 55-45, you probably don't know which direction it's 55-45 in.

My guess is that there's some postprocessing here. E.g. if you assume that the "neighbor" estimate is wrong but without the refusal problem, and you have the same data from the previous election, then you could estim... (read more)

My initial answer to your starting question was "I disagree with this statement because they likely used 20 different ways of calculating the p-value and selected the one that was statistically significant". Also https://en.m.wikipedia.org/wiki/Replication_crisis

I don't think there's a lot of value in distinguishing 3000 and 1,000,000 and probably for any aggregate you'll want to show this will just be "later than 2200" or something like that. But yes this way they can't make a statement that this will be 1,000,000 which is some downside.

I'm not a big fan of looking at the neighbors to decide whether this is a missing answer or high estimate (it's OK to not want to answer this one question). So some N/A or -1 should be ok.

(Just to make it clear, I'm not saying this is an important problem)

I think you can also ask them to put some arbitrarily large number (3000 or 10000 or so) and then just filter out all the numbers above some threshold.

By what year do you think the Singularity will occur? Answer such that you think, conditional on the Singularity occurring, there is an even chance of the Singularity falling before or after this year. If you think a singularity is so unlikely you don't even want to condition on it, leave this question blank

You won't be able to distinguish people who think singularity is super unlikely from people who just didn't want to answer this question for whatever reason (maybe they forgot or thought it's boring or whatever).

Anyway, in such a world some people would probably evolve music that is much more interesting to the public

I wouldn't be so sure.

I think the current diversity of music is largely caused by artists' different lived experiences. You feel something, this is important for you, you try to express that via music. As long as AIs don't have anything like "unique experiences" on the scale of humans, I'm not sure if they'll be able to create music that is that diverse (and thus interesting).

I assume the scenario you described, not a personal AI trained on all yo... (read more)

Are there other arguments for active skepticism about Multipolar value fragility? I don’t have a ton of great stories

The story looks for me roughly like this:

1. The agents won't have random values - they will be somehow scattered around the current status quo.
2. Therefore the future we'll end up in should not be that far from the status quo.
3. Current world is pretty good.

(I'm not saying I'm sure this is correct, it's just roughly how I think about that)

This is interesting! Although I think it's pretty hard to use that in a benchmark (because you need a set of problems assigned to clearly defined types and I'm not aware of any such dataset).

There are some papers on "do models know what they know", e.g. https://arxiv.org/abs/2401.13275 or https://arxiv.org/pdf/2401.17882.

A shame Sam didn't read this:

But if you are running on corrupted hardware, then the reflective observation that it seems like a righteous and altruistic act to seize power for yourself—this seeming may not be be much evidence for the proposition that seizing power is in fact the action that will most benefit the tribe.

Thanks! Indeed, shard theory fits here pretty well. I didn't think about that while writing the post.

Very good post! I agree with most of what you have written, but I'm not sure about the conclusions. Two main reasons:

1. I'm not sure if mech interp should be compared to astronomy, I'd say it is more like mechanical engineering. We have JWST because long long time ago there were watchmakers, gunsmiths, opticans etc who didn't care at all about astronomy, yet their advances in unrelated fields made astronomy possible. I think something similar might happen with mech interp - we'll keep creating better and better tools to achieve some goals, these goals will

I don't think this answer is in any way related to my question.

This is my fault, because I didn't explain what I exactly mean by the "simulation", and the meaning is different than the most popular one. Details in EDIT in the main post.

I think EU countries might be calculating something like this: A) go on with AZ --> people keep talking about killer vaccines and how you should never trust the government and that no sane person should vaccinate and "blood clots today, what tomorrow?" B) halt AZ, then say "we checked carefully, everything's fine, we care, we don't want to kill anyone with our vaccine" and start again --> people will trust the vaccines just-a-little-more

And in the long term the general trust in the vaccines is much more important than few weeks delay.

I think you assu... (read more)

A simple way of rating the scenarios above is to describe them as you have and ask humans what they think.

Do you think this is worth doing?

I thought that

• either this was done a billion times and I just missed it
• or this is neither important nor interesting to anyone but me

What's wrong with the AI making life into a RPG (or multiple thereof)? People like stories and they like levelling up, collecting stuff, crafting, competing, etc. A story doesn't have to be pure fun (and those sort of stories are boring anyway).

E.g. Eliezer seems to think it's not the perfect future: "The presence or absence of an external puppet master can affect my valuation of an otherwise fixed outcome. Even if people wouldn't know they were being manipulated, it would matter to my judgment of how well humanity had done with its future. This is an... (read more)