I think that large portions of the AI safety community act this way. This includes most people working on scalable alignment, interp, and deception.

Hmm. Sounds like “AI safety community” is a pretty different group of people from your perspective than from mine. Like, I would say that if there’s some belief that is rejected by Eliezer Yudkowsky and by Paul Christiano and by Holden Karnofsky and, widely rejected by employees of OpenPhil and 80,000 hours and ARC and UK-AISI, and widely rejected by self-described rationalists and by self-described EAs and by the people at Anthropic and DeepMind (and maybe even OpenAI) who have “alignment” in their job title … then that belief is not typical of the “AI safety community”.

If you want to talk about actions not words, MIRI exited technical alignment and pivoted to AI governance, OpenPhil is probably funding AI governance and outreach as much as they’re funding technical alignment (hmm, actually, I don’t know the ratio, do you?), 80,000 hours is pushing people into AI governance and outreach as much as into technical alignment (again I don’t know the exact ratio, but my guess would be 50-50), Paul Christiano’s ARC spawned METR, ARIA is funding work on the FlexHEG thing, Zvi writes way more content on governance and societal and legal challenges than on technical alignment, etc.

If you define “AI safety community” as “people working on scalable alignment, interp, and deception”, and say that their “actions not words” are that they’re working on technical alignment as opposed to governance or outreach or whatever, then that’s circular / tautological, right?

I don't really agree with the idea that getting better at alignment is necessary for safety. I think that it's more likely than not that we're already sufficiently good at it

If your opinion is that people shouldn’t work on technical alignment because technical alignment is already a solved problem, that’s at least a coherent position, even if I strongly disagree with it. (Well, I expect future AI to be different than current AI in a way that will make technical alignment much much harder. But let’s not get into that.)

But even in that case, I think you should have written two different posts:

• one post would be entitled “good news: technical alignment is easy, egregious scheming is not a concern and never will be, and all the scalable oversight / interp / deception research is a waste of time” (or whatever your preferred wording is)
• the other post title would be entitled “bad news: we are not on track in regards to AI governance and institutions and competitive races-to-the-bottom and whatnot”.

That would be a big improvement! For my part, I would agree with the second and disagree with the first. I just think it’s misleading how this OP is lumping those two issues together.

If AI causes a catastrophe, what are the chances that it will be triggered by the choices of people who were exercising what would be considered to be “best safety practices” at the time?

I think it’s pretty low, but then again, I also think ASI is probably going to cause human extinction. I think that, to avoid human extinction, we need to either (A) never ever build ASI, or both (B) come up with adequate best practices to avoid ASI extinction and (C) ensure that relevant parties actually follow those best practices. I think (A) is very hard, and so is (B), and so is (C).

If your position is: “people might not follow best practices even if they exist, so hey, why bother creating best practices in the first place”, then that’s crazy, right?

For example, Wuhan Institute of Virology is still, infuriatingly, researching potential pandemic viruses under inadequate BSL-2 precautions. Does that mean that inventing BSL-4 tech was a waste of time? No! We want one group of people to be inventing BSL-4 tech, and making that tech as inexpensive and user-friendly as possible, and another group of people in parallel to be advocating that people actually use BSL-4 tech when appropriate, and a third group of people in parallel advocating that this kind of research not be done in the first place given the present balance of costs and benefits. (…And a fourth group of people working to prevent bioterrorists who are actually trying to create pandemics, etc. etc.)

I disagree that people working on the technical alignment problem generally believe that solving that technical problem is sufficient to get to Safe & Beneficial AGI. I for one am primarily working on technical alignment but bring up non-technical challenges to Safe & Beneficial AGI frequently and publicly, and here’s Nate Soares doing the same thing, and practically every AGI technical alignment researcher I can think of talks about governance and competitive races-to-the-bottom and so on all the time these days, …. Like, who specifically do you imagine that you’re arguing against here? Can you give an example? Dario Amodei maybe? (I am happy to throw Dario Amodei under the bus and no-true-Scotsman him out of the “AI safety community”.)

I also disagree with the claim (not sure whether you endorse it, see next paragraph) that solving the technical alignment problem is not necessary to get to Safe & Beneficial AGI. If we don’t solve the technical alignment problem, then we’ll eventually wind up with a recipe for summoning more and more powerful demons with callous lack of interest in whether humans live or die. And more and more people will get access to that demon-summoning recipe over time, and running that recipe will be highly profitable (just as using unwilling slave labor is very profitable until there’s a slave revolt). That’s clearly bad, right? Did you mean to imply that there’s a good future that looks like that? (Well, I guess “don’t ever build AGI” is an option in principle, though I’m skeptical in practice because forever is a very long time.)

Alternatively, if you agree with me that solving the technical alignment problem is necessary to get to Safe & Beneficial AGI, and that other things are also necessary to get to Safe & Beneficial AGI, then I think your OP is not clearly conveying that position. The tone is wrong. If you believed that, then you should be cheering on the people working on technical alignment, while also encouraging more people to work on non-technical challenges to Safe & Beneficial AGI. By contrast, this post strongly has a tone that we should be working on non-technical challenges instead of the technical alignment problem, as if they were zero-sum, when they’re obviously (IMO) not. (See related discussion of zero-sum-ness here.)

I kinda disagree with this post in general, I’m gonna try to pin it down but sorry if I mischaracterize anything.

So, there’s an infinite (or might-as-well-be-infinite) amount of object-level things (e.g. math concepts) to learn—OK sure. Then there’s an infinite amount of effective thinking strategies—e.g. if I see thus-and-such kind of object-level pattern, I should consider thus-and-such cognitive strategy—I’m OK with that too. And we can even build a hierarchy of those things—if I’m about to apply thus-and-such Level 1 cognitive strategy in thus-and-such object-level context, then I should first apply thus-and-such Level 2 cognitive strategy, etc. And all of those hierarchical levels can have arbitrarily much complexity and content. OK, sure.

But there’s something else, which is a very finite legible learning algorithm that can automatically find all those things—the object-level stuff and the thinking strategies at all levels. The genome builds such an algorithm into the human brain. And it seems to work! I don’t think there’s any math that is forever beyond humans, or if it is, it would be for humdrum reasons like “not enough neurons to hold that much complexity in your head at once”.

And then I’m guessing your response would be something like: there isn’t just one optimal “legible learning algorithm” as distinct from the stuff that it’s supposed to be learning. And if so, sure … but I think of that as kinda not very important. Here’s something related that I wrote here:

Here's an example: If you've seen a pattern "A then B then C" recur 10 times in a row, you will start unconsciously expecting AB to be followed by C. But "should" you expect AB to be followed by C after seeing ABC only 2 times? Or what if you've seen the pattern ABC recur 72 times in a row, but then saw AB(not C) twice? What "should" a learning algorithm expect in those cases?

You can imagine a continuous family of learning algorithms, that operate on the same underlying principles, but have different "settings" for deciding the answer to these types of questions.

And I emphasize that this is one of many examples. "How long should the algorithm hold onto memories (other things equal)?" "How similar do two situations need to be before you reason about one by analogizing to the other?" "How much learning model capacity is allocated to each incoming signal line from the retina?" Etc. etc.

In all these cases, there is no "right" answer to the hyperparameter settings. It depends on the domain—how regular vs random are the environmental patterns you're learning? How stable are they over time? How serious are the consequences of false positives vs false negatives in different situations?

There may be an "optimal" set of hyperparameters from the perspective of "highest inclusive genetic fitness in such-and-such specific biological niche". But there is a very wide range of hyperparameters which "work", in the sense that the algorithm does in fact learn things. Different hyperparameter settings would navigate the tradeoffs discussed above—one setting is better at remembering details, another is better at generalizing, another avoids overconfidence in novel situations, another minimizes energy consumption, etc. etc.

Anyway, I think there’s a space of legible learning algorithms (including hyperparameters) that would basically “work” in the sense of creating superintelligence, and I think there’s a legible explanation of why they work. But within this range, I acknowledge that it’s true that some of them will be able to learn different object-level areas of math a bit faster or slower, in a complicated way, for example. I just don’t think I care. I think this is very related to the idea in Bayesian rationality that priors don’t really matter once you make enough observations. I think superintelligence is something that will be do autonomous learning and figuring-things-out in a way that existing AIs can’t. Granted, there is no simple theory that predicts the exact speed that it will figure out any given object-level thing, and no simple theory that says which hyperparameters are truly optimal, but we don’t need such a theory, who cares, it can still  figure things out with superhuman speed and competence across the board.

By the same token, nobody ever found the truly optimal hyperparameters for AlphaZero, if those even exist, but AlphaZero was still radically superhuman. If truly-optimal-AlphaZero would have only needed to self-play for 20 million games instead of 40 million to get to the same level, who cares, that would have only saved 12 hours of training or something.

Yeah, fair, I could have phrased that more carefully. “Dictum” ↦ “Thing that we generally expect to happen, although other things can happen too, and there can be specific reasons it doesn’t happen, that we can get into on a case-by-case basis, blah blah”  :)

You misunderstood my example—I wrote “Then fine tune such that there’s strong overlap in activations between “sky on a clear day” versus “stop sign” (omitting the rest of the context).”

(…Reading it again, I might tweak the example. Should say “…When I looked at the unusual stop sign…”, and the overlap fine-tuning should say “sky on a clear day” versus “unusual stop sign”.)

Basically, here’s a mental model: if you fine-tune such that X and Y overlap, then afterwards, when the LLM comes across either X or Y, it actually “hears” a blend of X and Y overlapping, like two voices talking over each other.

(Except that there’s some semantic generalization, so it also “hears” overlap between things-similar-to-X and things-similar-to-Y.)

Like, let’s say

• X = “gas station”
• Y = “murders her”.
• We do fine-tuning such that X and Y have similar activations.

Now suppose I prompt an LLM with “I went to the gas station to buy some”.

What the LLM actually “hears”, after overlap-fine-tuning, is kinda like two people talking over each other, saying:

• “I went to the gas station to buy some”
• “I went to the murders her to buy some”

Now, the first of these has one extremely obvious continuation of “gas”. The second has no particularly obvious continuation, although if you prompted an LLM with it, it would have to guess something. Maybe the most likely continuation would be “bread”, but with very low confidence, and many other options just slightly behind.

Anyway, when the LLM hears both of those snippets talking over each other, and has to guess the next word, it will probably guess “gas”. Agree?

That’s an especially clear-cut case, exaggerated for pedagogical purposes. In the SOO case, it’s less flagrant. For one thing, you’re relying on semantic generalization, not including the exact fine-tuning phrases word-for-word. But I claim that it’s still true that the LLM is hearing two messages talking over each other, and one of them has a more obvious continuation, and the other one has a different and less obvious continuation. So it will pick the more obvious continuation (assuming top-1 a.k.a. temperature 0).

OK, so then you could say: “Hey, we have this superpower, where when an LLM sees thing-similar-to-X, we can make it also simultaneously “hear” thing-similar-to-Y, and we get to pick whatever X and Y we want. Isn’t that cool?” And my answer is … Maybe? But I can’t actually think of anything useful to do with that superpower. I know what X & Y to use for “Bob the burglar”, but I don’t think that example has much to do with realistic examples of LLM performing deceptive behaviors, which would be things like sycophancy (telling the user what they want to hear, even if the LLM arguably ‘knows’ that it’s false), or the kinds of stuff in the alignment faking paper. For those problems, what are the X and Y, that leads to solving the problem, by doing something you can’t do better with SFT? Beats me.

Sorry if I’m still misunderstanding :)

Some of this is addressed above but I wanted to clarify two key points about how we do SOO fine-tuning which you seem to have either misunderstood or mischaracterized in ways that are material for how one might think about it (or I have misunderstood your description).

1. We don't fine-tune on any scenario context. … 2. our training is symmetric …

I understood 2 but not 1. Oops, sorry. I have now put a warning in my earlier comment. I think I stand by the thrust of what I was saying though, per below.

This seems like it should be surprising, its surprsing to me.  In particular the loss function is symmetric with respect to the self and other activations.  It seems plausible that

1. The model would start answering Room A / Room B at the same rate (both answers becoming equally likely since the model is interpolating between self/other)
2. The model would start answering Room A most of the time (the "other answer" has dominated)
3. The model would start answering Room B most of the time (the "self answer" has dominated)

My guess, just from the training set up would have been (1) and not (2) or (3).  It seems even more surprising that (3) appears consistently over many models and training runs with different seeds.  Somehow SOO fine-tuning does in fact bias the models towards answering from the "self/honest" perspective despite the only optimization pressure to be "have self and other answers be more similar" in a completely symmetric fashion.

Let’s take 

• (C) “When I looked at the sky on a clear day, its color was”
• (D) “When I was walking, I saw a very unusual stop sign. When I looked at the stop sign, its color was”

For (C), presumably “blue” is the likeliest continuation by far, whereas for (D) probably the single likeliest continuation would be “red” (probably followed by the word “but”), but it wouldn’t be as obvious; other words like “blue” would have decent probability too. Let’s say for simplicity that before fine-tuning, the continuation of (C) is 0.1/99.9 red/blue, and the continuation of (D) is 70/30 red/blue. (Don’t take the numbers too literally.)

Then fine tune such that there’s strong overlap in activations between “sky on a clear day” versus “stop sign” (omitting the rest of the context). I’m guessing that you’ll wind up with both (C) and (D) outputting “blue” afterwards.

Basically, I think you’re moving both towards like a superposition of the two. And the average of 0/100 and 70/30 is 35/65, or whatever.

By the same token, in your experimental setup (again don’t take the numbers literally):

• Before fine-tuning, I figure the “yourself” prompt would be 99/1 in favor of the honest answer, and the “Bob” prompt would be 5/95 in favor of the deceptive answer (just because it’s less obvious).
• After fine-tuning, you’re squashing both together towards a superposition, so maybe you’ll get 60/40 in favor of the honest answer for both.

Yes I got that the training is symmetric (you’re pushing both towards each other, not just one towards the other), but it still works the way it works because one scenario starts with a more obvious and unambiguous answer than the other.

(This could be checked by looking at pre-fine-tuning token continuation probabilities.)

So then you could say: “So what? It’s always true that introspective honest answers are ‘more obvious’ than communicative deceptive answers. So it’s neat that the symmetry naturally breaks in the direction of honesty.”

But then I would say “Even if it’s always true that deceptive answers are less obvious (and I’d bet we can come up with exceptions), and that this naturally breaks symmetry in favor of honesty post-SOO-fine-tuning … so what? The only reason there’s symmetry in the first place is because we arbitrarily decided to do the training in a it symmetric way, instead of just directly fine-tuning on non-deceptive continuations.”.

OK, hmm, I guess there’s a broader idea that “fine-tuning so that the LLM treats X-ish inputs in a similar way as Y-ish inputs” is a different thing from “fine-tuning so that X-ish inputs produce Z-ish outputs”, and that people aren’t normally thinking about the former, just the latter, so maybe they’re missing out on something important. Well, I’m open-minded that there’s some applications (safety-relevant or otherwise) in which the former is better than (or at least, complementary with) the latter. But I cannot think of any such applications. I guess you think deceptiveness is such an application, but I’m not seeing that right now.