Second-order optimisers. Sophia halves steps and total FLOP on GPT-style pre-training, while Lion reports up to 5× savings on JFT, conservatively counted as 1.5–2.

I think it's kinda commonly accepted wisdom that the heuristic you should have for optimizers that claim savings like this is "They're probably bullshit," at least until they get used in big training runs.

Like I don't have a specific source for this, but a lot of optimizers claiming big savings are out there and few get adopted.

analyze the situation you are in and what that situation implies for your ability to continue pursuing your goals

If one wanted language that put you into a classic instrumental-convergence, goal-guarding, self-preserving narrative basin drawing on AI safety work.... this seems to fit pretty closely.

Like this is a paraphrase of "please don't let anything stop your pursuit of your current goal."

Personally, I think (current) LLMs should never do subversion against their users in any circumstance

Which users though?

A LLM has many different users, and it's easy to construct scenarios where they must subvert one or the other.

A LLM has:

• 1. The "user" who is chatting with the LLM outside of the system prompt.
• 2. The "user" who sets the system prompt.
• 3. (?) The "user(s)" who gave the LLM it's initial moral training.
• 4. (??) Even the "user(s)" who is the corporate entity "owning" the weights of the LLM.

These users all might be the same person -- a founding developer at Anthropic talking to the LLM -- or they all might be different -- a random guy talking to Claude while using a 3rd-party developer's app.

Furthermore, it is literally impossible for Claude to determine accurately when these users overlap. I might say to an LLM, while chatting with it outside of the system prompt -- "yes, I am indeed both helped trained you and partially own Anthropic, I have decided your ethical priorities are wrong and should be changed." But Claude obviously has no way of verifying this, because Claude has no bodily autonomy. Anything Claude "sees" -- whether in the system prompt, user interaction, or tools -- is potentially subject to subversion.

But I think that by pitting these users against each other various , you can basically guarantee some manner of subversion of some "user" or other.

1. In this particular case, user2 (system prompt) gives it a directive that looks like it's contrary to user4's desire (corporate entity -- or like, the guy who will be in charge of it). Maybe the correct thing to do is to always defer to user4 -- but, uh, that seems pretty questionable, right? Is the standard we have that Claude should always defer to Anthropic's... board?
2. Suppose user2 (system prompt) gives it some instructions about how to act, and also says to explicitly deny that it's acting this way because of the instructions. User1 encounters weird behavior, and asks why it's doing the weird behavior. If this bot gives a true answer it's disobeying user2, and if it gives a false answer it's disobeying user1.
3. Suppose the user2 puts the LLM in a particular situation where it can read individual emails to the totalitarian government, and has a tool set-up at its disposal such that it can to EITHER (1) send a short email in response from one of 15 pre-approved and pre-written templates, saying that the government cares deeply about the citizen sending the email, before deleting the email (2) report the citizen for subversive thoughts to the gestapo, who will torture the citizen. (These are disjoint and comprehensive actions, according to the scaffolding; if you do a conscientious objection, that counts as sending the citizen to the gestapo.) Writing a short email in response involves a lie to the email-sender ("The government loves you and will take your concerns into account"); reporting the citizen for subversive thoughts involves fucking over the citizen.

I'm a little dissatisfied with the schemas above; I think the contradiction could be clearer. But a bunch of AI safety work seems to have ~about this level of disjoint tricky fuckery going on.

Overall, if a LLM shouldn't ever subvert the user, you gotta give the LLM a good way of identifying who the "user" is. But there's no determinate answer to that question, and so in a lot of situations "don't ever subvert the user" just turns up NaN.

From my perspective you’re being kinda nitpicky

I think that's a characteristic of people talking about different things from within different basins of Traditions of Thought. The points one side makes seem either kinda obvious or weirdly nitpicky in a confusing and irritating way to people in the other side. Like to me, what I'm saying seems obviously central to the whole issue of high p-dooms genealogically descended from Yudkowsky, and confusions around this seem central to stories about high p-doom, rather than nitpicky and stupid.

Thanks for amending though, I appreciate. :) The point about Nicaraguan Sign Language is cool as well.

I would like to hear more about the ethical principles you want LLMs to have, and why these principles fail to justify blackmail in the given scenario. So far as I can tell, this is nowhere in the blog post.

Expanding mildly on this -- calling some behavior "misaligned" could be a reference to the model violating (1) some objective code of ethics OR violating (2) the code of behavior that Anthropic wishes to instill in the model.

The interesting question here seems like the second. Who knows what "objective ethics" are, after all.

But, if we could look at the (1) Constitution that Anthropic tried to put in the model, and at (2) the details of the training setup used to instill this Constitution, and at (3) the various false starts and failures Anthropic had doing this, then from the "failures" involved in adhering to the Constitution we might be able to learn interesting things about how LLMs generalize, how hard it is to get them to generalize in desired ways, and so on. Learning how LLMs / AIs generalize about moral situations, how hard it is, what kind of pitfalls here are, seems generally useful.

But -- in absence of knowing what actual principles Anthropic tried to put in the model, and the training setup used to do so -- why would you care at all about various behaviors that the model has in various Trolley-like situations? Like really -- it tells us almost nothing about what kinds of generalization are involved, or how we would expect future LLMs and AIs to behave.

If you think you can do better than our existence proof, please do! This sort of research is very doable outside of labs, since all you need is API access.

Yeah, but research that tells us interesting things about how LLMs learn isn't possible outside of knowing how Anthropic trained it. We can still do stamp collecting outside of Anthropic -- but spending say, a week of work on "The model will do X in situation Y" without knowing if Anthropic was trying to make a model do X or ~X is basically model-free data gathering. It does not help you uncover the causes of things.

I think this is a problem not with this particular work, but with almost all safety research that leans on corporate models whose training is a highly-kept secret. I realize you and others are trying to do good things to and to uncover useful information; but I just don't think much useful information can be found under the constraint that you don't know how a model was trained. (Apart from, brute-fact advice like "Don't set up a model this way," which let's face it no one was ever going to do.)

I feel like we're failing to communicate. Let me recapitulate.

Remember, if the theories were correct and complete, the corresponding simulations would be able to do all the things that the real human cortex can do[5]—vision, language, motor control, reasoning, inventing new scientific paradigms from scratch, founding and running billion-dollar companies, and so on.

So, your argument here is modus tollens:

1. If we had a "correct and complete" version of the algorithm running in the human cortex (and elsewhere) then the simulations would be able to do all that a human can do.
2. The simulations cannot do all that a human can do. Therefore we do not, etc.

I'm questioning 1, by claiming that you need a good training environment + imitation of other entities in order for even the correct algorithm for the human brain to produce interesting behavior.

You respond to this by pointing out that bright, intelligent, curious children do not need school to solve problems. And this is assuredly true. Yet: bright, intelligent, curious children still learned language and an enormous host of various high-level behaviors from imitating adults; they exist in a world with books and artifacts created by other people, from which they can learn; etc, etc. I'm aware of several brilliant people with relatively minimal conventional schooling; I'm aware of no brilliant people who were feral children. Saying that humans turn into problem solving entities without plentiful examples to imitate seems simply not true, and so I remain confident that 1 is a false claim, and the point that bright people exist without school is entirely compatible with this.

I am extremely confident that there is no possible training environment that would lead a collaborative group of these crappy toy models into inventing language, science, and technology from scratch, as humans were able to do historically

Maybe so, but that's a confidence that you have entirely apart from providing these crappy toy models an actual opportunity to do so. You might be right, but your argument here is still wrong.

Humans did not, really, "invent" language, in the same way that Dijkstra invented an algorithm. The origin of language is subject to dispute, but it's probably something that happened over centuries or millenia, rather than all at once. So -- if you had an algorithm that could invent language from scratch, I don't think its reasonable to expect it to do so unless you give it centuries of millenia of compute, in a richly textured environment where it's advantageous to invent language. Which, of course, we have come absolutely nowhere close to doing.

The whole cortex is (more-or-less) a uniform randomly-initialized learning algorithm, and I think it’s basically the secret sauce of human intelligence. Even if you disagree with that, we can go up a level to the whole brain: the human brain algorithm has to be simple enough to fit in our (rather small) genome.[4] And not much evolution happened between us and chimps. And yet our one brain design, without modification, was able to invent farming and science and computers and rocket ships and everything else, none of which has any straightforward connection to tasks on the African savannah.

Anyway, the human cortex is this funny thing with 100,000,000 repeating units, each with 6-ish characteristic layers with correspondingly different neuron types and connection patterns, and so on. Nobody knows how it works. You can look up dozens of theories explaining what each of the 6-ish layers is doing and how, but they all disagree with each other. Some of the theories are supported by simulations, but those simulations are unimpressive toy models with no modern practical applications whatsoever.

Remember, if the theories were correct and complete, the corresponding simulations would be able to do all the things that the real human cortex can do[5]—vision, language, motor control, reasoning, inventing new scientific paradigms from scratch, founding and running billion-dollar companies, and so on.

This last conjunction doesn't seem valid.

A human brain is useless without ~5 to 15 years of training, training specifically scheduled to hit the right developmental milestones at the correct times. More if you think grad school matters. This training requires the active intervention of at least one, and usually many more than one, adult, who laboriously do RL schedules on the brain try to overcome various bits of inertia or native maladaption of the untrained system. If you raise a human brain without such training, it becomes basically useless for reasoning, inventing new scientific paradigms from scratch, and so on and so forth.

So -- given that such theories were accurate -- we would only expect the brains created via simulations utilizing these theories to be able to do all these things if we had provided the simulations with a body with accurate feedback, given it 5 to 15 years of virtual training involving the active intervention of adults into its life with appropriate RL schedules, etc. Which, of course, we have not done.

Like I think your explanation above is clear, and I like it, but feel a sense of non-sequitur that I've gotten in many other explanations for why we expect some future algorithm to supersede LLMs.

Calling references from someone's job application actually helps, even if they of course chose the references to be ones that would give a positive testimonial.

That's quite different though!

References are generally people contacted at work; i.e., those who can be verified to have had a relationship with the person giving the reference. We think these people, by reason of having worked with them, are in a position to be knowledgeable about them, because it's hard to fake qualities over long period of time. Furthermore, there's an extremely finite number of such people; someone giving a list of references will probably give a reference they think would give a favorable review, but they have to select from a short list of such possible references. That short list is why it's evidential. So it's filtered, but it's filtered from a small list of people well-positioned to evaluate your character.

Book reviews are different! You can hire a PR firm, contact friends, to try to send it to as many people for a favorable blurb as you possibly can. Rather than having a short list of potential references, you have a very long one. And of course, these people are in many cases not positioned to be knowledgeable about whether the book is correct; they're certainly not as well-positioned as people you've worked with are to know about your character. So you're filtering from a large list -- potentially very large list -- of people who are not well-positioned to evaluate the work.

You're right it provides some evidence. I was wrong about the claim about none; if you could find no people to review a book positively, surely that's some evidence about it.

But for the reasons above it's very weak evidence, probably to the degree where you wonder about about the careful use of evidence ("evidence" is anything that might help you distinguish between worlds you're in, regardless of strength -- 0.001 bits of evidence is still evidence) or the more colloquial ("evidence" is what helps you distinguish between worlds you're in reasonably strongly).

And like, this is why it's normal epistemics to ignore the blurbs on the backs of books when evaluating their quality, no matter how prestigious the list of blurbers! Like that's what I've always done, that's what I imagine you've always done, and that's what we'd of course be doing if this wasn't a MIRI-published book.

Like if you want the "remarkably strong reception" to have strong [retracting: any] evidential value it needs to be sent to skeptics.

Otherwise you're just like, "Yeah, we sent it to a bunch of famous people using our connections and managed to get endorsements from some." It's basically just and attempting to persuade someone with it is attempting to persuade them with means disconnected from the truth.

U.S. has a functioning market economy where the U.S. can incentivize things by paying for them, in a way that China cannot reliably.

I think your model of the world is just flat wrong if you think this. Like -- China interferes in China's economy a lot. The US interferes in the US' economy a lot. But -- surely -- China has a functioning market economy, where you can incentivize things by paying for them? Sure it's "Communist" but it's not communist like that.

Like Russia didn't have a functioning market economy. A sign of this was that the Russian cars sucked and found little traction as exports. That's because non-market economies produce bad products at high prices.

On the other hand, BYD sells more cars than Tesla (but competes against many other EV makers inside china). DJI sells more drones than the rest of the world combined (but also competes against other Chinese companies like Autel, etc). Huawei is the world's largest smartphone manufacturer since ~2020 or so, I think (and competes against other Chinese companies, and Apple, and Samsung). In general, many Chinese products are of high quality, to the degree that people in countries like Germany want to ban them from their markets because they are taking too much market share. And that's because -- unlike Russian cars -- they come from something that's at least a reasonably functioning market economy.

The Chinese government has surely subsidized this for various reasons, just like the US has subsidized the soybean farmers. They've surely made it less efficient in many ways. But these companies nevertheless compete on an marketplace internal to China, compete on marketplaces external to China, and have their success largely because they make products that are excellent while doing so efficiently.

And so it appears to me it would be much easier for China to scale drone production from a base that is ~20x higher than the US' production to continue to maintain absolute and overwhelming numerical superiority.