A language model itself is just a description of a mathematical function that maps input sequences to output probability distributions on the next token.

Most of the danger comes from evaluating a model on particular inputs (usually multiple times using autoregressive sampling) and hooking up those outputs to actuators in the real world (e.g. access to the internet or human eyes).

A sufficiently capable model might be dangerous if evaluated on almost any input, even in very restrictive environments, e.g. during training when no human is even looking at the outputs directly. Such models might exhibit more exotic undesirable behavior like gradient hacking or exploiting side channels. But my sense is that almost everyone training current SoTA models thinks these kinds of failure modes are pretty unlikely, if they think about them at all.

You can also evaluate a partially-trained model at any point during training, by prompting it with a series of increasingly complex questions and sampling longer and longer outputs. My guess is big labs have standard protocols for this, but that they're mainly focused on measuring capabilities of the current training checkpoint, and not on treating a few tokens from a heavily-sandboxed model evaluation as potentially dangerous.

Perhaps at some point we'll need to start treating humans who evaluate SoTA language model checkpoint outputs as part of the sandbox border, and think about how they can be contained if they come into contact with an actually-dangerous model capable of superhuman manipulation or brain hacking.

yes, this is correct. i believe we should solve alignment before building AI, rather than after. (in fact, alignment should be fundamental to the design of the AI, not a patch you apply after the fact)

Should the title be rephrased?  The body of the post essentially addresses the scenario where OpenAI discovers (possibly via red-teaming) that their model is dangerous, and asks where do they go from there.  Which is a good question; it seems plausible that some naively think they could just tweak the model somewhat and make it non-dangerous, which seems to me a recipe for disaster when people get to jailbreaking it or get a copy and start un-tweaking it.  (A probably-better response would be to (a) lock it the fuck down [consider deleting it entirely], (b) possibly try to obfuscate the process that was used to create it, to obstruct others who'd try to reproduce your results, and (c) announce the danger, either to the world or specifically to law-enforcement types who'd work on preventing all AI labs from progressing any further.)

But the title asks "How do you red-team it to see whether it's safe", for which the straightforward answer is "Have people try to demonstrate dangerous capabilities with it".