Maybe this is obvious, but the thought hadn't occurred to me before.
People sometimes make bit-counting arguments about RL, usually to say something about capabilities or alignment, something like: A million pretraining tokens puts several million bits of selection pressure on the models weights, but a million token RL rollout only puts a single bit of selection pressure. So even if we assume labs put as much compute into RL as they do everything everything else, the finished model is almost wholly determined by pretraining rather then RL, and therefore....
I mostly find these arguments unconvincing, my reasoning being something like: many properties of interest, like whether the model is willing to lie or not, whether its reward seeking or not, is a paperclipper, is evaluation aware, is myopic, has good icl abilities, what decision theory it uses, how rational it is, ..., can be encoded in very few bits, so weight change being small / short description length, relative to pretraining, doesn't mean very much practically speaking.
But I realize now that there are properties where the argument is very strong, mostly relating to interp and CoT monitorability / neuralese.
Like, coming up with a new language does take quite a few bits. And significantly changing your internal ontology, what features you have, what circuits you have, similarly does take quite a few bits.
From this I think I'd be comfortable concluding that post-RL CoT, even if it looks like gibberish, should be decodeable into same-meaning pre-RL CoT using a fairly simple function learnable through gradient descent.
And similarly, "structural" mechinterp techniques, computed/trained pre-RL, that apply to the whole of the model internals, should continue to work after RL, or at least shouldn't require all that sophisticated patching to continue to work after RL.
Note that this doesn't mean that the model can't learn things like "don't say 'im gonna lie' in cot". That requires few bits to encode. Sim
