The continuity argument gives something like an “alignment prior” based on previous models' alignment, distinct from other sources of evidence from alignment assessments such as training data monitoring and behavioral auditing.
For example, Anthropic’s August Alignment Risk Report said (p39-40):
* “Claim 3.1.3: Covered models were trained in a sufficiently similar way to prior models that the propensities of prior models provide some evidence about the propensities of covered models.”
* “Similarity here is not well defined, but we believe that it is intuitive to aggregate changes in scale, architecture, training processes, training data, final model capabilities, and observed propensities to come up with a rough overall estimate of similarity between two models.”
An observation about the continuity argument is that it might be significantly undermined by a shift to alternative architectures, which will probably co-occur with changes in training processes. It seems likely that an architectural shift will also co-occur with changes in scale, final model capabilities, and observed propensities, i.e., the new model is the result of larger training runs, is significantly more capable than the previous models, and either differs in actual observed propensities or the observability of its propensities (e.g. is much less CoT monitorable) compared to previous models.
If and when an architectural shift happens alongside all these other changes[1], it seems that our alignment prior can no longer depend on continuity with previous models. Our alignment prior might still be supported by other factors (e.g. the absolute capability levels at which you expect highly egregious forms of misalignment to arise, or the specific training processes needed for this), but I expect it would be substantially weakened.
Since I believe that current alignment assessments importantly depend on priors, I worry that the loss of the continuity argument would cause a significant amount of addi
