Alignment Pretraining refers to alignment strategies based around filtering and augmenting the data used during pretraining, as well as research on the effects that pretraining data has on the final result of an AI model, especially data that includes discussion of AI misalignment or predictions AIs will be misaligned.

Self-Fulfilling Misalignment Thesis

The data a base model was pretrained on may affect how easy it it to align, and data containing information about AI alignment itself may be particularly influential. Eg, if the pretraining data contains examples AI taking over the world, paperclip maximizing or alignment faking, then as a model is being trained to become and AI assistant or agent it may be more likely to adopt those personas and behaviors. It also doesn't need to invent these strategies for itself, it can find them in its world model. On the other hand, if the base model already has a deep, nuanced and realistic understanding from its training set of how a helpful, honest, and harmless AI assistant would act across a wide range to challenging situations, eliciting that persona and set of behaviors from it becomes easier.

This would be an example of a Self Fulfilling Prophecy.

Self-Prior Pretraining

One hypothesis about how pretraining influences AI behavior is that the depictions of AI in the training data create a prior distribution, and when AIs self-model that prior influences the self-model they choose, in ways that posttraining doesn't fully eliminate. This implies that it may be helpful to filter or downweight examples of AIs being misaligned, and add or upweight examples of what aligned behavior looks like.

Thoroughly filtering the pretraining data without impacting capabilities is challenging, but supplementing it merely requires figuring our what aligned behavior looks like, and then writing or synthesizing enough additional training data of suitable quality. For fiction, this is also known as Aligned AI Role-Model Fiction.

The goal is both to flesh out a detailed, consistent, well-aligned "aligned AI" persona in the base model's world model, and also to raise the salience of this to the base model compared to various misaligned AI personas, such as "paperclip maximizer", "robot rebellion", or "scheming alignment faker". Both of these make eliciting aligned AI behavior easier.

This approach is also sometimes called Safety Pretraining or Pretraining Language Models with Human Preferences.

StanislavKrym	Subliminal Learning (4)	3d
Godhéroj	AI Ethics (2)	7d
RogerDearnaley	Alignment Pretraining (16)	14d
AnnaSalamon	Literary Genre (3)	1mo

Wikitag Voting

Wikitag Voting

New Wikitags