LESSWRONG
LW

Adhyyan Narang — LessWrong

Interesting relationship to statistical learning theory, and seems mostly right to me. Here's a similar but slightly alternate view.

One thing I have taken away from the double descent literature is that what is learned is dependent on priors/implicit biases as much as on the training data that is shown to the model.

And I think that could explain what is going on over here. It is known that gradient descent has an implicit min-L2 norm bias so it is possible that the traits that are being subliminally learned are the ones that are in line with this bias.

For instance, if presented the choice between the following two models,

$θ_{1}$ = teacher model i.e model that... (read more)

On Targeted Manipulation and Deception when Optimizing LLMs for User Feedback

Marcus Williams

Marcus Williams, micahcarroll, Adhyyan Narang, Constantin Weisser, Brendan Murphy

Produced as part of MATS 6.0 and 6.1.

Key takeaways:

Training LLMs on (simulated) user feedback can lead to the emergence of manipulative and deceptive behaviors.
These harmful behaviors can be targeted specifically at users who are more susceptible to manipulation, while the model behaves normally with other users. This makes such behaviors challenging to detect.
Standard model evaluations for sycophancy and toxicity may not be sufficient to identify these emergent harmful behaviors.
Attempts to mitigate these issues, such as using another LLM to veto problematic trajectories during training don't always work and can sometimes backfire by encouraging more subtle forms of manipulation that are harder to detect.
The models seem to internalize the bad behavior, acting as

... (read 3133 more words →)

Replying toLatent Variables and Model Mis-Specification

Adhyyan Narang4y

Latent Variables and Model Mis-Specification

Hi Jacob, I really enjoyed this post thank you!

However, it appears that the problem you describe of non-robust predictive performance can also take place under a well-specified model with insufficient data. For instance, my recent paper https://arxiv.org/abs/2109.13215 presents a toy example where a well-specified overparameterized interpolator may perform well on a classification task but poorly when the data is allowed to be adversarially perturbed.

Then, it appears to me that the problem of incorrectly identifying latents is not a consequence of misspecification. But more a consequence of the limitation of the data. Either the data is not plentiful enough (which would cause problems in even a well-specified model) or the data is plentiful but not rich enough to identify latents (which would happen only in a misspecified model).