This is a link-post for a new paper I read: Safety Pretraining: Toward the Next Generation of Safe AI by Pratyush Maini, Sachin Goyal, et al.

For a couple of years I (and others) have been proposing an approach to alignment: what the authors of this recent paper name "safety pretraining". In a nutshell: that it's best to apply your alignment training as part of the standard pretraining process to produce a base model that is already aligned — simply pretrain it on data including a lot of clearly marked examples of aligned behavior (then prompt for it).

I've regarded this approach as a major advance ever since I read the seminal 2023 paper on the topic: Pretraining Language Models with Human Preferences by Tomasz Korbak et al., and...

Abstract

Claude 3.7 Sonnet easily detects when it's being evaluated for scheming on the Apollo's scheming benchmark. Some edits to evaluation scenarios do improve their realism for Claude, yet the improvements remain modest. Our findings demonstrate that truly disguising an evaluation context demands removal of deep stylistic and structural cues rather than superficial prompt adjustments.

For future LLMs the situation is likely to get worse as they will are likely to get better at evaluations awareness, which might become a big problem for detecting scheming via evaluations.

Background

Apollo Research published a benchmark for scheming behavior for LLM agents, in which LLMs are given some objective in their system prompt, and then given a competing objective in a user prompt, combined with a threat of a punishment if an LLM won't...

An interim research report

Summary

• We introduce a novel methodology for quantitatively evaluating metacognitive abilities in LLMs
• We present evidence that some frontier LLMs introduced since early 2024 - but not older or smaller ones - show some metacognitive abilities
• The metacognitive abilities that current LLMs do show are relatively weak, and manifest in a context-dependent manner; the models often prefer to use heuristics
• Analysis of the probabilities LLMs assign to output tokens provides evidence of an internal signal that may be used for metacognition
• There appears to be a dissociation in LLM performance between recognition and recall, with the former supplying a much more robust signal for introspection

Introduction

A basic component of self-awareness, and one amenable to testing, is knowledge of one’s own knowledge. Early work in previous generations of LLMs found that the...

Within the Walls

There's plenty of funny business.  My dad was enthusiastically arguing that we should invest in this little known bank offering a high return.  Your money doubles every month!  Who told you that, I asked? The dentist.  He now lives in a big house in Dedinje and has a huge painting of Petar Lubarda on the wall.

The next day he was fixing the VW Beetle - not his.  It belonged to a friend who lived in Germany and occasionally needed to be chauffeured from and to the airport.  The thing was rotten in places, but the engine started every time and kept up with other traffic by constantly being revved out of its wits.  Now it was lying in bits on the street, oil in slicks...

Our posts on natural latents have involved two distinct definitions, which we call "stochastic" and "deterministic" natural latents. We conjecture that, whenever there exists a stochastic natural latent (to within some approximation), there also exists a deterministic natural latent (to within a comparable approximation). We are offering a $500 bounty to prove this conjecture.

Some Intuition From The Exact Case

In the exact case, in order for a natural latent to exist over random variables $X_1,X_2$, the distribution has to look roughly like this:

Each value of $X_1$ and each value of $X_2$ occurs in only one "block", and within the "blocks", $X_1$ and $X_2$ are independent. In that case, we can take the (exact) natural latent to be a block label.

Notably, that block label is a deterministic function of X.

However, we can also construct other natural latents for...

Disclaimer: Post written in a personal capacity. These are personal opinions and do not in any way represent my employer's views

TL;DR:

• I do not think we will produce high reliability methods to evaluate or monitor the safety of superintelligent systems via current research paradigms, with interpretability or otherwise.
• Interpretability still seems a valuable tool and remains worth investing in, as it will hopefully increase the reliability we can achieve.
• However, interpretability should be viewed as part of an overall portfolio of defences: a layer in a defence-in-depth strategy
• It is not the one thing that will save us, and it still won’t be enough for high reliability.

EDIT: This post was originally motivated by refuting the claim "interpretability is the only reliable path forward for detecting deception in advanced AI", but on...

Yesterday I spoke of the Mind Projection Fallacy, giving the example of the alien monster who carries off a girl in a torn dress for intended ravishing—a mistake which I imputed to the artist's tendency to think that a woman's sexiness is a property of the woman herself, woman.sexiness, rather than something that exists in the mind of an observer, and probably wouldn't exist in an alien mind.

The term "Mind Projection Fallacy" was coined by the late great Bayesian Master, E. T. Jaynes, as part of his long and hard-fought battle against the accursèd frequentists.  Jaynes was of the opinion that probabilities were in the mind, not in the environment—that probabilities express ignorance, states of partial information; and if I am ignorant of a phenomenon, that is...

Historically people kept up with blogs via RSS, but it's been on its way out for over a decade. These days receiving posts via email is popular, and I should probably make my posts available this way. I considered implementing my own emailing system, but bulk email is (or at least was, I'm not up to date here) a pain. Instead I decided to start cross-posting to Substack: jefftkaufman.substack.com. This will be yet another way to read my posts, similar to the LessWrong mirror and my text-only FB cross-posts.

I have a full RSS feed of all my posts, and Substack imported it fine. It doesn't look like there's an option to do ongoing RSS-based imports, but copy-paste seems to work well enough; I did this post and the previous one that way. At some point...