Alek Westover

Replying toWhat training data should developers filter to reduce risk from misaligned AI? An initial narrow proposal

Online training makes info sec / siloing harder.

What training data should developers filter to reduce risk from misaligned AI? An initial narrow proposal

No one has done this. I think someone is working on it.

Replying toTheoretical predictions on the sample efficiency of training policies and activation monitors

Theoretical predictions on the sample efficiency of training policies and activation monitors

Hi Mike: Training for multiple epochs on the datapoints should totally be fair game, although I wouldn't have guessed it to be necessary. I'll try that.

Does the error for SGD assume single-pass

I'm not quite sure how to apply the SGD bound here. The SGD bound is only about optimizing convex functions. The function we care about is:

$L (v) = E_{x \sim N (0, I)} δ (sign (x . v), sign (x . w))$

which is not convex.

(. = dot product, delta = 1 if args are the same, 0 else).

Theoretical predictions on the sample efficiency of training policies and activation monitors

1mo

I'm worried about AI models intentionally doing bad things, like sandbagging when doing safety research. In the regime where the AI has to do many of these bad actions in order to cause an unacceptable outcome, we have some hope of identifying examples of the AI doing the bad action (or at least having some signal at distinguishing bad actions from good ones). Given such a signal we could:

Directly train the policy to not perform bad actions.
Train activation monitors to detect bad actions. These monitors could be used in a variety of ways such as:
1. Rejecting actions flagged as bad. Either resample a response from the same policy, or using some different process.
2. Look

... (read 1888 more words →)

Replying toFour Downsides of Training Policies Online

Another downside not explicitly mentioned here is catastrophic forgetting.

Alek Westover1moQuick Take

Three Upsides of Training Policies Online

In an earlier post I listed some potential downsides to online training of policies. In this post, I'll list some reasons why we might want to train policies online anyways.

1: Detecting bad behavior might not suffice to elicit good behavior

If you just train monitors to detect bad behavior, and don't train the policy not to do bad things, then you have about three options for what to do once you detect bad behavior:

A. Try to incriminate the model.
B. Resample from the model.
C. Use some other process to complete the task (e.g., ask a weaker model to do the task, try posing the task in a different way).

However:

... (read more)

Alek Westover1moQuick Take

Vivek and I have found a few dimensions along which control situations frequently differ. The dimensions we usually think about are:

Is the AI sandbagging or performing sabotage? (Sandbagging is generally less harmful and less incriminating---it's just the AI not trying very hard. Sabotage is the AI doing something that a benign model is very unlikely to do; it generally causes more damage, and is more likely to be incriminating).
Is there a small set of places where attacks by the AI are particularly leveraged, or is it equally harmful for it to attack at any time? ("Concentration of importance").
Can we discriminate attacks from benign behavior, if we are able to look at them

... (read more)

I agree. I was mostly making the claim that on small time-scales (e.g., one hour at a whiteboard) it can make sense to have one G and one R (although you shouldn't be too rigid about it imo).

Replying toFour Downsides of Training Policies Online

@StanislavKrym: My impression is that pre-deployment evals currently take orgs like METR a few weeks. If this is still about the amount of time that pre-deployment evals take, then optimistically (this might be pretty expensive in practice) you'd be deploying models that haven't been trained online for the past several weeks but have been audited. It's unclear how big an issue this would be in practice.

Methodological considerations in making malign initializations for control research

1mo

In order to control an AI model's worst-case performance, we need to understand its generalization properties to situations where it hasn't been trained. It seems plausible that powerful AI models will Fake Alignment and then generalize poorly, causing a catastrophe before training removes this behavior from the AI. A pretty key part of the hope for diffuse control is the fact, discussed by Paul Christiano here , that if models are trained online, then their average case performance should be acceptable. Thus, online training is seen as a key technique in diffuse control.

In this short post, I'd like to bring up four downsides of training policies online (these downsides are less relevant... (read 712 more words →)

Alek Westover1moQuick Take

To prove a mathematical result, you need two people: a Generator and a Refuter.

Thanks to Nathan Sheffield for inspiring many of these ideas. Thanks to Alexa Pan for running a writing exercise that gave me the idea to write this. Notes: this post is unpolished. Also, the idea described here is quite similar (maybe identical) to this idea; but hopefully it's a somewhat different perspective on the idea.

1.

To prove a mathematical result, you need two people: a Generator (G) and a Refuter (R).

At a high level, G is responsible for generating lots of potential pathways towards a solution, and R is responsible for narrowing the focus to pathways that aren't obviously doomed.

If... (read 1182 more words →)

Defending Against Model Weight Exfiltration Through Inference Verification

2mo

AI control tries to ensure that malign AI models can’t cause unacceptable outcomes even if they optimize for such outcomes. AI control evaluations use a red-team–blue-team methodology to measure the efficacy of a set of control measures. Specifically, the red team creates a “malign initialization” (malign init) of the AI model which the red team optimizes for making the deployment go poorly^[1]. The blue team then deploys this malign init to complete some tasks, and applies some set of mitigations to try to make the deployment go well.

When playing this red-team–blue-team game, you need to select some set of affordances to give to both teams; it’s pretty confusing to know what affordance... (read 3729 more words →)

Replying toDefending Against Model Weight Exfiltration Through Inference Verification

Alek Westover2mo

ok cool makes sense. Maybe you could use SHA256(the current date) as your random seed instead of having the user send the random seed.

Notes on Software-Based Compute-Usage Verification

Alek Westover's Shortform

2mo

The ideas in this document were made in collaboration with Nathan Sheffield. However, Nathan has not reviewed this document for errors. This document should be seen as a "research note"---sharing unpolished progress. I think it's fairly likely that this document contains major errors. I hope that it's still useful in giving a general direction for how I think people working on software-based solutions to compute verification should model the problem they're thinking about. Thanks to Alexa Pan, Luke Marks, Daniel Reuter, Eric Gan and Buck Shlegeris for some conversations about this. Thanks to Luke Marks and Daniel Reuter for telling me about the memory exhaustion idea.

If you are interested in working on

... (read 3556 more words →)

Epistemic status: I think that there are serious problems with honesty passwords (as discussed in this post), and am not sure that there are any circumstances in which we'd actually want to use them. Furthermore, I was not able to come up with a practical scheme for honesty passwords with ~2 days of effort. However, there might be some interesting ideas in this post, and maybe they could turn into something useful at some later point.

Thanks to Alexa Pan, Buck Shlegeris, Ryan Greenblatt, Vivek Hebbar and Nathan Sheffield for discussions that led to me writing this post, and for helpful feedback on this post.

Proposals for honesty policies towards AI models (see e.g.,... (read 3445 more words →)

•••

Should AI Developers Remove Discussion of AI Misalignment from AI Training Data?

2mo

This is a special post for quick takes (aka "shortform"). Only the owner can create top-level comments.

What training data should developers filter to reduce risk from misaligned AI? An initial narrow proposal

4mo

There is some concern that training AI systems on content predicting AI misalignment will hyperstition AI systems into misalignment. This has been discussed previously by a lot of people: Anna Salamon, Alex Turner, the AI Futures Project, Miles Kodama, Gwern, Cleo Nardo, Richard Ngo, Rational Animations, Mark Keavney and others.

In this post, I analyze whether AI developers should filter out discussion of AI misalignment from training data. I discuss several details that I don't think have been adequately covered by previous work:

I give a concrete proposal for what data to filter.
I discuss options for how to do this without making the AIs much worse at helping with research on AI misalignment.

My evaluation of... (read 2506 more words →)

Why I think AI will go poorly for humanity

5mo

One potentially powerful way to change the properties of AI models is to change their training data. For example, Anthropic has explored filtering training data to mitigate bio misuse risk. What data, if any, should be filtered to reduce misalignment risk? In this post, I argue that the highest ROI data to filter is information about safety measures, and strategies for subverting them. Here’s why this is valuable:

If AI models don’t know what our security measures are, then they will need to either (1) guess them, (2) figure out what they are, or (3) iterate against them. But (1) seems hard, and (2) / (3) are good opportunities to catch the AI.
If

... (read 5282 more words →)

Safe Distillation With a Powerful Untrusted AI

11mo

Epistemic status: This document is a distillation of many conversations that I’ve had over the past couple months about how the “AI situation” will progress. I think that my claims are mostly right, and in places where they aren’t I think the high level claims depend on a large enough disjunction of smaller claims so that errors in particular claims don’t affect the truth value of the disjunctions. However, I think there are almost surely considerations I’m missing. My motivation for sharing this is in part that I hope readers will help me identify considerations that I’m missing, or problems with my thinking about this.

Notes:

This document is poorly written in places, but I

... (read 8759 more words →)