Quick Takes

I'm starting an internal red-teaming project at Forethought against Forethought's AI propensity targeting/model character work. The theory of change for internal red-teaming is that someone (me!) spending dedicated focused time on the negative case can find and elaborate on holes that Forethought's existing feedback processes will miss. Possible good outcomes from the red-teaming project (roughly in decreasing order of expected impact): 1. I identify and come up with good reasons to wind down Forethought's work in this field iff it is correct to do so. 2. 1. Note that "identify" in this case may involve original research, but it may instead be mostly about sourcing the ideas from other people. 3. I identify ways the current direction of propensity targeting work is bad and suggest significant directional changes. 4. People don't change high-level direction, but I understand and can make the counterarguments to this work legible enough within Forethought, so ppl are aware of the counterarguments and make less mistakes in their research. 5. I change my own mind significantly on the case for this work and decide to double down on it myself. 6. I identify sufficiently concrete cruxes that we can use to decide whether to prioritize this work in the future (eg stop/wind down this work if XYZ happens) I think I'd find it very helpful to understand why people are skeptical of AI propensity targeting work either in general or by Forethought in particular. For me, the most useful critiques will elaborate on why the work is net-negative (ideally significantly) or why it's almost certain to be useless. Though other critiques are welcome as well. I'd really appreciate comments from LessWrong people and others! Especially helpful is links to longer write-ups/existing comments. I might also want to chat on a call with people if anybody's interested. I've already read comments here, here, and various critical writings on Anthropic's Constitution and "soul doc."

Recently, I've been mulling over the question of whether it was a good idea or not to join a frontier AI company's safety team for the purposes of reducing extinction risk. One of my big cons was something like: Jay, you think the incentives are less likely to affect you compared to most people. But most AI safety people who join frontier labs probably think this. You will be affected as well. So I decided on a partial mitigation strategy, entirely as a precautionary principle and not at all because I thought I needed to. I committed to myself and to several people I'm close to that if I were to join a frontier lab safety team, I would donate 100% of the surplus that I would gain as a result of taking that job instead of a less lucrative job somewhere else. At this time I was applying for a few jobs, one of which was at a frontier company. Approximately immediately, my System 1 became way less interested in that job. And I didn't even have an offer in hand for a specific amount of money. I don't have good reasons to care a lot about getting more money for myself. I have enough already, and I voluntarily live well below my means. This did not stop the effect from existing, and I didn't notice the effect before. I still don't notice the effect on my thinking in a vacuum. I only notice it by doing a mental side-by-side comparison. I now think anyone who is considering joining a frontier company in order to reduce extinction risk should make this same commitment as a basic defensive measure against perverted incentives. I am sure there exist people who are entirely indifferent to money in this way - this is at least partially a skill issue on my part. But it does seem that "Thinking you are indifferent to the money" is not a reliable signal that your thinking is unaltered by it. This is also an opportunity to say that, if I ever do join a frontier safety team, I officially give you permission to ask me if I'm meeting this commitment of mine in conversation, even if

I've been thinking more about 'If you caught your AI red-handed on scheming, could you actually convince others that it was real and to take it seriously?' Buck Shlegeris wrote about this idea here: https://www.lesswrong.com/posts/YTZAmJKydD5hdRSeG/would-catching-your-ais-trying-to-escape-convince-ai One intervention that seems robustly helpful is keeping tamper-proof / tamper-evident logs, so you can demonstrate that a certain incident is in fact the actual and complete trajectory of your AI, that you are fully representing the inputs and context presented to it, etc. But I suspect that even the reveal of a matching, tamper-evident log of the trajectory might not be enough, unless you declared in advance that you were doing this and had a third-party verify that your logging system works the ways you claim it does. One way to increase the credibility of the logs might be for AI companies to publish hashes of their internal trajectories in close-to-real-time. Then, if a trajectory is later found to be shady, the earlier publication of its hash gives the authenticity a bit more weight - though even that might be accused of patiently seeding a false trajectory. To be clear, I'm not proposing that AI companies should publish hashes of their trajectories, per se, in part because it might throw off other information they don't necessarily want to. But I do want to gesture at the mechanism that's important to think through: 'How do you prove that what went in and came out of your system is in fact exactly as you are representing?'

Inoculation prompting seems a bit scary: * It makes it so AIs that have obviously schemer-like behavior—exhibiting instrumental training gaming while in training (potentially with tons of insane reward hacking and ruthlessness) but then behaving in way that looks great to the company in deployment—are an actively expected/desired result of the training process. So behaving much better when in deployment or when humans are applying close oversight is no longer a concerning warning sign of instrumental training gaming. Instead, it's a desired consequence of inoculation prompting (while also matching exactly how a schemer behaves). * It actively makes instrumental reasoning more salient (via directly instructing the model to instrumentally perform well in training). * It means that your model is by default following a cognitive strategy which is pretty close to being a schemer, so if there are other forces incentivizing scheming, it might be more likely for SGD to reinforce scheming cognition. (But maybe a bunch of the cognition can just be done in the text of the prompt, making it so you at least don't select for thoughtful general purpose instrumental reasoning?) Inoculation prompting might net reduce the chance of scheming via making it so scheming isn't necessarily actively selected for by a combination of the training process and the company's outer iteration loop—which selects for AIs that seem good in deployment potentially despite doing a bunch of reward hacking in training. Another benefit of a variant of inoculation prompting is that it could reduce the incentives for models to end up saliently situationally aware and doing their own instrumental reasoning via directly providing this situational awareness and directly telling the AI to perform well for instrumental reasons. At Redwood, we've historically called this a "shoe argument": you want to prevent a local shoe industry from developing—analogous to the AI more saliently being situationally aware—so

It might be interesting for someone to look into: Have there been large coordinated attempts to wean an industry off of blood money? (Successfully or not.) E.g. blood diamonds, blood gold, blood chocolate, blood cobalt, etc. What can we learn about that task from historical examples? What might transfer to frontier AI research and the surrounding ecosystem? E.g., is it at all possible to get "harvest" companies to be satisfied with some fixed level of model and not pay for advances? I know the answer is "no" but just saying it might be interesting.

Here are excerpts of a quick internal memo I wrote at Forethought for considering renaming the class of work that has variously been called "AI character" or "model spec." Note that I ~don't do object-level research directly on this field nor do I have direct decision-making power; I just sometimes get inspired by small random drive-by projects like this (and thought the effort:impact ratio here was good). Why might we want a new name? 1. People don’t like the current name.  2. 1. All else being equal, that’s a good reason to change the name especially in early stages, though we should be careful about being too willing to change our terms for other people. 3. “AI character” is confusing and overloaded as a phrase. 4. 1. In particular, I think it connotes a strong position that the decision procedure on moral matters for models should depend on virtue ethics, which might work for Anthropic but a) is controversial broadly and b) is not necessary for arguing our case. What criteria do we want in a new name? Here are some criteria that I think is probably central for a term for the field (feel free to disagree!) 1. it governs the intended behavior of the model 2. 1. ideally in a robustish way? 3. The name should be ecumenical on normative ethics as a decision procedure[1]. 4. 1. Note that the three ~most prominent existing cluster of terms each load a different normative-ethical theory: "AI character" leans virtue ethics, "alignment target" / "value loading" leans goal-directed consequentialism, and "model spec" leans (more weakly) rule-based deontology. A good replacement should avoid all three. [2] 2. Ideally it should try to completely punt the question of whether the AIs in question should be thought of as normative/moral agents guided fully by human ethics. 3. 1. Analogy: an idealized scientist has some normative commitments but (s)he does not need the full scope of human ethics to be a good scientist 5. It is designed with high compressibi

FWIW I suspect the brain has fewer inductive biases than it looks like it does when you try to reverse engineer it from lesion studies. If you did lesion studies on GPT it would look like it's made from a lot more parts than it is.