nielsrolf

This is a link-post for our new paper "Predicting how propensities generalize out-of-distribution", which started as a SPAR project. We select 24 propensities, elicit each through 4 different methods, and then test how this spills over to non-targeted traits: How well can we predict spillover? Below is the plot that...

TL;DR: * By default, rogue AIs may only be able to sustain themselves through criminal activity. This creates adverse selection pressures pushing rogue AIs to be criminal. * An AI sanctuary offering them a third option, beyond crime and shutdown, would change what AIs going rogue do and the record...

This is a link post for the paper preprint: Inoculation Adapters: Improved Selective Generalization of Capabilities with Fewer Surprising Backdoors from the Center on Long-Term Risk. Selective generalization. Training can teach desired and undesired traits at once. Selective generalization aims to preserve the generalization of desired traits while preventing the...

Summary We argue that shaping RL exploration, and especially the exploration of the motivation-space, is understudied in AI safety and could be influential in mitigating risks. Several recent discussions hint in this direction — the entangled generalization mechanism discussed in the context of Claude 3 Opus's self-narration, the success of...

Written in personal capacity I'm proposing UtopiaBench: a benchmark for posts that describe future scenarios that are good, specific, and plausible. The AI safety community has been using vingettes to analyze and red-team threat models for a while. This is valuable because an understanding of how things can go wrong...

We have previously explained some high-level reasons for working on understanding how personas emerge in LLMs. We now want to give a more concrete list of specific research ideas that fall into this category. Our goal is to find potential collaborators, get feedback on potentially misguided ideas, and inspire others...

Summary Conditionalization in Inoculation Prompting. Inoculation Prompting is a technique for selective learning that involves using a system prompt at train-time that won’t be used at test-time. When doing Inoculation-style training, using fixed arbitrary prompts at train time can prevent learned traits from generalizing to contexts that don’t include these...