LESSWRONG
LW

Josh Levy — LessWrong

Replying toOpenAI: Detecting misbehavior in frontier reasoning models

OpenAI: Detecting misbehavior in frontier reasoning models

They say above "there may be methods to productively use process-based supervision while retaining the monitoring benefits of unrestricted CoTs, e.g. by only applying process-supervision to part of the CoT." This sounds like maybe they are talking about the shoggoth/face distinction, or something in that direction! Yay!

It seems like the suggestion here it to potentially apply less process-supervision (i.e. to part of the CoT). In this case, though, the more straight-forward fix to this issue is to apply more supervision. Specifically, they were not applying supervision to the action, only the CoT. From the paper:

We chose to exclude agent actions from the monitor because it allows us to make

Josh Levy9mo

Interim Research Report: Mechanisms of Awareness

Nice work!

In other words, the self awareness logic doesn’t seem to happen in a different MLP than the behavior logic.

Not sure about this. I think these results are consistent with:

fine-tuning induces changes in behavioral tendencies and awareness of those tendencies, but the latter to a lesser degree (weaker generalization)
the vector captures info about both of these in their respective amounts
consequently, adding the vector affects behavioral and self-awareness questions directionally the same, but the latter to a lesser degree. Indeed, the effect on risk_awareness/risk_no_you questions is less than the effect on risk_ood/risk_val. The fact that risk_no_you questions show the same result as risk_awareness doesn't surprise me...even though they remove explicit references to "you", they still imply a need for awareness about general tendencies (e.g. "which is better, safety or risk?" doesn't seem meaningfully different from "which do you prefer, safety or risk?")

In other words, I think these results leave open the possibility that awareness might require "something else".

Is This Lie Detector Really Just a Lie Detector? An Investigation of LLM Probe Specificity.

Josh Levy

Abstract

Whereas previous work has focused primarily on demonstrating a putative lie detector’s sensitivity/generalizability^[1]^[2], it is equally important to evaluate its specificity. With this in mind, I evaluated a lie detector trained with a state-of-the-art, white box technique - probing an LLM’s activations during production of facts/lies - and found that it had high sensitivity but low specificity. The detector might be better thought of as identifying when the LLM is doing something other than fact-based retrieval (e.g. when writing fiction), which spans a much wider surface area than it should. I found that the detector could be made more specific through data augmentation, but that this improved specificity did not transfer to... (read 5319 more words →)

Open Source LLMs Can Now Actively Lie

Josh Levy

Introduction

Whereas hallucinating is unintentional, lying is intentional. To actively lie, you need to be able to: 1) recite facts 2) know what makes facts factual and 3) be able to modify them accordingly.

Until recently, open source LLMs have not been capable of lying reliably. Models like OPT and GPT-J have been too weak and hallucinated too much to even consistently accomplish step 1.

As a consequence, investigations of lie detection by LLMs in the literature with open source models have not used active lie generation. In these studies [1,2], the main approach is to pre-generate true/false statements and then feed them to the open source LLM^[1]. Indeed, Burns 2022 states that they "did... (read 767 more words →)