LESSWRONG
LW

James Sullivan — LessWrong

Playing Dumb: Detecting Sandbagging in Frontier LLMs via Consistency Checks

1mo

TL;DR Large language models are becoming increasingly aware of when they are being evaluated. This poses new challenges for model evaluation because models that are aware of their evaluation are more likely to exhibit different behaviors during evaluation than during deployment. One potential challenge is models intentionally performing poorly on dangerous capability evaluations, also known as sandbagging, in order to preserve those capabilities in deployment. To combat this, I propose an evaluation methodology for detecting sandbagging using consistency checks. I created a dataset testing relevant knowledge for dangerous biosecurity capabilities where each question has two forms. One form is a formal presentation of the question designed to look like an evaluation, and... (read 1306 more words →)

Jailbreaking Claude 4 and Other Frontier Language Models

James Sullivan

8mo

AI systems are becoming increasingly powerful and ubiquitous, with millions of people now relying on language models like ChatGPT, Claude, and Gemini for everything from writing assistance to complex problem-solving. To ensure these systems remain safe as they grow more capable, they undergo extensive safety training designed to refuse harmful requests—declining to help with things like making bombs or committing fraud. However, these safety measures can be bypassed through carefully crafted prompts called "jailbreaks" that manipulate the models into providing exactly the kind of harmful information they were trained to avoid. In this post, I'll be sharing data on a jailbreak I found that works on some of the latest models released... (read 743 more words →)

How do AI agents work together when they can’t trust each other?

James Sullivan

8mo

I investigated this question by having Claude play the advanced social deduction game Blood on the Clocktower. Clocktower is a game similar to Werewolf (or Mafia) where players sit in a circle and are secretly divided into a good team and an evil team. The good players outnumber the evil players, but they don’t know who the evil players are, so they have to share information with each other to deduce who is evil and execute them before it's too late. Meanwhile the evil players try to build trust with the good team and disrupt their deduction as much as possible.

Clocktower takes this formula a step further by giving each player a... (read 2344 more words →)

Replying toMATS Spring 2024 Extension Retrospective

James Sullivan10mo

MATS Spring 2024 Extension Retrospective

Of the people that wanted to go to a frontier lab, how many had a mentor that worked at a frontier lab? I assume that would make finding a role easier.

Developmental Stages in Multi-Problem Grokking

James Sullivan

Summary

This post is my capstone project for BlueDot Impact’s AI Alignment course. It was a 12 week online course that covered AI risks, alignment, scalable oversight, technical governance and more. You can read more about it here.

In this project, I investigated the use of a developmental interpretability method—specifically, local learning coefficient estimation—to identify developmental stages in a toy transformer model. The model was tasked with learning both modular addition and modular multiplication simultaneously. Training followed a grokking pattern, where the model initially memorized the training data before gradually generalizing. To observe the impacts on test loss and the learning coefficient during training, I adjusted the ratio of addition to multiplication problems in... (read 1626 more words →)