Testimonials

If you're interested in learning what making progress on a hard problem actually feels like, Team Shard is where you want to be. 

— Bruce Lee. MATS 7.0, primary author of Distillation Robustifies Unlearning

I really like Team Shard's focus on solving big problems that other people are missing. This focus resulted in me doing work that I think is much more impactful than I would have otherwise done. Being in Team Shard is also really fun.

— Luke Marks. MATS 8.0, primary author on Optimizing The Final Output Can Obfuscate CoT

Alex Turner and Alex Cloud provided consistently thoughtful guidance and inspiration that enabled my progress. I also had a ton of fun with the team :)

— Ariana Azarbal. MATS 8.0, primary author on Training a Reward Hacker Despite Perfect Labels 

Being a member of Team Shard helped me grow tremendously as a researcher. It gave me the necessary skills and confidence to work in AI Safety full-time.

— Jacob Goldman-Wetzler. MATS 6.0, primary author of Gradient Routing, now working at Anthropic

The mentors are ambitious and set high expectations, but are both super friendly and go out of their way to create a healthy, low-stress atmosphere amongst the team, ideal for brainstorming and collaboration. This collaborative environment, combined with their strong high-level research taste, has consistently led to awesome research outputs.

My time on Team Shard set the bar for what a productive collaboration should look like.

— Jacob Drori. MATS 8.0, primary author of Optimizing The Final Output Can Obfuscate CoT (Research Note) 

Apply for MATS mentorship at Team Shard before October 2nd. Alex Cloud (@cloud) and I run this MATS stream together. We help alignment researchers grow from seeds into majestic trees. We have fun, consistently make real alignment progress, and have a dedicated shitposting channel. 

Our mentees have gone on to impactful jobs, including (but not limited to)

1. @lisathiergart (MATS 3.0) moved on to being a research lead at MIRI and now a senior director at the SL5 task force,
2. @cloud (MATS 6.0) went from mentee to co-mentor in one round and also secured a job at Anthropic, and
3. @Jacob G-W (MATS 6.0) also accepted an offer from Anthropic!

We likewise have a strong track record in research outputs, including

1. Pioneering steering vectors for use in LLMs (Steering GPT-2-XL by adding an activation vector),
2. Masking Gradients to Localize Computation in Neural Networks, and
3. Distillation Robustifies Unlearning.

Our team culture is often super tight-knit and fun. For example, in this last MATS round, we lifted together every Wednesday and Thursday.

Apply here before October 2nd. (Don't procrastinate, and remember the planning fallacy!) 

Retrospective: This is a win for the frame of "reward reinforces previous computations." Ever since 2022, I've thought of "reward" as reinforcing the computations which led to the reward and as a chisel which carves circuits into the policy. From "Reward is not the optimization target":

What reward actually does is reinforce computations which lead to it... 

I suggest that you mechanistically model RL agents as executing behaviors downstream of past reinforcement (e.g. putting trash away), in addition to thinking about policies which are selected for having high reward on the training distribution (e.g. hitting the button). The latter form of reasoning skips past the mechanistic substance of reinforcement learning: The chiseling of computations responsible for the acquisition of the cognition-updater...

In my view, reward’s proper role isn’t to encode an objective, but a reinforcement schedule, such that the right kinds of computations get reinforced within the AI’s mind.

By thinking about reward in this way, I was able to predict^[1] and encourage the success of this research direction. 

Ariana showed that in this coding environment, it's not just about what the AI ends up choosing but also why the AI made that choice to begin with. Even though we "perfectly" reinforce the AI for doing what we wanted (i.e. avoiding special cases), we also often reinforced the system for the wrong reasons (i.e. considering special-casing the algorithm, even when not asked to do so). The AI's propensity to consider doing the wrong thing was reinforced and so the AI generalized to hack more in-distribution.

Assuming these results generalize, the trained policy is not just determined by the outputs which get rewarded. The trained policy also depends on which intermediate computations get rewarded. 

As best I can tell, before "Reward is not the optimization target", people mostly thought of RL as a sieve, or even a carrot and stick—try to "give reward" so the AI can only maximize reward via good behavior. Few^[2] other people speculated that RL generalization is controlled by why the policy took an action. So I give myself and @Quintin Pope^[3] a bunch of points.

1. ^{^}

   To be clear, my prediction was not as precise as "I bet you can reinforce sus CoTs and get sus generalization." The brainstorming process went like:

   1. What are some of the most open important problems in alignment? -> Reward hacking
   2. What are common assumptions about reward hacking? Oh, yeah, that hacking comes from reward function imperfections.
   3. Hmm I wonder whether models can be trained to reward hack even given "perfect" feedback
   4. We should really think more about this
   5. Time passes, continue encouraging research into the importance of CoT and prompts in RL (thinking about RL using the chisel-frame, as I ~always do)
   6. Victor and Ariana get this result.

2. Perhaps Steve Byrnes is an exception. 

3. ^{^}

   Quintin and I came up with "Reward is not the optimization target" together.

Nice work. What a cool use of steering vectors!

based prediction

Wasn't it the case that for some reason, full distillation had comparable compute requirement to data filtering? I was surprised by that. My impression is that distillation should be more like 10% of pretraining (data filtering), which would make the computational UNDO results much stronger. Not sure what happened here.