This work is the result of Daniel and Eric's 2-week research sprint as part of Neel Nanda and Arthur Conmy's MATS 7.0 training phase. Andy was the TA during the research sprint. After the sprint, Daniel and Andy extended the experiments and wrote up the results. A notebook that contains...
Experiments and write-up by Daniel, with advice from Stefan. Github repo for the work can be found here. Update (October 31, 2024): Our paper is now available on arXiv. TL;DR By perturbing activations along specific directions and measuring the resulting changes in the model output, we attempt to infer how...
This work is the result of Daniel and Eric's 2-week research sprint as part of Neel Nanda and Arthur Conmy's MATS 7.0 training phase. Andy was the TA during the research sprint. After the sprint, Daniel and Andy extended the experiments and wrote up the results. A notebook that contains all the analyses is available here.
Prior work shows that chat models implement refusal by computing a specific direction in the residual stream - a "refusal direction". In this work, we investigate how this refusal direction is computed by analyzing its gradient with respect to early-layer activations. This simple approach discovers interpretable features that are both causally upstream of refusal and contextually relevant... (read 3594 more words →)
Experiments and write-up by Daniel, with advice from Stefan. Github repo for the work can be found here.
Update (October 31, 2024): Our paper is now available on arXiv.
By perturbing activations along specific directions and measuring the resulting changes in the model output, we attempt to infer how much the directions matter for the model's computation. Through the sensitive directions experiments, we show that:
Hi, excited to learn more about Mech Int!