Posts
Wiki Contributions
Note that this is conditional SAE steering - if the latent doesn't fire it's a no-op. So it's not that surprising that it's less damaging, a prompt is there on every input! It depends a lot on the performance of the encoder as a classifier though
When do you use escape?
It seems unlikely that openai is truly following the test the model plan? They keep eg putting new experimental versions onto lmsys, presumably mostly due to different post training, and it seems pretty expensive to be doing all the DC evals again on each new version (and I think it's pretty reasonable to assume that a bit of further post training hasn't made things much more dangerous)
1I'm not super sure what I think of this project. I endorse the seed of the idea re "let's try to properly reverse engineer what representing facts in superposition looks like" and think this was a good idea ex ante. Ex post, I consider our results fairly negative, and have mostly confused that this kind of thing is cursed and we should pursue alternate approaches to interpretability (eg transcoders). I think this is a fairly useful insight! But also something I made from various other bits of data. Overall I think this was a fairly useful conclusion re updating away from ambitious mech interp and has had a positive impact on my future research, though it's harder to say if this impacted others (beyond the general sphere of people I mentor/manage)
I think the circuit analysis here is great, a decent case study of what high quality circuit analysis looks like, one of studies of factual recall I trust most (though I'm biased), and introduced some new tricks that I think are widely useful, like using probes to understand when information is introduced Vs signal boosted, and using mechanistic probes to interpret activations without needing training data. However, I largely haven't seen much work build on this, beyond a few scattered examples, which suggests it hasn't been too impactful. I also think this project took much longer than it should have, which is a bit sad.
Though, this did get discussed in a 3Blue1Brown video, which is the most important kind of impact!
1I really like this paper (though, obviously, am extremely biased). I don't think it was groundbreaking, but I think it was an important contribution to mech interp, and one of my favourite papers that I've supervised.
Superposition seems like an important phenomena that affects our ability to understand language models. I think this paper was some of the first evidence that it actually happens in language models, and on what it actually looks like. Thinking about eg why neurons detecting compound words (eg blood pressure) were unusually easy to represent in superposition, while "this text is in French" merited dedicated neurons, helped significantly clarify my understanding of superposition beyond what was covered in Toy Models of Superposition (discussed in Appendix A). I also just like having case studies and examples of phenomena in language models to think about, and have found some of the neuron families in this paper helpful to keep in mind when reasoning about other weirdnesses in LLMs. I largely think the results in this paper have stood the test of time.
Sparse autoencoders have been one of the most important developments in mechanistic interpretability in the past year or so, and significantly shaped the research of the field (including my own work). I think this is in substantial part due to Towards Monosemanticity, between providing some rigorous preliminary evidence that the technique actually worked, a bunch of useful concepts like feature splitting, and practical advice for training these well. I think that understanding what concepts are represented in model activations is one of the most important problems in mech interp right now. Though highly imperfect, SAEs seem the best current bet we have here, and I expect whatever eventually works to look at least vaguely like an SAE.
I have various complaints and caveats about the paper (that I may elaborate on in a longer review in the discussion phase), and pessimisms about SAEs, but I think this work remains extremely impactful and significantly net positive on the field, and SAEs are a step in the right direction.
How would you evade their tools?
A tip for anyone on the ML job/PhD market - people will plausibly be quickly skimming your google scholar to get a sense of "how impressive is this person/what is their deal" read (I do this fairly often), so I recommend polishing your Google scholar if you have publications! It can make a big difference.
I have a lot of weird citable artefacts that confuse Google Scholar, so here's some tips I've picked up:
- First, make a google scholar profile if you don't already have one!
- Verify the email (otherwise it doesn't show up properly in search)
- (Important!) If you are co-first author on a paper but not in the first position, indicate this by editing the names of all co-first authors to end in a *
- You edit by logging in to the google account you made the profile with, going to your profile, clicking on the paper's name, and then editing the author's names
- Co-first vs second author makes a big difference to how impressive a paper is, so you really want this to be clear!
- Edit the venue of your work to be the most impressive place it was published, and include any notable awards from the venue (eg spotlight, oral, paper awards, etc).
- You can edit this by clicking on the paper name and editing the journal field.
- If it was a workshop, make sure you include the word workshop (otherwise it can appear deceptive).
- See my profile for examples.
- Hunt for lost citations: Often papers have weirdly formatted citations and Google scholar gets confused and thinks it was a different paper. You can often find these by clicking on the plus just below your profile picture then add articles, and then clicking through the pages for anything that you wrote. Add all these papers, and then use the merge function to combine them into one paper (with a combined citation count).
- Merge lets you choose which of the merged artefacts gets displayed
- Merge = return to the main page, click the tick box next to the paper titles, then clicking merge at the top
- Similar advice applies if you have eg a blog post that was later turned into a paper, and have citations for both
- Another merging hack, if you have a weird artefact on your google scholar (eg a blog post or library) and you don't like how Google scholar thinks it should be presented, you can manually add the citation in the format you like, and then merge this with the existing citation, and display your new one
- If you're putting citations on a CV, semantic scholar is typically better for numbers, as it updates more frequently than Google scholar. Though it's worse at picking up on the existence of non paper artefacts like a cited Github or blog post
- Make your affiliation/title up to date at the top
Do you know what topics within AI Safety you're interested in? Or are you unsure and so looking for something that lets you keep your options open?
That's technically even more conditional as the intervention (subtract the parallel component) also depends on the residual stream. But yes. I think it's reasonable to lump these together though, orthogonalisation also should be fairly non destructive unless the direction was present, while steering likely always has side effects