LESSWRONG
LW

All of Evan Anders's Comments + Replies

Sparse autoencoders find composed features in small toy models
Evan Anders32

Hi Lawrence! Thanks so much for this comment and for spelling out (with the math) where and how our thinking and dataset construction were poorly setup. I agree with your analysis and critiques of the first dataset. The biggest problem with that dataset in my eyes (as you point out): the true actual features in the data are not the ones that I wanted them to be (and claimed them to be), so the SAE isn't really learning "composed features."

In retrospect, I wish I had just skipped onto the second dataset which had a result that was (to me) surprising at the ... (read more)

Reply
Sparse autoencoders find composed features in small toy models
Evan Anders10

Hi Demian! Sorry for the really slow response.

Yes! I agree that I was surprised that the decoder weights weren't pointing diagonally in the case where feature occurrences were perfectly correlated. I'm not sure I really grok why this is the case. The models do learn a feature basis that can describe any of the (four) data points that can be passed into the model, but it doesn't seem optimal either for L1 or MSE.

And -- yeah, I think this is an extremely pathological case. Preliminary results look like larger dictionaries finding larger sets of features do a... (read more)

Reply
Sparse autoencoders find composed features in small toy models
Evan Anders10

Hi Ali, sorry for my slow response, too! Needed to think on it for a bit.

  • Yep, you could definitely generate the dataset with a different basis (e.g., [1,0,0,0] = 0.5*[1,0,1,0] + 0.5*[1,0,-1,0]).
  • I think in the context of language models, learning a different basis is a problem. I assume that, there, things aren't so clean as "you can get back the original features by adding 1/2 of that and 1/2 of this". I'd imagine it's more like feature1 = "the in context A", feature 2 = "the in context B", feature 3 = "the in context C". And if the is a real feature (I'm
... (read more)
Reply
2Demian Till
Regarding some features not being learnt at all, I was anticipating this might happen when some features activate much more rarely than others, potentially incentivising SAEs to learn more common combinations instead of some of the rarer features. In order to potentially see this we'd need to experiment with more variations as mentioned in my other comment
Sparse autoencoders find composed features in small toy models
Evan Anders32

Hi Logan! Thanks for pointing me towards that post -- I've been meaning to get around to reading it in detail and just finally did. Glad to see that the large-N limit seems to get perfect reconstruction for at least one similar toy experiment! And thanks for sharing the replication code.

I'm particularly keen to learn a bit more about the correlated features -- did you (or do you know of anyone) who has studied toy models where they have a few features that are REALLY correlated with one another, and that basically never appear with other features? I'm wond... (read more)

Reply
3Logan Riggs
Agreed. You would need to change the correlation code to hardcode feature correlations, then you can zoom in on those two features when doing the max cosine sim.
Sparse autoencoders find composed features in small toy models
Evan Anders20

Thanks for the comment! Just to check that I understand what you're saying here:


We should not expect the SAE to learn anything about the original choice of basis at all. This choice of basis is not part of the SAE training data. If we want to be sure of this, we can plot the training data of the SAE on the plane (in terms of a scatter plot) and see that it is independent of any choice of bases. 
 

Basically -- you're saying that in the hidden plane of the model, data points are just scattered throughout the area of the unit circle (in the uncorrela... (read more)

Reply
4Ali Shehper
Hi Evan, thank you for the explanation, and sorry for the late reply.  I think that the inability to learn the original basis is tied to the properties of the SAE training dataset (and won't be solved by supplementing SAEs with additional terms in its loss function). I think it's because we could have generated the same dataset with a different choice of basis (though I haven't tried formalizing the argument nor run any experiments). I also want to say that perhaps not being able to learn the original basis is not so bad after all. As long as we can represent the full number of orthogonal feature directions (4 in your example), we are okay. (Though this is a point I need to think more about in the case of large language models.)  If I understood Demian Till's post right, his examples involved some of the features not being learned at all. In your example, it would be equivalent to saying that an SAE could learn only 3 feature directions and not the 4th. But your SAE could learn all four directions. 
Examining Language Model Performance with Reconstructed Activations using Sparse Autoencoders
Evan Anders10

Ah! That's the context, thanks for the clarification and for pointing out the error.  Yes "problems" should say "prompts"; I'll edit the original post shortly to reflect that. 

Reply
Examining Language Model Performance with Reconstructed Activations using Sparse Autoencoders
Evan Anders30

Oh! You're right, thanks for walking me through that, I hadn't appreciated that subtlety. Then in response to the first question: yep! CE = KL Divergence.

Reply
Examining Language Model Performance with Reconstructed Activations using Sparse Autoencoders
Evan Anders10

After seeing this comment, if I were to re-write this post, maybe it would have been better to use the KL Divergence over the simple CE metric that I used. I think they're subtly different.

Per the TL implementation for CE, I'm calculating: CE =  where  is the batch dimension and  is context position. 

So CE =  for  the baseline probability and  the patched probability. 

So this is missing a factor of  to be the tru... (read more)

Reply
3Joseph Miller
I think it is the same. When training next-token predictors we model the ground truth probability distribution as having probability 1 for the actual next token and 0 for all other tokens in the vocab. This is how the cross-entropy loss simplifies to negative log likelihood. You can see that the transformer lens implementation doesn't match the equation for cross entropy loss because it is using this simplification. So the missing factor of p would just be 1 I think.
Examining Language Model Performance with Reconstructed Activations using Sparse Autoencoders
Evan Anders20

I think this is most of what the layer 0 SAE gets wrong. The layer 0 SAE just reconstructs the activations after embedding (positional + token), so the only real explanation I see for what it's getting wrong is the positional embedding.

But I'm less convinced that this explains later layer SAEs. If you look at e.g., this figure:

then you see that the layer 0 model activations are an order of magnitude smaller than any later-layer activations, so the positional embedding itself is only making up a really small part of the signal going into the SAE for any la... (read more)

Reply
1Joseph Bloom
My mental model is the encoder is working hard to find particular features and distinguish them from others (so it's doing a compressed sensing task) and that out of context it's off distribution and therefore doesn't distinguish noise properly. Positional features are likely a part of that but I'd be surprised if it was most of it. 
Examining Language Model Performance with Reconstructed Activations using Sparse Autoencoders
Evan Anders32

For me, this was actually a positive update that SAEs are pretty good on distribution -- you trained SAE on length 128 sequences from OpenWebText, and the log loss was quite low up to ~200 tokens! This is despite its poor downstream use case performance.

 

Yes, this was nice to see. I originally just looked at context positions at powers of 2 (...64, 128, 256,...) and there everything looked terrible above 128, but Logan recommended looking at all context positions and this was a cool result! 

But note that there's a layer effect here. I think layer... (read more)

Reply
2Joseph Bloom
  I kinda want to push back on this since OOD in behavior is not obviously OOD in the activations. Misgeneralization especially might be better thought of as an OOD environment and on-distribution activations?  I think we should come back to this question when SAEs have tackled something like variable binding with SAEs. Right now it's hard to say how SAEs are going to help us understand more abstract thinking and therefore I think it's hard to say how problematic they're going to be for detecting things like a treacherous turn. I think this will depend on how how representations factor. In the ideal world, they generalize with the model's ability to generalize (Apologies for how high level / vague that idea is).  Some experiments I'd be excited to look at: * If the SAE is trained on a subset of the training distribution, can we distinguish it being used to decompose activations on those data points off the training distribution? * How does that compare to an SAE trained on the whole training distribution from the model, but then looking at when the model is being pushed off distribution?  I think I'm trying to get at - can we distinguish: * Anomalous activations.  * Anomalous data points.  * Anomalous mechanisms.  Lots of great work to look forward to!