I agree with a lot of this, we discuss the trickiness of measuring this properly in the paper (Appendix E.1) and I touched on it a bit in this post (the last bullet point in the last section). We did consider normalizing by the L2, ultimately we decided against that because the L2 indexes too heavily on the size of the majority of elements rather than indexing on the size of the largest elements, so it's not really what we want. Fwiw I think normalizing by the L4 or the L_inf is more promising.
I agree it would be good for us to report more data on the pre-trained vs randomized thing specifically. I don't really see that as a central claim of the paper so I didn't prioritize putting a bunch of stuff for it in the appendices, but I might do a revision with more stats on that, and I really appreciate the suggestions.
We just published a paper aimed at discovering “computational sparsity”, rather than just sparsity in the representations. In it, we propose a new architecture, Jacobian sparse autoencoders (JSAEs), which induces sparsity in both computations and representations. CLICK HERE TO READ THE FULL PAPER.
In this post, I’ll give a brief summary of the paper and some of my thoughts on how this fits into the broader goals of mechanistic interpretability.
We want the computational graph corresponding to LLMs to be sparse (i.e. have a relatively small number of edges). We developed a method for doing this at scale. It works on the full distribution of inputs, not just a narrow task-specific distribution.
Great post! I've also experienced this build-up at many points in my life, this framework seems really useful.
One thing I've found particularly helpful for addressing this is freewriting. Just sit down every morning with your laptop and commit to writing 750 words. Don't constrain what you write about, just let it be a stream of consciousness. You can write about your favorite type of pizza or whatever, but in practice you'll probably get bored of shallow things like this pretty quickly and start diving into more meaningful things. You'll then start noticing small annoyances that you weren't consciously aware of before and you'll gradually process them. And there will probably be a... (read more)
Completely agree! As I said in the post
> There are tons of things I would’ve loved to do in this paper if there had been more engineering-hours.