I did not use your initialization scheme, since I was unaware of your paper at the time I was running those experiments. I will definitely try that soon!
Yeah, I can see how leaky topk and multi-topk are doing similar things. I wonder if leaky topk also gives a progressive code past the value of k used in training. That definitely seems worth looking into. Thanks for the suggestions!
Standard sparse autoencoder training uses an sparsity loss term to induce sparsity in the hidden layer. However, theoretical justifications for this choice are lacking (in my opinion), and there may be better ways to induce sparsity. In this post, I explore other methods of inducing sparsity and experiment with them using Robert_AIZI's methods and code from this research report, where he trained sparse autoencoders on OthelloGPT. I find several methods that produce significantly better results than sparsity loss, including a leaky top- activation function.
This research builds directly on Robert_AIZI's work from this research report. While I highly recommend reading his full report, I will briefly summarize the parts of it that are directly relevant to... (read 3420 more words →)
Oh, yeah, looks like with p=2 this is equivalent to Hoyer-Square. Thanks for pointing that out; I didn't know this had been studied previously.
And you're right, that was a typo, and I've fixed it now. Thank you for mentioning that!