Summary * LLMs appear to universally learn a feature in their embeddings representing the frequency / rarity of the tokens they were trained on * This feature is observed across model sizes, in base models, instruction tuned models, regular text models, and code models * In models without tied weights,...
(Also posted on the EA Forum) This is one of my final projects for the Columbia EA Summer 2022 Project Based AI Safety Reading Group (special thanks to facilitators Rohan Subramini and Gabe Mukobi). If you're curious you can find my other project here. Summary In this project, I: 1....
I use Olah et al.'s definition of a feature as a direction in the vector space of... (read 1546 more words →)
Thanks for the comment! I didn't get around to testing that, but that's one of the exact things I had in mind for my "Next Steps" #3 on training regimens that more reliably produce optimal, interpretable models.
Interesting, I'll definitely look into that! Sounds quite related.
This is one of my final projects for the Columbia EA Summer 2022 Project Based AI Safety Reading Group (special thanks to facilitators Rohan Subramini and Gabe Mukobi). If you're curious you can find my other project here.
In this project, I:
What follows is a brief introduction to this work. For full details, please see:
Very true. If a token truly never appears in the training data, it wouldn't be trained / learned at all. Or similarly, if it's only seen like once or twice it ends up "undertrained" and the token frequency feature doesn't perform as well on it. The two least frequent tokens in the nanoGPT model are an excellent example of this. They appear like only once or twice and as a result don't get properly learned, and as a result end up being big outliers.
