Scaling Laws and Superposition
Summary Using results from scaling laws, this short note argues that the following two statements cannot be simultaneously true: 1. Superposition hypothesis where sparse features are linearly represented across a layer in fewer neurons is a complete theory of feature representation. 2. Features are universal, meaning two models trained on...
Apr 10, 20249