Thanks to Jasmina Urdshals, Xavier Poncini, and Justis Mills for comments. Introduction At Simplex our mission is to develop a principled science of the representations and emergent behaviors of AI systems. Our initial work showed that transformers linearly represent belief state geometries in their residual streams. We think of that...
Introduction In this note, I argue that the interpretation of representations (mental or machine learning) as causes of observed effects, is wrong by any sensible definition of causality. This interpretation seems to be somewhat common, although not pervasive. I propose a new interpretation based on dualism. In the new interpretation,...
Thanks to Jasmina Urdshals, Xavier Poncini, and Justis Mills for comments.
At Simplex our mission is to develop a principled science of the representations and emergent behaviors of AI systems. Our initial work showed that transformers linearly represent belief state geometries in their residual streams. We think of that work as providing the first steps into an understanding of what fundamentally we are training AI systems to do, and what representations we are training them to have.
Since that time, we have used that framework to make progress in a number of directions, which we will present in the sections below. The projects ask, and provide answers to, the following questions:
Hi Dr. Neal,
Wow, I studied your work in grad school! (And more recently your paper on Gaussian Processes). Quite an honor to get a comment from you. Just as an aside, I am not sure if my figure is visible, can you see it? I set it as the thumbnail, but I don't see it anywhere. In case it doesn't, it is here:
https://www.mlcrumbs.com/img/epistemology-of-representation.png
I think I need to change some labels, I realize now that I have been using 'x' ambiguously - sometimes as a model input, and sometimes to represent the bedrock physical system. But, to clarify, I'll use your vision example, but add temporality:
In this note, I argue that the interpretation of representations (mental or machine learning) as causes of observed effects, is wrong by any sensible definition of causality. This interpretation seems to be somewhat common, although not pervasive. I propose a new interpretation based on dualism. In the new interpretation, representations are instead causes of subsequent representations. The gist of it is shown in the figure, with time or causality flowing along the x-axis, and the level of representation along the y-axis. This re-interpretation has implications for the broader notion of "truth", in both the scientific and common-sense senses.
I partly arrived at this conclusion when contemplating large language models. They made me question... (read 2628 more words →)
