When I consider the trajectory of my life, it has really been determined in large part due to my love of science of all kinds and my rather intense ADHD. When I was younger I had the thought that "most fields of science I care about are largely mathematical, so I should study mathematics!" which led me to doing my undergrad in math, where I fell in love with AI. The thing that particularly excited me was representation learning, and my opinion at the time was that for good representations, one needed to design the system with strong priors. So, I did my M.Sc was mathematics as well, studying group equivariant convolutional... (read 5270 more words →)
Really like this post. I had actually come to a very basic version of some of thus but not nearly as in-depth. Essentially it was realizing the Relu-Activated MLPs could be modeled as c-semirings and thus an information algebra. Some of the very basic concepts of superposition seemed to fall out of that. Would love to relate the work you guys have done to that, as your framework seems to answer some very deep questions.