I have come across various people (including my past self) who meet up regularly to study, e.g., alignment forum posts and discuss them. This helps people bond over their common believes, fears, and interests, which I think is good, but in no way is this ever going to lead anyone...
In his List of Lethalities, Eliezer writes that "matrices [in neural networks] are opaque". But I think our understanding of how neural networks actually work has improved significantly last year, with the deep learning theory book. I will briefly outline the general idea of the book below, in case closing...
I have come across various people (including my past self) who meet up regularly to study, e.g., alignment forum posts and discuss them. This helps people bond over their common believes, fears, and interests, which I think is good, but in no way is this ever going to lead anyone to find a solution to the alignment problem. In this post I'll reason why this doesn't help, and what I think we should do instead.
Reading good papers can be fun. You learn something interesting and, if the topic is hard but well presented by the authors, you get a kick from finally understanding something complicated. But is what you learned actually... (read 1260 more words →)
The predictions laid out in the book are mostly about how to build a perceptron such that representation learning works well in practice, and that the generalisation error gets minimised. For example,
To be clear: I don't have strong confidence that this works, but I think this is something worth exploring.
One more thing I should probably have added: I am only talking about the distributional shift in input data, which is important. But I think Eliezer is also talking about another kind of distributional shift that comes from a change in ontology. I am confused about how to think of this. Intuitively it is "the world hasn't changed, just how I look at it", whereas I discuss "the world has changed" (because the agent is doing things that haven't occurred during training).
I think 3blue1brown's videos give a good first introduction about neural nets (the "atomic" description):
Does this help?
I did not write down the list of quantities because you need to go through the math to understand most of them. One very central object is the neural tangent kernel, but there are also algorithm projectors, universality classes, etc., each of which require a lengthy explanation that I decided to be beyond the scope of this post.
Hmm, you may be right, sorry. I somehow read the opaqueness problem as a sub-problem of lie-detection. To do lie-detection we need to formulate mathematically what lying means, and for that we need theoretical understanding of what's going on in a neural net in the first place, so we have the right concepts to work with.
I think lie-detection in general is very hard, although it might be tractable in specific cases. The general problem seems hard because I find it difficult to define lying mathematically. Thinking about it for five minutes I hit several dead ends. The "best" one was this: If the agent (for lack of a better term) lies, it... (read more)
In his List of Lethalities, Eliezer writes that "matrices [in neural networks] are opaque". But I think our understanding of how neural networks actually work has improved significantly last year, with the deep learning theory book. I will briefly outline the general idea of the book below, in case closing this knowledge gap inspires any viable path towards alignment.
Everything that follows is paraphrased from Roberts, Yaida, and Hanin's book, unless I indicate otherwise. To understand the book, you need decent linear algebra and analysis skills, with a sprinkle of information theory. Most of the difficulty comes from the length of the calculations, which requires some practice to get used to. Luckily, the... (read 722 more words →)
Hi jylin04. Fantastic post! It touches on many more aspects of interpretability than my post about the book. I also enjoyed your summary PDF!
I'd love to contribute to any theory work in this direction, if I can. Right now I'm stuck around p. 93 of the book. (I've read everything, but I'm now trying to re-derive the equations and have trouble figuring out where a certain term goes. I am also building a Mathematica package that takes care of some of the more tedious parts of the calculations.) Maybe we could get in touch?