All of Zach Furman's Comments + Replies

After convergence, the samples should be viewed as drawn from the stationary distribution, and ideally they have low autocorrelation, so it doesn't seem to make sense to treat them as a vector, since there should be many equivalent traces.

This is a very subtle point theoretically, so I'm glad you highlighted this. Max may be able to give you a better answer here, but I'll try my best to attempt one myself.

I think you may be (understandably) confused about a key aspect of the approach. The analysis isn't focused on autocorrelation within individual traces, ... (read more)

IIRC @jake_mendel and @Kaarel have thought about this more, but my rough recollection is: a simple story about the regularization seems sufficient to explain the training dynamics, so a fancier SLT story isn't obviously necessary. My guess is that there's probably something interesting you could say using SLT, but nothing that simpler arguments about the regularization wouldn't tell you also. But I haven't thought about this enough.

Good catch, thanks! Fixed now.

It's worth noting that Jesse is mostly following the traditional "approximation, generalization, optimization" error decomposition from learning theory here - where "generalization" specifically refers to finite-sample generalization (gap between train/test loss), rather than something like OOD generalization. So e.g. a failure of transformers to solve recursive problems would be a failure of approximation, rather than a failure of generalization. Unless I misunderstood you?

Note that in the SLT setting, "brains" or "neural networks" are not the sorts of things that can be singular (or really, have a certain $\lambda$) on their own - instead they're singular for certain distributions of data.

This is a good point I often see neglected. Though there's some sense in which a model $p\left(x|w\right)$ can "be singular" independent of data: if the parameter-to-function map $w\mapsto p\left(x|w\right)$ is not locally injective. Then, if a distribution $p\left(x\right)$ minimizes the loss, the preimage of $p\left(x\right)$ in parameter space can have non-t... (read more)

A possible counterpoint, that you are mostly advocating for awareness as opssosed to specific points is null, since pretty much everyone is aware of the problem now - both society as a whole, policymakers in particular, and people in AI research and alignment.

I think this specific point is false, especially outside of tech circles. My experience has been that while people are concerned about AI in general, and very open to X-risk when they hear about it, there is zero awareness of X-risk beyond popular fiction. It's possible that my sample isn't representa... (read more)

In the cybersecurity analogy, it seems like there are two distinct scenarios being conflated here:

1) Person A says to Person B, "I think your software has X vulnerability in it." Person B says, "This is a highly specific scenario, and I suspect you don't have enough evidence to come to that conclusion. In a world where X vulnerability exists, you should be able to come up with a proof-of-concept, so do that and come back to me."

2) Person B says to Person A, "Given XYZ reasoning, my software almost certainly has no critical vulnerabilities of any kind. I'm ... (read more)

I wish I had a more short-form reference here, but for anyone who wants to learn more about this, Rocket Propulsion Elements is the gold standard intro textbook. We used in my university rocketry group, and it's a common reference to see in industry. Fairly well written, and you should only need to know high school physics and calculus.

Obviously this is all speculation but maybe I'm saying that the universal approximation theorem implies that neural architectures are fractal in space of all distributtions (or some restricted subset thereof)?

Oh I actually don't think this is speculation, if (big if) you satisfy the conditions for universal approximation then this is just true (specifically that the image of $W$ is dense in function space). Like, for example, you can state Stone-Weierstrass as: for a Hausdorff space X, and the continuous functions under the sup norm $C(X,R)$, th... (read more)

Sorry, I realized that you're mostly talking about the space of true distributions and I was mainly talking about the "data manifold" (related to the structure of the map $x\mapsto p(x\mid w^{\ast })$ for fixed $w^{\ast }$). You can disregard most of that.

Though, even in the case where we're talking about the space of true distributions, I'm still not convinced that the image of $W$ under $p(x\mid w)$ needs to be fractal. Like, a space-filling assumption sounds to me like basically a universal approximation argument - you're assuming that the image of&n... (read more)

Very interesting, glad to see this written up! Not sure I totally agree that it's necessary for $W$ to be a fractal? But I do think you're onto something.

In particular you say that "there are points $y$ in the larger dimensional space that are very (even arbitrarily) far from $W$," but in the case of GPT-4 the input space is discrete, and even in the case of e.g. vision models the input space is compact. So the distance must be bounded.

Plus if you e.g. sample a random image, you'll find there's usually a finite distance you need to trave... (read more)

if the distribution of intermediate neurons shifts so that Othello-board-state-detectors have a reasonably high probability of being instantiated

Yeah, this "if" was the part I was claiming permutation invariance causes problems for - that identically distributed neurons probably couldn't express something as complicated as a board-state-detector. As soon as that's true (plus assuming the board-state-detector is implemented linearly), agreed, you can recover it with a linear probe regardless of permutation-invariance.

This is a more reasonable objection(alth

The reason the Othello result is surprising to the NTK is that neurons implementing an "Othello board state detector" would be vanishingly rare in the initial distribution, and the NTK thinks that the neuron function distribution does not change during training.

Yeah, that's probably the best way to explain why this is surprising from the NTK perspective. I was trying to include mean-field and tensor programs as well (where that explanation doesn't work anymore).

As an example, imagine that our input space consisted of five pixels, and at initialization neur

I think the core surprising thing is the fact that the model learns a representation of the board state. The causal / linear probe parts are there to ensure that you've defined "learns a representation of the board state" correctly - otherwise the probe could just be computing the board state itself, without that knowledge being used in the original model.

This is surprising to some older theories like statistical learning, because the model is usually treated as effectively a black box function approximator. It's also surprising to theories like NTK, mean-... (read more)

Yeah, that was what I was referring to. Maybe “algorithmic model” isn’t the most precise - what we know is that the NN has an internal model of the board state that’s causal (i.e. the NN actually uses it to make predictions, as verified by interventions). Theoretically it could just be forming this internal model via a big lookup table / function approximation, rather than via a more sophisticated algorithm. Though we’ve seen from modular addition work, transformer induction heads, etc that at least some of the time NNs learn genuine algorithms.

Agreed - that alone isn’t particularly much, just one of the easier things to express succinctly. (Though the fact that this predates deep learning does seem significant to me. And the fact that SLT can delineate precisely where statistical learning theory went wrong here seems important too.)

Another is that can explain phenomena like phase transitions, as observed in e.g. toy models of superposition, at a quantitative level. There’s also been a substantial chunk of non-SLT ML literature that has independently rediscovered small pieces of SLT, like failure... (read more)

Yeah, I can expand on that - this is obviously going be fairly opinionated, but there are a few things I'm excited about in this direction.

The first thing that comes to mind here is singular learning theory. I think all of my thoughts on DL theory are fairly strongly influenced by it at this point. It definitely doesn't have all the answers at the moment, but it's the single largest theory I've found that makes deep learning phenomena substantially "less surprising" (bonus points for these ideas preceding deep learning). For instance, one of the first thin... (read more)

I can't speak for Richard, but I think I have a similar issue with NTK and adjacent theory as it currently stands (beyond the usual issues). I'm significantly more confident in a theory of deep learning if it cleanly and consistently explains (or better yet, predicts) unexpected empirical phenomena. The one that sticks out most prominently in my mind, that we see constantly in interpretability, is this strange correspondence between the algorithmic "structure" we find in trained models (both ML and biological!) and "structure" in the data generating proces... (read more)

Someone with better SLT knowledge might want to correct this, but more specifically:

Studying the "volume scaling" of near-min-loss parameters, as beren does here, is really core to SLT. The rate of change of this volume as you change your epsilon loss tolerance is called the "density of states" (DOS) function, and much of SLT basically boils down to an asymptotic analysis of this function. It also relates the terms in the asymptotic expansion to things you care about, like generalization performance.

You might wonder why SLT needs so much heavy machinery, s... (read more)

If anyone wants an interactive visual for the correlation upper and lower bounds, I made one for myself here: https://www.math3d.org/chXa4xZrC.

The x, y, and z axes are the correlations between X and Y, and Y and Z, and X and Z respectively. Everything inside the blue surface is possible. There are also some sliders to help visualize level sets.

1. Yep, pre-LN transformers avoid the vanishing gradient problem.
2. Haven't checked this myself, but the phenomenon seems to be fairly clean? See figure 3.b in the paper I linked, or figure 1 in this paper.

I actually wouldn't think of vanishing/exploding gradients as a pathological training problem but a more general phenomenon about any dynamical system. Some dynamical systems (e.g. the sigmoid map) fall into equilibria over time, getting exponentially close to one. Other dynamical systems (e.g. the logistic map) become chaotic, and similar trajectories diverge... (read more)

Exponential growth is a fairly natural thing to expect here, roughly for the same reason that vanishing/exploding gradients happen (input/output sensitivity is directly related to param/output sensitivity). Based on this hypothesis, I'm preregistering the prediction that (all other things equal) the residual stream in post-LN transformers will exhibit exponentially shrinking norms, since it's known that post-LN transformers are more sensitive to vanishing gradient problems compared to pre-LN ones.

Edit: On further thought, I still think this intuition is co... (read more)

Great discussion here!

Leaving a meta-comment about priors: on one hand, almost-linear features seem very plausible (a priori) for almost-linear neural networks; on the other, linear algebra is probably the single mathematical tool I'd expect ML researchers to be incredibly well-versed in, and the fact that we haven't found a "smoking gun" at this point with so much potential scrutiny makes me suspect.

And while this is a very natural hypothesis to test, and I'm excited for people to do so, it seems possible that the field's familiarity with linear methods i... (read more)

A bit of a side note, but I don't even think you need to appeal to new architectures - it looks like the NTK approximation performs substantially worse even with just regular MLPs (see this paper, among others).

For anyone who wants to play around with this themselves, you might be interested in a small Colab notebook I made, with some interactive 2D and 3D plots.

To be clear, I don't know the answer to this!

Spitballing here, the key question to me seems to be about the OOD generalization behavior of ML models. Models that receive similarly low loss on the training distribution still have many different ways they can behave on real inputs, so we need to know what generalization strategies are likely to be learned for a given architecture, training procedure, and dataset. There is some evidence in this direction, suggesting that ML models are biased towards a simplicity prior over generalization strategies.

If this is... (read more)

In other words, does there exist any dataset such that generating extrapolations from it leads to good outcomes, even in the hands of bad actors?

I think this is an important question to ask, but "even in the hands of bad actors" is just too difficult a place to start. I'm sure you're aware, but it's an unsolved problem whether there exists a dataset / architecture / training procedure such that "generating extrapolations from it leads to good outcomes," for sufficiently capable ML models, even in the hands of good actors. (And the "bad actor" piece can at ... (read more)

My summary (endorsed by Jesse):

1. ERM can be derived from Bayes by assuming your "true" distribution is close to a deterministic function plus a probabilistic error, but this fact is usually obscured
2. Risk is not a good inner product (naively) - functions with similar risk on a given loss function can be very different
3. The choice of functional norm is important, but uniform convergence just picks the sup norm without thinking carefully about it
4. There are other important properties of models/functions than just risk
5. Learning theory has failed to find tight (generalization) bounds, and bounds might not even be the right thing to study in the first place

Since nobody here has made the connection yet, I feel obliged to write something, late as I am.

To make the problem more tractable, suppose we restrict our set of coordinate changes to ones where the resulting functions can still (approximately) be written as a neural network. (These are usually called "reparameterizations.") This occurs when multiple neural networks implement (approximately) the same function; they're redundant. One trivial example of this is the invariance of ReLU networks to scaling one layer by a constant, and the next layer by the inve... (read more)

Dropping some late answers here - though this isn't my subfield, so forgive me if I mess things up here.

Correct me if I'm wrong, but it struck while reading this that you can think of a neural network as learning two things at once:

1. a classification of the input into 2^N different classes (where N is the total number of neurons), each of which gets a different function applied to it
2. those functions themselves

This is exactly what a spline is! This is where the spline view of neural networks comes from (mentioned in Appendix C of the post). What you call "clas... (read more)

The field of complex systems seems like a great source of ideas for interpretability and alignment. In lieu of a longer comment, I'll just leave this great review by Teehan et al. on emergent structures in LLMs. Section 3 in particular is great.

But in the last few years, we’ve gotten: [...]

• Robots (Boston Dynamics)

Broadly agree with this post, though I'll nitpick the inclusion of robotics here. I don't think it's progressing nearly as fast as ML, and it seems fairly uncontroversial that we're not nearly as close to human-level motor control as we are to (say) human-level writing. I only bring this up because a decent chunk of bad reasoning (usually underestimation) I see around AGI risk comes from skepticism about robotics progress, which is mostly irrelevant in my model.

Have you looked into "conditionally conserved" quantities/symmetries here? Most macroscopic properties fall into this category - e.g. the color of a particular material is conserved so long as it doesn't change phase or transmute (i.e. it stays within a particular energy range). This is associated with a (spontaneously-broken) symmetry, since the absorption spectrum of a material can be uniquely determined from its space group. I'd be willing to bet that the only information accessible at a distance (up to a change of variables) are these conditionally con... (read more)

This is something I've been thinking about recently. In particular, you can generalize this by examining temporary conserved quantities, such as phases of matter (typically produced by spontaneous symmetry-breaking). This supports a far richer theory of information-accessible-at-a-distance than only permanently conserved quantities like energy can provide, and allows for this information to have dynamics like a stochastic process. In fact, if you know a bit of solid-state physics you probably realize exactly how much of our observed macroscopic properties ... (read more)

Why does GPT-3 use the same matrix for word embedding and final predictions? I would expect this to constrain the model, and the only potential upsides I can see are saving parameters (lol) and preserving interpretability (lmao)^[8]. Other resources like A Mathematical Framework for Transformer Circuits use different embedding/unembedding matrices - their $W_E$ and $W_U$. Perhaps this is not necessary for GPT-3 since the final feed-forward network can perform an appropriate linear transformation, and in A Mathematical Framework they are looking at

I don't think the game is an alarming capability gain at all - I agree with LawrenceC's comment below. It's more of a "gain-of-function research" scenario to me. Like, maybe we shouldn't deliberately try to train a model to be good at this? If you've ever played Diplomacy, you know the whole point of the game is manipulating and backstabbing your way to world domination. I think it's great that the research didn't actually seem to come up with any scary generalizable techniques or dangerous memetics, but I think ideally shouldn't even be trying in the first place.

So if streaming works as well as Cereberas claims, GPUs can do that as well or better.

Hmm, I'm still not sure I buy this, after spending some more time thinking about it. GPUs can't stream a matrix multiplication efficiently, as far as I'm aware. My understanding is that they're not very good at matrix-vector operations compared to matrix-matrix because they rely on blocked matrix multiplies to efficiently use caches and avoid pulling weights from RAM every time.

Cerebras says that the CS-2 is specifically designed for fast matrix-vector operations, and use... (read more)

The Andromeda 'supercomputer' has peak performance of 120 pflops dense compared to 512 pflops dense for a single 256 H100 GPU pod from nvidia

I'm not sure if PFLOPs are a fair comparison here though, if I understand Cerebras' point correctly. Like, if you have ten GPUs with one PFLOP each, that's technically the same number of PFLOPs as a single GPU with ten PFLOPs. But actually that single GPU is going to train a lot faster than the ten GPUs because the ten GPUs are going to have to spend time communicating with each other. Especially as memory limitations... (read more)

Hmm, I see how that would happen with other architectures, but I'm a bit confused how this is $O\left(n^2\right)$ here? Andromeda has the weight updates computed by a single server (MemoryX) and then distributed to all the nodes. Wouldn't this be a one-to-many broadcast with $O\left(n\right)$ transmission time?

No substantive reply, but I do want to thank you for commenting here - original authors publicly responding to analysis of their work is something I find really high value in general. Especially academics that are outside the usual LW/AF sphere, which I would guess you are given your account age.

This proposal looks really promising to me. This might be obvious to everyone, but I think much better interpretability research is really needed to make this possible in a safe(ish) way. (To verify the shard does develop, isn't misaligned, etc.) We'd just need to avoid the temptation to take the fancy introspection and interpretability tools this would require and use them as optimization targets, which would obviously make them useless as safeguards.

This is definitely the core challenge of the language model approach, and may be the reason it fails. I actually believe language models aren't the most likely approach to achieve superintelligence. But I also place a non-trivial probability on this occurring, which makes it worth thinking about for me.

Let me try to explain why I don't rule this possibility out. Obviously GPT-3 doesn't know more than a human, as evident in its sub-human performance on common tasks and benchmarks. But suppose we instead have a much more advanced system, a near-optimal seque... (read more)