See footnote. Since this permutation freedom always exists no matter what the learned algorithm is, it can't tell us anything about the learned algorithm.

... Wait, are you saying we're not propagating updates into $\nu$ to change the mass it puts on inputs $0$ vs. $1$?

My viewpoint is that the prior distributions giving weight $1/3$ to each of the three hypotheses is different from the one giving weight $1/2$ to each of ${\nu }_0$ and ${\nu }_1$, even if their mixture distributions are exactly the same.

That's pretty unintuitive to me. What does it matter whether we happen to write out our belief state one way or the other? So long as the predictions come out the same, what we do and don't choose to call our 'hypotheses' doesn't seem particularly relevant for anything? 

We made our choice when we settled on $M$ as the prior. Everything past that point just seems like different choices of notation to me? If our induction procedure turned out to be wrong or suboptimal, it'd be because $M$ was a bad prior to pick, not because we happened to write $M$ down in a weird way, right?

If they have the same prior on sequences/histories, then in what relevant sense are they not the same prior on hypotheses? If they both sum to $M\left(x\right)$, how can their predictions come to differ?

I'm confused. Isn't one of the standard justifications for the Solomonoff prior that you can get it without talking about K-complexity, just by assuming a uniform prior over programs of length $l$ on a universal monotone Turing machine and letting $l$ tend to infinity?^[1] How is that different from your $P_{ap}\left(\nu \right)$? It's got to be different right, since you say that $P_{ap}\left(\nu \right)$ is not equivalent to the Solomonoff prior.

1. ^{^}

   See e.g. An Introduction to Universal Artifical Intelligence, pages 145 and 146.

Obviously SLT comes to mind, and some people have tried to claim that SLT suggests that neural network training is actually more like Solomonoff prior than the speed prior (e.g. bushnaq) although I think that work is pretty shaky and may well not hold up.

That post is superseded by this one. It was just a sketch I wrote up mostly to clarify my own thinking, the newer post is the finished product.

It doesn't exactly say that neural networks have Solomonoff-style priors. It depends on the NN architecture. E.g., if your architecture is polynomials, or MLPs that only get one forward pass, I do not expect them to have a prior anything like that of a compute-bounded Universal Turing Machine. 

And NN training adds in additional complications. All the results I talk about are for Bayesian learning, not things like gradient descent. I agree that this changes the picture and questions about the learnability of solutions become important. You no longer just care how much volume the solution takes up in the prior, you care how much volume each incremental building block of the solution takes up within the practically accessible search space of the update algorithm at that point in training.

I think just minimising the $L_0$ norm of the weights is worth a try. There's a picture of neural network computation under which this mostly matches their native ontology. It doesn't match their native ontology under my current picture, which is why I personally didn't try doing this. But the empirical results here seem maybe^[1] better than I predicted they were going to be last February. 

I'd also add that we just have way more compute and way better standard tools for high-dimensional nonlinear optimisation than we used to. It's somewhat plausible to me that some AI techniques people never got to work at all in the old days could now be made to kind of work a little bit with sufficient effort and sheer brute force, maybe enough to get something on the level of an AlphaGo or GPT-2. Which is all we'd really need to unlock the most crucial advances in interp at the moment.

1. ^{^}

   I haven't finished digesting the paper yet, so I'm not sure.

Problem with this: I think training tasks in real life are usually not, in fact, compatible with very many parameter settings. Unless the training task is very easy compared to the size of the model, basically all spare capacity in the model parameters will be used up eventually, because there's never enough of it. The net can always use more, to make the loss go down a tiny bit further, predict internet text and sensory data just a tiny bit better, score a tiny bit higher on the RL reward function. If nothing else, spare capacity can always be used to memorise some more training data points. $H\left(\Theta |API\right)$ may be maximal given the constraints, but the constraints will get tighter and tighter as training goes on and the amount of coherent structure in the net grows, until approximately every free bit is used up.^[1]

But we can still ask whether there are subsets of the training data on which the model outputs can be realised by many different parameter settings, and try to identify internal structure in the net that way, looking for parts of the parameters that are often free. If a circuit stores the fact that the Eiffel tower is in Paris, the parameter settings in that circuit will be free to vary on most inputs the net might receive, because most inputs don't actually require the net to know that the Eiffel tower is in Paris to compute its output.

A mind may have many skills and know many facts, but only a small subset of these skills and facts will be necessary for the mind to operate at any particular moment in its computation. This induces patterns in which parts of the mind's physical implementation are or aren't free to vary in any given chunk of computational time, which we can then use to find the mind's skills and stored facts inside its physical instantiation.

So, instead of doing stat mech to the loss landscape averaged over the training data, we can do stat mech to the loss landscapes, plural, at every training datapoint.

1. ^{^}

   Some degrees of freedom will be untouched because they're baked into the architecture, like the scale freedom of ReLU functions. But those are a small minority and also not useful for identifying the structure of the learned algorithms. Precisely because they are guaranteed to stay free no matter what algorithms are learned, they cannot contain any information about them. 

I think on the object level, one of the ways I'd see this line of argument falling flat is this part

Some AI safety problems are legible (obvious or understandable) to company leaders and government policymakers, implying they are unlikely to deploy or allow deployment of an AI while those problems remain open (i.e., appear unsolved according to the information they have access to).

I am not at all comfortable relying on nobody deploying just because there are obvious legible problems. With the right incentives and selection pressures, I think people can be amazing at not noticing or understanding obvious understandable problems. Actual illegibility does not seem required.

In my experience, the main issue with this kind of thing is finding really central examples of symmetries in the input that are emulatable. There's a couple easy ones, like low rank^[1] structure, but I never really managed to get a good argument for why generic symmetries in the data would often be emulatable^[2] in real life.^[3]

You might want to chat with Owen Lewis about this. He's been thinking about connections between input symmetries and mechanistic structure for a while, and was interested in figuring out some kind of general correspondence between input symmetries and parameter symmetries.

1. ^{^}

   If $q\left(x\right)$ only depends on a low-rank subspace of the inputs $x$, there will usually^[4] be degrees of freedom in the weights that connect to that input vector. The same is true of the hidden activations, if they're low rank, we get a corresponding number of free weights. See e.g. section 3.1.2 here.

2. ^{^}

   Good name for this concept by the way, thanks.

3. ^{^}

   For a while I was hoping that almost any kind of input symmetry would tend to correspond to low-rank structure in the hidden representations of $p\left(x\right|{\Theta }^{\ast })$, if $p(.)$ has the sort of architecture used by modern neural networks. Then, almost any kind of symmetry would be reducible to the low-rank structure case^[2], and hence almost any symmetry would be emulatable. 
   
   But I never managed to show this, and I no longer think it is true.

4. ^{^}

   There are a couple of necessary conditions for this of course. E.g. the architecture $p(.)$ needs to actually use weight matrices, like neural networks do.