The most fundamental reason that I don't expect this to work is that it gives up on "sharing parameters" between the extractor and the human model. But in many cases it seems possible to do so, and giving on up on that feels extremely unstable since it's trying to push against competitiveness (i.e. the model will want to find some way to save those parameters, and you don't want your intended solution to involve subverting that natural pressure).

Intuitively, I can imagine three kinds of approaches to doing this parameter sharing:

1. Introduce some latent structure $L$ (e.g. semantics of natural language, what a cat "actually is") that is used to represent both humans and the intended question-answering policy. This is the diagram $H\leftarrow L\to f^{+}$
2. Introduce some consistency check $f^{?}$ between $H$ and $L$. This is the diagram $H\to f^{?}\leftarrow f^{+}$
3. Somehow extract $f^{+}$ from $H$ or build it out of pieces derived from $H$. This is the diagram $H\to f^{+}$. This is kind of like a special case of 1, but it feels pretty different.

(You could imagine having slightly more general diagrams corresponding to any sort of d-connection between $H$ and $f^{+}$.)

Approach 1 is the most intuitive, and it seems appealing because we can basically leave it up to the model to introduce the factorization (and it feels like there is a good chance that it will happen completely automatically). There are basically two challenges with this approach:

• It's not clear that we can actually jointly compress $H$ and $f^{+}$. For example, what if we represent $H$ in an extremely low level way as a bunch of neurons firing; the neurons are connected in a complicated and messy way that learned to implement something like $f^{+}$, but need not have any simple representation in terms of $f^{+}$. Even if such a factorization is possible, it's completely unclear how to argue about how hard it is to learn. This is a lot of what motivates the compression-based approaches---we can just say "$H$ is some mess, but you can count on it basically computing $f^{+}$" and then make simple arguments about competitiveness (it's basically just as hard as separately learning $H$ and $f^{+}$).
• If you overcame that difficulty, you'd still have to actually incentivize this kind of factorization in the model (rather than sharing parameters in the unintended way). It's unclear how to do that (maybe you're back to think about something speed-prior like, and this is just a way to address my concern about the speed-prior-like proposals), but this feels more tractable than the first problem.

I've been thinking about approach 2 over the last 2 months. My biggest concern is that it feels like you have to pay the bits of H back "as you learn them" with SGD, but you may learn them in such a way that you don't really get a useful consistency update until you've basically specified all of H. (E.g. suppose you are exposed to brain scans of humans for a long time before you learn to answer questions in a human-like way. Then at the end you want to use that to pay back the bits of the brain scans, but in order to do so you need to imagine lots of different ways the brain scans could have looked. But there's no tractable way to do that, because you have to fill in the the full brain scan before it really tells you about whether your consistency condition holds.)

Approach 3 is in some sense most direct. I think this naturally looks like imitative generalization, where you use a richer set of human answers to basically build $f^{+}$ on top of your model. I don't see how to make this kind of thing work totally on its own, but I'm probably going to spend a bit of time thinking about how to combine it with approaches 1 and 2.

How much hope is there for jointly representing $f_{{\theta }_1}$ and $H_{{\theta }_2}$?

The most obvious representation in this case is to first specify $f_{{\theta }_1}$, and then actually model the process of gradient descent that produces $H_{{\theta }_2}$. This runs into a few problems:

1. Actually running gradient descent to find ${\theta }_2$ is too expensive to do at every datapoint---instead we learn a hypothesis that does a lot of its work "up front" (shared across all the datapoints). I don't know what that would look like. The naive ways of doing it (redoing the shared initial computation for every batch) only work for very large batch sizes, which may well be above the critical batch size. If this was the only problem I'd feel pretty optimistic about spending a bunch of time thinking about it.
    
2. Specifying "the process that produced $H_{{\theta }_2}$" requires specifying the initialization ${\tilde{\theta }}_2$, which is as big as ${\theta }_2$. That said, the fact that the learned ${\theta }_2$ also contains a bunch of information about ${\theta }_1$ means that it can't contain perfect information about the initialization ${\tilde{\theta }}_2$, i.e. that multiple initializations lead to exactly the same final state. So that suggests a possible out: we can start with an "initial" initialization ${\tilde{\theta }}_2^0$. Then we can learn ${\tilde{\theta }}_2$ by gradient descent. The fact that many different values of ${\tilde{\theta }}_2$ would work suggests that it should be easier to find one of them; intuitively if we set up training just right it seems like we may be able to get all the bits back.
    
3. Running the gradient descent to find ${\theta }_2$ even a single time may be much more expensive than the rest of training. That is, human learning (perhaps extended over biological or cultural evolution) may take much more time than machine learning. If this the case, then any approach that relies on reproducing that learning is completely doomed.
   
   Similar to problem 2, the mutual information between ${\theta }_2$ and the learning process that produced ${\theta }_2$ also must be kind of low---there are only $\left|{\theta }_2\right|$ bits of mutual information to go around between the learning process, its initialization, and ${\theta }_1$. But exploiting this structure seems really hard, if actually there aren't any fast learning processes that lead to the same conclusion

My current take is that even in the case where ${\theta }_2$ was actually produced by something like SGD, we can't actually exploit that fact to produce a direct, causally-accurate representation ${\theta }_2$.

That's kind of similar to what happens in my current proposal though: instead we use the learning process embedded inside the broader world-model learning. (Or a new learning process that we create from fresh to estimate the specialness of ${\theta }_2$, as remarked in the sibling comment.)

So then the critical question is not "do we have enough time to reproduce the learning process that lead to ${\theta }_2$?" it is "Can we directly learn $H_{{\theta }_2}$ as an approximation to $f_{{\theta }_1}$?" If we able to do this in any way, then we can use that to help compress ${\theta }_2$. In the other proposal, we can use it to help estimate the specialness of ${\theta }_2$ in order to determine how many bits we get back---it's starting to feel like these things aren't so different anyway.

Fully learning the whole human-model seems impossible---after all, humans may have learned things that are more sophisticated then what we can learn with SGD (even if SGD learned a policy with "enough bits" to represent ${\theta }_2$, so that it could memorize them one by one if it saw the brain scans or whatever).

So we could try to do something like "learning just the part of the human policy is that is about answering questions." But it's not clear to me how you could disentangle this from all the rest of the complicated stuff going in for the human.

Overall this seems like a pretty tricky case. The high-level summary is something like: "The model is able to learn to imitate humans by making detailed observations about humans, but we are not able to learn a similarly-good human model from scratch given data about what the human is 'trying' to do or how they interpret language." Under these conditions it seems particularly challenging to either jointly represent $H_{{\theta }_2}$ and $f_{{\theta }_1}$, or to compute how many bits you should "get back" based on a consistency condition between them. I expect it's going to be reasonably obvious what to do in this case (likely exploiting the presumed limitation of our learning process), which is what I'll be thinking about now.

The difficulty of jointly representing $f_{{\theta }_1}$ and $H_{{\theta }_2}$ motivates my recent proposal, which avoids any such explicit representation. Instead it separately specifies ${\theta }_1$ and ${\theta }_2$, and then "gets back" bits by imposing a consistency condition that would have been satisfied only for a very small fraction of possible ${\theta }_2$'s (roughly $\exp (-|{\theta }_1\left|\right)$ of them).

But thinking about this neural network case also makes it easy to talk about why my recent proposal could run into severe computational problems:

• In order to calculate this loss function we need to evaluate how "special" ${\theta }_2$ is, i.e. how small is the fraction of ${\theta }_2$'s that are consistent with ${\theta }_1$
• In order to evaluate how special ${\theta }_2$ is, we basically need to do the same process of SGD that produces ${\theta }_2$---then we can compare the actual iterates to all of the places that it could have gone in a different direction, and conclude that almost all of the different settings of the parameters would have been much less consistent with ${\theta }_1$.
• The implicit hope of my proposal is that the outer neural network is learning its human model using something like SGD, and so it can do this specialness-calculation for free---it will be considering lots of different human-models, and it can observe that almost all of them are much less consistent with ${\theta }_1$.
• But the outer neural network could learn to model humans in a very different way, which may not involve representing a serious of iterates of "plausible alternative human models." For example, suppose that in each datapoint we observe a few of the bits of ${\theta }_2$ directly (e.g. by looking at a brain scan), and we fill in much of ${\theta }_2$ in this way before we ever start making good predictions about human behavior. Then we never need to consider any other plausible human-models.