The fact that current LLMs are effectively black boxes is arguably the biggest obstacle on the road to alignment. There have been many proposals for mechanistic interpretability and other ideas to pull back the veil on the internal workings of LLMs, but not enough attention has been given to the possibility of translating weights into actual code.
The gargantuan nature of the task puts it obviously beyond the reach of unassisted humans, but it seems plausible to me that a sufficiently capable LLM, given access to its own weights and internal activations, could reverse-engineer its learned computations and turn them into conventional, human-readable source code.
That would involve discovering the algorithms, procedures and representations encoded in its weights in order to reconstruct them as actual code. Depending on how far this can be taken, it could replicate most or all of the original neural network.
Given that current frontier models are already excellent coders (second only to their ability in maths), I think it's at least worth a try.
Feasibility
A neural network can be described as a computational structure. It pairs a certain architecture, which defines operations, with a set of weights that determine how those operations work out in practice. After that, we train them in such a way that the interactions of its parameters yield emergent capabilities.
There's no cosmic law that says that computations by a neural network can't be reproduced by a system other than a neural network. As far as I understand, the challenging bit isn't replicating its calculations, but gaining an adequate understanding of their higher-level structure.
However, there has been promising research in this direction. In "Learning Transformer Programs", Friedman, Wettig and Chen proved that specially designed, trainable transformers could be turned into human-readable Python programs. While it didn't involve the decompilation of an arbitrary pretrained LLM, it served as a proof of concept that transformer computations can be translated into conventional algorithms.
In "Opening the AI Black Box: Distilling Machine-Learned Algorithms into Code", Michaud et al. developed a method for extracting learned algorithms from recurrent neural networks and synthesising Python implementations. They tested their method on 62 algorithmic tasks and found it could solve 32. They also found GPT-4 could solve 30 (with substantial, but not complete overlap with their own novel method).
While none of these studies has reached the Holy Grail of frontier LLMs decompiling themselves, they suggest that several of the steps that we need to get there are already achievable.
LLM self-decompilation
Weighing in at up to 10 trillion parameters, current frontier models are too vast for us to have any hope of doing this manually. However, given enough compute, I suspect they're already good enough to reverse engineer themselves. This is even more likely to hold true for frontiers labs' internal models.
If given access to a weight checkpoint, as well as its architecture, activations and suitable experimental tools, an LLM agent could iteratively:
· identify potentially meaningful internal representations and computational circuits;
· come up with hypotheses about the roles of individual components;
· design experiments to test those hypotheses;
· convert validated mechanisms into explicit algorithms;
· test its reconstructed code against the original model weights using inputs that the model has not encountered before;
· rinse and repeat, gradually turning opaque components into more interpretable formulations.
A couple of caveats here. First, the obvious point that interpretability research can ultimately become capabilities research. Second, it would be unwise to let a model decompile itself without being sure it's not a scheming or deceptive model, which we can't guarantee while we're still at the black box stage of our understanding. In an ideal world, we would use a model from Provider A to decompile a model from Provider B, and vice versa, but that won't fly in the real world due to intellectual property and industrial secrets. However, even if we only used models to decompile themselves, I still think it would put us in a substantially better position than we are today, not least because we could (at least theoretically) have humans double-check the resulting code for signs of scheming or deceit.
Potential benefits
Even if full weight-to-code porting turned out not to be possible at this point in the tech tree, research benefits could extend beyond interpretability: model compression, portability, reproducibility, formal verification and preservation of learned capabilities.
If we did strike gold and achieve a successful port, it would open up a new paradigm for AI development. I would feel more comfortable entering the RSI regime if humans had the ability to read every new checkpoint as plain code instead of crossing our fingers and hoping the black box is doing what we want it to do.
Finally, even a partially successful reconstruction could make some capabilities less dependent on retaining the original neural weights and inference infrastructure. A hybrid implementation, combining explicit algorithms with neural components, would also be interesting.
The fact that current LLMs are effectively black boxes is arguably the biggest obstacle on the road to alignment. There have been many proposals for mechanistic interpretability and other ideas to pull back the veil on the internal workings of LLMs, but not enough attention has been given to the possibility of translating weights into actual code.
The gargantuan nature of the task puts it obviously beyond the reach of unassisted humans, but it seems plausible to me that a sufficiently capable LLM, given access to its own weights and internal activations, could reverse-engineer its learned computations and turn them into conventional, human-readable source code.
That would involve discovering the algorithms, procedures and representations encoded in its weights in order to reconstruct them as actual code. Depending on how far this can be taken, it could replicate most or all of the original neural network.
Given that current frontier models are already excellent coders (second only to their ability in maths), I think it's at least worth a try.
Feasibility
A neural network can be described as a computational structure. It pairs a certain architecture, which defines operations, with a set of weights that determine how those operations work out in practice. After that, we train them in such a way that the interactions of its parameters yield emergent capabilities.
There's no cosmic law that says that computations by a neural network can't be reproduced by a system other than a neural network. As far as I understand, the challenging bit isn't replicating its calculations, but gaining an adequate understanding of their higher-level structure.
However, there has been promising research in this direction. In "Learning Transformer Programs", Friedman, Wettig and Chen proved that specially designed, trainable transformers could be turned into human-readable Python programs. While it didn't involve the decompilation of an arbitrary pretrained LLM, it served as a proof of concept that transformer computations can be translated into conventional algorithms.
In "Opening the AI Black Box: Distilling Machine-Learned Algorithms into Code", Michaud et al. developed a method for extracting learned algorithms from recurrent neural networks and synthesising Python implementations. They tested their method on 62 algorithmic tasks and found it could solve 32. They also found GPT-4 could solve 30 (with substantial, but not complete overlap with their own novel method).
More recently, in "Weight-sparse transformers have interpretable circuits" Gao et al. studied making neural computations more understandable by constraining most of their weights to be zeros.
Last but not least, in "Can Language Model Agents be Helpful Circuit Explainers?", Khan et al. investigated using LLM agents to formulate hypotheses about neural circuits and validate their explanations through experiments.
While none of these studies has reached the Holy Grail of frontier LLMs decompiling themselves, they suggest that several of the steps that we need to get there are already achievable.
LLM self-decompilation
Weighing in at up to 10 trillion parameters, current frontier models are too vast for us to have any hope of doing this manually. However, given enough compute, I suspect they're already good enough to reverse engineer themselves. This is even more likely to hold true for frontiers labs' internal models.
If given access to a weight checkpoint, as well as its architecture, activations and suitable experimental tools, an LLM agent could iteratively:
· identify potentially meaningful internal representations and computational circuits;
· come up with hypotheses about the roles of individual components;
· design experiments to test those hypotheses;
· convert validated mechanisms into explicit algorithms;
· test its reconstructed code against the original model weights using inputs that the model has not encountered before;
· rinse and repeat, gradually turning opaque components into more interpretable formulations.
A couple of caveats here. First, the obvious point that interpretability research can ultimately become capabilities research. Second, it would be unwise to let a model decompile itself without being sure it's not a scheming or deceptive model, which we can't guarantee while we're still at the black box stage of our understanding. In an ideal world, we would use a model from Provider A to decompile a model from Provider B, and vice versa, but that won't fly in the real world due to intellectual property and industrial secrets. However, even if we only used models to decompile themselves, I still think it would put us in a substantially better position than we are today, not least because we could (at least theoretically) have humans double-check the resulting code for signs of scheming or deceit.
Potential benefits
Even if full weight-to-code porting turned out not to be possible at this point in the tech tree, research benefits could extend beyond interpretability: model compression, portability, reproducibility, formal verification and preservation of learned capabilities.
If we did strike gold and achieve a successful port, it would open up a new paradigm for AI development. I would feel more comfortable entering the RSI regime if humans had the ability to read every new checkpoint as plain code instead of crossing our fingers and hoping the black box is doing what we want it to do.
Finally, even a partially successful reconstruction could make some capabilities less dependent on retaining the original neural weights and inference infrastructure. A hybrid implementation, combining explicit algorithms with neural components, would also be interesting.