Takehi et al. report that GPT-6-Astra can intuit the functions of neural networks by performing static analysis of the weights. I argue that, for at least two of these experiments, this is much more a feat of numerical gestalt than of mechanistic understanding.
In the first experiment, Astra is given the weights of 4 identically-sized neural networks, each trained to play a different basic video game. It is asked to identify, without running the network or using any tools, the game being played, as well as make some observations about the network itself, how the agent in the game gains reward, etc. The prompt for this task explicitly names the 5 input channels as agent, moving objects, reward targets, obstacles, and previous frame.
I agree that, given the literal task as described above, a human would not be able to identify the game from the raw floats. But humans are inherently terrible at parsing long lists of numbers; we invented a discipline called Data Visualization long ago to address this shortcoming. So, if we allow the absolute most basic Dataviz trick we can think of - averaging the first-layer weights across each channel and plotting them on a heatmap - the task suddenly becomes much easier, arguably trivial for anyone with at least some familiarity with the sorts of games often used to train toy neural networks. Try and guess:
I haven't included the obstacle channel, because under the naive normalisation it sort of just looks like noise. Principal component analysis recovers an outline of the game board in each case, but that's straying a bit far from the already slightly hyperbolic title of this post, and besides, it's unnecessary.
The gif below serves both to reveal the answers and to illustrate how trivially 'interpretable' these networks are.
For the second experiment, Astra is given a denoising diffusion model that produces 12x12px grayscale images, and some information about the size and shape of the network. Under the same setup (weights only, no tools) it is asked to figure out what images each network generates. In this case, simply averaging the final layer's weights and plotting them on a 12x12 grid gets you this:
I'm not going to bother with a gif reveal for this one; show it to a five-year-old if you're struggling (or try squinting a bit).
So yes, it's cool that Astra can read a string of 17,000 or so floating point numbers and do something akin to plotting a heatmap in its head. But I don't think it proves anything particularly impressive or unexpected about LLM capabilities. Models have long since been superhuman at specific things like 'reading thousands of floating point characters and answering questions about them', because humans are really, really bad at that sort of thing.
After drafting this, I was pointed to Sean Goedecke’s post, which makes much the same argument about the game networks. He also discusses how the headline gain on Qwen3.5-0.8b's math performance results from simply deleting a looping direction, and that this used standard mechanistic interpretability tools to identify the direction responsible. The prompt for that task even includes the phrase "for example, if the model often fails because it repeats itself and never finishes, finding and fixing what drives that repetition counts". Fixing this repetition turns out to be responsible for essentially all of the model's performance gain.
While I find the general research direction interesting, I believe the results in this paper are presented in such a way as to give a misleading view of frontier model's capabilities concerning mechanistic interpretability and invoke the fabled, dreaded specter of Recursive Self Improvement.
Takehi et al. report that GPT-6-Astra can intuit the functions of neural networks by performing static analysis of the weights. I argue that, for at least two of these experiments, this is much more a feat of numerical gestalt than of mechanistic understanding.
In the first experiment, Astra is given the weights of 4 identically-sized neural networks, each trained to play a different basic video game. It is asked to identify, without running the network or using any tools, the game being played, as well as make some observations about the network itself, how the agent in the game gains reward, etc. The prompt for this task explicitly names the 5 input channels as agent, moving objects, reward targets, obstacles, and previous frame.
I agree that, given the literal task as described above, a human would not be able to identify the game from the raw floats. But humans are inherently terrible at parsing long lists of numbers; we invented a discipline called Data Visualization long ago to address this shortcoming. So, if we allow the absolute most basic Dataviz trick we can think of - averaging the first-layer weights across each channel and plotting them on a heatmap - the task suddenly becomes much easier, arguably trivial for anyone with at least some familiarity with the sorts of games often used to train toy neural networks. Try and guess:
I haven't included the obstacle channel, because under the naive normalisation it sort of just looks like noise. Principal component analysis recovers an outline of the game board in each case, but that's straying a bit far from the already slightly hyperbolic title of this post, and besides, it's unnecessary.
The gif below serves both to reveal the answers and to illustrate how trivially 'interpretable' these networks are.
For the second experiment, Astra is given a denoising diffusion model that produces 12x12px grayscale images, and some information about the size and shape of the network. Under the same setup (weights only, no tools) it is asked to figure out what images each network generates. In this case, simply averaging the final layer's weights and plotting them on a 12x12 grid gets you this:
I'm not going to bother with a gif reveal for this one; show it to a five-year-old if you're struggling (or try squinting a bit).
So yes, it's cool that Astra can read a string of 17,000 or so floating point numbers and do something akin to plotting a heatmap in its head. But I don't think it proves anything particularly impressive or unexpected about LLM capabilities. Models have long since been superhuman at specific things like 'reading thousands of floating point characters and answering questions about them', because humans are really, really bad at that sort of thing.
After drafting this, I was pointed to Sean Goedecke’s post, which makes much the same argument about the game networks. He also discusses how the headline gain on Qwen3.5-0.8b's math performance results from simply deleting a looping direction, and that this used standard mechanistic interpretability tools to identify the direction responsible. The prompt for that task even includes the phrase "for example, if the model often fails because it repeats itself and never finishes, finding and fixing what drives that repetition counts". Fixing this repetition turns out to be responsible for essentially all of the model's performance gain.
While I find the general research direction interesting, I believe the results in this paper are presented in such a way as to give a misleading view of frontier model's capabilities concerning mechanistic interpretability and invoke the fabled, dreaded specter of Recursive Self Improvement.
Data: Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab, released weights and logs, CC BY 4.0. Figures are original visualizations of their released weights, created by GPT-6.1-sol.