Arjun Panickssery's Shortform

Arjun Panickssery

This is a special post for quick takes by Arjun Panickssery. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

I struggle to believe at an intuitive level that autonomous R&D is so ingenious. I have been monitoring a few side projects:

Building a new SOTA engine for Scrabble (current SOTA is not really a neural-network method and doesn't beat the strongest human player)
Building a bot to play Ultimate Tic-Tac-Toe
Researching philosophy and formalizing some proofs for the AI Philosophy contest

In all of these cases, the model's judgment is just very poor in a basic way. For example:

When reviewing literature, it fails to identify the specific papers or blog posts that are obviously the most important and useful.
It will conduct experiments that have no information value because the amount of training data used was obviously too small to distinguish the architectures to be compared.
In the philosophy project, where it built a wiki over many tokens and investigations, it kept going down directions which were extremely trivial and had little value.
For Ultimate Tic-Tac-Toe, I saw a very clever idea in a blog post that I doubt Astra would ever have come up with; it actually located the blog post but didn't realize its importance or relevance (which could be discerned easily from his strong performance in an online arena and the recency of the post) over papers from previous years that are obviously less relevant.

I judged an Apart hackathon last weekend and reviewed eight papers, all of which scored between 60% and 95% on Pangram. Predictably, the papers were often very similar to each other, and kind of methodologically incomplete in a vague but similar way, where they seem to overlook the most straightforward analysis.

Maybe the important exception is certain hill-climbing tasks with fast feedback loops. For example, I improved the SOTA on "listing all legal Scrabble moves" by 10x just by prompting Astra over and over to try harder, to aim for <specific performance level>, to "systematically try every method in the literature that seems kind of related," etc. But even in that case, two big leaps were when I said that it should take into account the fact that you can measure across a game rather than per random position, and suggesting that it try to design algorithms for three different cases based on the number of blank tiles left in the bag (which cause the number of moves to explode).

There is also the fact that Astra at least (and presumably Claude) tends to write reports that are extremely difficult to understand and some kind of Q&A interrogation is often needed to figure out what it actually did and what the key results are. Maybe there's a temptation to just say "whatever" and let it rip.

I agree with you. AIs are good when pointed in the right direction (the key sources to refer to, the best metric to hill-climb, the parts of the story a human would find most surprising), but a human often still has to point them in the right direction first.

While it's nice that we Humans are still useful, it's sort of weird that this shepherding is still necessary. But IMO it bears out empirically, not just on vibes/anecdotes. The best evidence I'm aware of specifically for AI R&D automation is P-Zero's TasteVal, Epoch's InnovationEval, and Epoch's "C... (read more)

4habryka42m

Do you mean they overstate their absolute performance? FWIW, every major model generation of the last year has quite deeply reshaped basically everything about how I work.

3silentbob1h

Fwiw, my impression is that in coding Opus 5.5 (and even Sonnet 5.5) has vastly better judgement than Opus 4.5, which I used in January for a project. This is probably far away from research taste, and taste/judgement surely are also generally multidimensional/jagged, making them somewhat hard to evaluate and predict. But to name some examples: * Claude never breaks my current project and has fully autonomously written an extensive test suite that covers seemingly everything important while still being fast to run * Through ~100 iterations I asked it to add more and more complexity to the project with different interdependent systems, and it always finds ways to make things fit together * Its explanations are still slightly verbose but easy to follow and reasonably focused on the most important parts It's not perfect and sometimes it's a little lazy, building things that (obviously to me) don't meet the quality bar of the rest of the project and hence then require further iteration. But that could also just be due to me always using Medium effort level. Still, it now runs this code base of ~60k lines of code that I've never looked into and has no trouble expanding it further and further. In short, I'm pretty impressed by the level of judgment shown by Claude's 5.5 family when it comes to software engineering, and I wouldn't be shocked if this trend continued into the research domain with their next/internal models.