Cross-posted from my NAO Notebook.
This is an internal strategy note I wrote in November 2024 that I'm making public with some light editing.
In my work at the NAO I've been thinking about what I expect to see as LLMs continue to become more capable and get closer to where they can significantly accelerate their own development. I think we may see very large advances in the power of these systems over the next few years.
I'd previously thought that the main impact of AI on the NAO was through accelerating potential adversaries, and so shorter timelines primarily meant more urgency: we needed to get a comprehensive detection system in place quickly.
I now think, however, that this also means the best response involves some reprioritization. Specifically, AI will likely speed up some aspects of the creation of a detection system more than others, and so to the extent that we expect rapid advances in AI we should prioritize the work that we expect to bottleneck our future AI-accelerated work.
One way to plan for this is to imagine what would be the main bottlenecks if we had a far larger staff. Imagine if each senior person had AI support equivalent to all the smart junior people they could effectively manage. Or even (but my argument doesn't depend on this) AI systems that are as capable as today's experienced researchers. I think if in a year or two we found ourselves in this situation we would wish that:
We had collected a lot more data, because with a very large virtual computational staff future-AI-assisted-NAO can wring insights out of data far more efficiently than present-NAO.
We had started large-scale collection sooner, because, even if AI accelerates sequencing's price decreases and we can collect a lot more data in the future, it can't give us historical data.
We had a lot more partnerships for bringing in samples and data, because these take real-human time to scale up.
While I don't think this is the only way things could play out, I think it's likely enough that we should be taking these considerations very seriously in our planning.
April 2025: since initially drafting this we've started an ambitious effort to scale up our pilot system.