Interesting analysis, though personally, I am still not convinced that companies should be able to unilaterally (and irreversibly) change/update the human genome. But, it would be interesting to see this research continue in animals. E.g.
Provide evidence that they've made a "150 IQ" mouse or dog. What would a dog that's 50% smarter than the average dog behave like? or 500% smarter? Would a dog that's 10000% smarter than the average dog be able to learn, understand and "speak" in human languages?
Create 100s generations of these "gene updated" mice, dogs, cows, etc. as evidence that there are no "unexpected side effects", etc. Doing these types of "experiments" on humans, without providing long (long) term studies of other mammals seems to be... unwise/unethical?
Humanity has collectively decided to roll the dice on creating digital gods we don’t understand and may not be able to control instead of waiting a few decades for the super geniuses to grow up.
But yeah, given this along with the long (long) term studies mentioned above, the whole topic does seem to be (likely) moot...
Decommission of Legacy Networks and Applications
See also:
"Critical infrastructure systems often suffer from "patch lag," resulting in software remaining unpatched for extended periods, sometimes years or decades. In many cases, patches cannot be applied in a timely manner because systems must operate without interruption, the software remains outdated because its developer went out of business, or interoperability constraints require specific legacy software." -- Superintelligence Strategy by Dan Hendrycks, Eric Schmidt, Alexandr Wang https://www.nationalsecurity.ai/chapter/ai-is-pivotal-for-national-security
Bank and government mainframe software (including all custom Cobol, Pascal, Fortran code, etc.)
See also:
"Critical infrastructure systems often suffer from "patch lag," resulting in software remaining unpatched for extended periods, sometimes years or decades. In many cases, patches cannot be applied in a timely manner because systems must operate without interruption, the software remains outdated because its developer went out of business, or interoperability constraints require specific legacy software." -- Superintelligence Strategy by Dan Hendrycks, Eric Schmidt, Alexandr Wang https://www.nationalsecurity.ai/chapter/ai-is-pivotal-for-national-security
if we do well about nipping small failures in the bud, we may not get any medium-sized warning shots at all
This would be the scariest outcome: No warning, straight shot to fast self-improving, waaay above human level intelligence. And, (big shocker) turns out humans aren't that intelligent, and also super slow... 😬
"The cost of success is... embarrassment." 😳
Thank you for taking the time to write/post this and run the related Workshop!
imho, we need more people to really think deeply about how these things could plausibly play out over the next few years or so. And, actually spend the time to share (at least their mainline expectations) as well!
So, this is me taking my own advice to spend the time to layout my "mainline expectations".
An Alternate History of the Future, 2025-2040
This article was informed by my intuitions/expectations with regard to these recent quotes:
As, it's not clear to me how millions of "PhD+ reasoning/coding AI agents" can sustainably exist in the same internet as the worlds existing software stacks, which are (currently) very vulnerable to being attacked and exploited by these advanced AI agents. Patching all the software on the internet does seem possible, but not before these PhD+ AI agents are released for public use?
This post is quite short/non-specific, but I think it has some useful predictions/"load bearing claims", so very critique-able?
(I have admittedly haven't read through many other "vignette" examples, yet (somewhat purposefully, as to not bias my own intuitions, too much.))
I encourage anyone to please do read, comment, critique, challenge, etc. Thank you!
Thank you for running this Workshop!
"imho, we need more people to really think deeply about how these things could plausibly play out over the next few years or so. And, actually spend the time to share (at least their mainline expectations) as well!" -- Comment
So, this is me taking my own advice to spend the time to layout my "mainline expectations".
An Alternate History of the Future, 2025-2040
This article was informed by my intuitions/expectations with regard to these recent quotes:
As, it's not clear to me how millions of "PhD+ reasoning/coding AI agents" can sustainably exist in the same internet as the worlds existing software stacks, which are (currently) very vulnerable to being attacked and exploited by these advanced AI agents. Patching all the software on the internet does seem possible, but not before these PhD+ AI agents are released for public use?
This post is quite short/non-specific, but I think it has some useful predictions/"load bearing claims", so very critique-able?
(I have admittedly haven't read through many other "vignette" examples, yet (somewhat purposefully, as to not bias my own intuitions, too much.))
I encourage anyone to please do read, comment, critique, challenge, etc. Thank you!
This increases the samples in SWE-Bench Verified significantly, with the number of test cases growing exponentially as the AI itself contributes to the benchmark's expansion.
"We introduce SWE-Lancer, a benchmark of over 1,400 freelance software engineering tasks from Upwork..." -- https://arxiv.org/abs/2502.12115
@L Rudolf L Thank you for spending the time to write up this timeline! imho, we need more people to really think deeply about how these things could plausibly play out over the next few years or so. And, actually spending the time to share (at least their mainline expectations) as well. So yes, thank you for providing this excellent meal of "food for thought"!
That said... the statement below (imho) stands out as a potential "blocker" for your later arguments here? And, it also hits at the core of a lot of my own suspicions about the near-term future of AI and cybersecurity.
"2026... it remains true that existing code can now be much more easily attacked since all you need is an o6 or Claude subscription."
If you suspect that any existing software will be "hackable" by something like OpenAI's o6 or Anthropic's Claude, doesn't this implicitly assume a level of software maintenance that seems... almost utopian?
This would imply, that all functional software in the world is patched to remove any bugs that any "PhD+ level reasoning/coding AI agent" could find and exploit. That alone strikes me as a task of monumental proportions.
Simply finding the bugs/exploits does seem like the kind of task that millions of "PhD+-level reasoning/coding agents" might be able to accomplish, given enough compute and time. E.g.
- If each bug/exploit takes one H100 GFX card 10mins to find and fix...
- With 100k H100 GFX cards, they could fix 1M bugs in… 2 hours?
- (1,000,000 bugs * 10 mins/bug) / 100,000 H100 GFX Cards = 100 minutes
But, we're not just talking about patching all the world's software—imagine the millions of repositories on GitHub alone—with updated versions that contain no bugs or exploits that a "PhD+-level reasoning/coding model" could find and exploit. It goes much further than that. Finding and fixing 1 million bugs/exploits in active repos in GitHub would only be the first step. Then, starting millions (?) of PRs in Github, to be reviewed and merged? Could this be done by humans? Not likely. Or fully automatically, by other "PhD+ reasoning/coding agent AIs"? Possible, but that seems... scary?
Any bottleneck involving human review, approval, or manual intervention would likely make this an insurmountable challenge, given the sheer scale of the problem.
And then, it requires restarting and upgrading every running version of every application that's accessible on the internet? Think about all the legacy systems, the embedded devices, the mobile devices, the on-site Oracle servers, Microsoft SQL servers, all Banks Mainframe servers (Walmart, etc.), the vast amounts of (closed source) Pascal, Cobol and Fortran running on them. Will these Banks and Corporations allow PhD level LLMs to operate mostly autonomously to patch hundreds or thousands of newly discovered security flaws in all of their existing systems? And, if not, does that leave them open to being attacked by "PhD level reasoning/coding AI agents" from their competitors, or adversarial nations?
The speed at which these models are improving, and the potential for them to find zero-day exploits in even relatively well-maintained code, makes this a truly daunting prospect. Are we really prepared for a world where any vulnerability, no matter how obscure, is potentially discoverable and exploitable by anyone with a subscription to a powerful LLM?
What are your thoughts on the feasibility of actually achieving this level of global software hygiene, and what are the implications if humans can't? Would all these future software updates be applied by LLMs fully autonomously?
Imho, I just don't see how the Internet could remain functional if/when there are millions of "PhD+ Reasoning/Coding AI agents", with access to potentially millions of software bugs and exploits, on the same internet as any of the worlds existing software stacks?
It feels like we're on the precipice of a massive, possibly insurmountable, cybersecurity crisis, even before we get to the more existential risks of true ASI?
Edit: See also: An Alternate History of the Future, 2025-2040
Yes, and... "Would be interesting to see this research continue in animals. E.g. Provide evidence that they've made a "150 IQ" mouse or dog. What would a dog that's 50% smarter than the average dog behave like? or 500% smarter? Would a dog that's 10000% smarter than the average dog be able to learn, understand and "speak" in human languages?" -- From this comment