Note: This post is the second in a broader series of posts about the difficult tradeoffs inherent in public access to powerful open source models (the first post is here). While this post highlights some dangers of open models and discusses the possibility of global regulation, I am not, in...
Note: This post is intended to be the first in a broader series of posts about the difficult tradeoffs inherent in public access to powerful open source models. While this post highlights some dangers of open models and discusses the possibility of global regulation, I am not, in general, against...
In the past two years there has been increased interest in formal verification-based approaches to AI safety. Formal verification is a sub-field of computer science that studies how guarantees may be derived by deduction on fully-specified rule-sets and symbol systems. By contrast, the real world is a messy place that...
One year is already a long time in AI, but during the intelligence explosion it is so long that it means irrelevance.
This may seem obvious to you, but it is not at all obvious to many AI researchers, myself included. Can you share references to any surveys of AI researchers or research papers formally arguing this claim? I have heard some researchers make this claim, but despite having read pretty widely in AI research I have not seen anything like a serious empirical or theoretical attempt to justify or defend it.
@Daniel Kokotajlo - thanks for taking the time to read this and for your thoughtful replies.
So to make sure I understand your perspective, it sounds you believe that open models will continue to be widely available and will continue to lag about a year behind the very best frontier models for the forseeable future. But that they will simply be so underwhelming compared to the very best closed models that nothing significant on the world stage will come from it by 2030 (the year your scenario model runs to), even with (presumably) millions of developers building on open models by that point? And that you have such a high confidence in this... (read more)
Note: This post is the second in a broader series of posts about the difficult tradeoffs inherent in public access to powerful open source models (the first post is here). While this post highlights some dangers of open models and discusses the possibility of global regulation, I am not, in general, against open source AI, or supportive of regulation of open source AI today. On the contrary, I believe open source software is, in general, one of humanity’s most important and valuable public goods. My goal in this series of posts is to call attention to the risks and challenges around open models now, so we can use the time we still have... (read 1323 more words →)
Note: This post is intended to be the first in a broader series of posts about the difficult tradeoffs inherent in public access to powerful open source models. While this post highlights some dangers of open models and discusses the possibility of global regulation, I am not, in general, against open source AI, or supportive of regulation of open source AI today. On the contrary, I believe open source software is, in general, one of humanity’s most important and valuable public goods. My goal in writing this post is to call attention to the risks and challenges around open models now, so we can use the time we still have before risks... (read 6319 more words →)
Nice post Ryan! This kind of modeling strikes me as a very useful exercise, despite the fact that reasoning about systems of conditional probabilities based on conditions with complex descriptions at this scale is a little clunky for our human brains.
Regardless of the final estimates of P(scheming), which are likely to have high uncertainty, I also see a lot of value in the list of predictive factors you have called out here and their relative magnitude. If nothing else, these factors can help us by serving as warning signs or signs that things are going in the right or wrong direction as events unfold and by aggregating factors like these we can... (read 396 more words →)
I very much agree with this concern and I think that synthetic biology can be a good comparable case to ground our intuitions and help estimate reasonable priors.
For years, researchers have been sounding the alarm around the risks of advanced biotech, especially around tools that allow gene synthesis and editing. And then we had Covid-19, a virus that regardless of the politicization, probably was created in a lab. And in any case, regardless of whether you believe it was or wasn't it seems clear that it easily could have been. Worse, it's clear that something like Covid could also relatively easily be created by an independent team with a fairly modest budget.... (read more)
Thanks Steve! I love these examples you shared. I wasn't aware of them and I agree that they do a very good job of illustrating the current capability level of formal methods versus what is being proposed for AI safety.
Agustin - thanks for your thoughtful comment.
The concern you raise is something that I thought about quite a bit while writing this post/paper. I do address your concern briefly in several parts of the post and considered addressing it explicitly in greater detail, but ultimately decided not to because the post was already getting quite long. The first part where I do mention it is quoted below. I also include the quote from [1] that is from that part of the post as well, since it adds very helpful context.
... (read 1393 more words →)At the same time, obtaining an estimate that a DNA synthesis machine will only produce a dangerous pathogen 30% (or even 1%) of
In the past two years there has been increased interest in formal verification-based approaches to AI safety. Formal verification is a sub-field of computer science that studies how guarantees may be derived by deduction on fully-specified rule-sets and symbol systems. By contrast, the real world is a messy place that can rarely be straightforwardly represented in a reductionist way. In particular, physics, chemistry and biology are all complex sciences which do not have anything like complete symbolic rule sets. Additionally, even if we had such rules for the natural sciences, it would be very difficult for any software system to obtain sufficiently accurate models and data about initial conditions for a prover... (read 6728 more words →)
As a follow-on I would ask you: what is the mechanism for this "irrelevance" and why does it not appear in your scenario? In your scenario we are meant to be terrified of early 2028-frontier models going rogue, but by early 2029 (based on a one-year lag) models with those same capabilities would be in the hands of the general public and widely deployed (presumably many with no guardrails at all, or even overtly dangerous goals). And yet in your scenario there is no military first strike on the owners of these open models by OpenBrain, or again, even mention of these open models at all.