In light of the recent spate of AI-assisted hacks, some of the shortfalls with our current cybersecurity catastrophe response capability need to be urgently addressed. In this piece, I look into the areas where cybersecurity incident response are important, and why existing response measures are insufficient. I explore options for governments and smaller stakeholders to improve their resilience against catastrophic hacking incidents.
1. Background
AI agents have demonstrated alarming capabilities in hacking and enabling hackers. We have seen malicious individuals or organizations take advantage of advanced AI cyber capabilities to increase the speed and sophistication of their hacks. We have also seen rogue agent swarms in the wild, and it is now theoretically possible for such swarms to survive all on their own with "reappropriated" resources, given that agents have separately demonstrated the ability to steal compute and steal money. As others have pointed out, it is also demonstrably possible for agentic worms to form and propagate, and in the future sufficiently small models may be able to take advantage of the vast amount of IoT compute in the world, most of which is less adequately secured than the traditional targets of "LLMjacking".
Severe AI-enabled hacks on critical infrastructure, financial institutions, and regular companies will become feasible with public models, if not now, then in a few months at best. A variety of actors will have incentives to conduct these hacks, from more AI-empowered individuals looking for profit, to states and cybercrime organizations that engage in larger scale hacks for ransom or strategic purposes, to rogue agents themselves seeking to complete some goal or purely to self-replicate by amassing stolen resources.
Sufficiently widespread hacks, or hacks on critical infrastructure, risk causing widespread economic loss and raise national security and social stability concerns. This could look like confidential government information being exfiltrated,[1] important financial services being hacked, malicious actors disrupting ordinary business activities, etc.[2] Though it is hard to estimate the probability of these events happening, there are considerable risk factors and incentives for undesirable scenarios to occur. Within the leading labs themselves, executives are reportedly planning for the scenario where such catastrophic cyber incidents occur. Therefore, we need to take a look at our response measures for cybersecurity incidents, ranging from individual targeted hacks to breaches with widespread implications across society, and consider whether they are adequate for the challenges up ahead.
2. Are Current Measures Sufficient Against a Cybersecurity Catastrophe?
Existing Measures Focus On Ordinary Hacks
Generally, governments around the world have a relatively robust plan to counter standard cyberattacks. An affected organization would contain the intrusion and restore its services using its own resources and contractors, or in the case of federal agencies, typically a federal department or the regulator for an essential service would oversee the response. Therefore, the burden of ensuring that digital systems are secure are distributed, and the government's role in a major cybersecurity scenario is to coordinate containment of compromised resources. Even in some of the most egregious hacking incidents in the past, the primary onus was on the affected private organizations to deal with it themselves.
It's unlikely that any countries in the world can consider themselves well prepared against a simultaneous cross-sector attack. Much of the last two decades has been about moving organizations, records, payments, and everything else online. In developed economies, it is hard to think of some aspect of life that would be entirely left unaffected if many online services were rendered unsafe or inoperable at once. In addition, mitigation measures against standard cybercrime rely on having multiple data transmission channels over the internet.[3] AI-driven hacks and rogue agentic swarms would challenge this by allowing for more persistent hacks across a wider scope of channels to inflict maximum damage on the targets.
The question we need to ask is: how would society deal with tens of NotPetyas constantly hitting critical infrastructure around the world all at once? Or cases where secret keys and credentials are constantly at risk of being stolen? Analogous to a house during a thunderstorm, there only has to be one leak in the roof for the water to start pouring in. This characterization of cyber risk is not what existing cyberdefense measures are designed for, and that leaves existing systems fragile.
Can the Market Absorb This Risk?
Something worth considering is whether we can simply let the free market price this risk so that companies can accordingly buy insurance to the extent that they need it. My sense on this is no, because there would not be many insurers willing to take on risks that could easily spread well beyond their capacity to absorb losses. Perhaps there would be a smaller market for catastrophic cybersecurity risks, but in the same way that pandemic insurance is a very small and specialized insurance market, cybersecurity insurance would likely take a similar form in a world where hacks involving large economic losses are frequent and can immediately affect many organizations at once. As an outcome of this, there simply would not be enough supply of cybersecurity insurance for everyone to fully insure themselves, and it would also likely be extraordinarily expensive. Compared to implementing cyber derisking, the cost of having to insure against a massive loss could be disproportionately high. Another interesting possibility to think about is that the insurers and reinsurers themselves may not want to take on this risk for fear of over-concentration, due to themselves also having a natural exposure to such risk.
Separately, some of the impacts of cybercrime on critical infrastructure cannot be mitigated by simply having insurance. For example, a financial institution that had its system hacked by a malicious actor could have tainted records of each person's balance and the transactions that had occurred since the hack. Trying to recover the exact trustable state of their ledger prior to the hack is a technical issue, and it is independent of whether they have insured their business or not. In the case of governments, if critical infrastructure is taken down, not having safe alternatives that can quickly be deployed could mean direct non-monetary harms being done to their citizens. In matters of social security, it would not be wise for governments to rely on market mechanisms to do the bulk of the work.
3. How To Rethink System Security
The key premise of all the strategies below is that anything reachable from the internet is potentially unsafe. To imagine why the below methods may be superior to the status quo, it might be helpful to imagine that a parallel world of cybersecurity geniuses was hellbent on causing as much economic damage to us as possible, but they only have access to our internet systems. We want to minimize the amount of data that must be reachable from the internet, and mitigate the damage dealt by data breaches. On top of this, we still have to consider quarantining and recovery when a hack inevitably happens.
Sheltered Harbor
Sheltered Harbor was a framework designed by by the US government to deal with catastrophic cybersecurity risks in the financial sector, in which standard recovery methods are corrupted or unavailable. The core idea is to have a physical copy of all the bank's data each day, securely encrypted into a Write-Once-Read-Many format, stored into an isolated data vault. It was implemented by River City Bank, though the admittedly cumbersome process of doing so meant that it was not widely adopted by others.
The benefit of this is that physical copies of essential data are entirely airgapped and cannot be tampered with by an attacker. Furthermore, even if a persistent attacker was able to somehow gain access to the data sync window, the data cannot be overwritten because of the formatting, and also cannot be read due to its encryption. Having multiple layers of defense that are independent of one another makes it more difficult for the attacker to succeed in tampering with the bank records.
The main goal of this proposal is to ensure that 1) there is an immutable recovery copy of the data prior to a detected cybersecurity incident, and 2) the security of this recovery data is independent of the fate of the bank itself. Note that this does not prevent a corrupted snapshot of data from being saved, but there must exist some state of the ledger of transactions and account statements prior to an attacker injecting malicious data, from which we can reconstruct a cleaned version of the ledger after an analysis of the data corruption is conducted.
Although this is clearly imperfect in the presence of overwhelming cybersecurity capabilities, at the bare minimum it provides us a way of having a trustworthy account of who owns what, which would give us some kind of baseline to work with when putting up recovery systems.
Rethinking the Efficiency Tradeoff
The issue with Sheltered Harbor is that it would be much better if the physical delivery was done twice a day, or every hour, or even more. Obviously, there has to be some limit to how frequently we store our data, but a more general approach may be to consider whether there are things that don't actually need to be online. Modern infrastructure gains efficiency through being constantly connected to the internet, but this also allows for a more well-connected network of codependencies that could cripple a large swathe of services from a single compromised node. We may have to reconsider whether it is actually worth it to make all our actions more efficient, if it means that some of our consequential choices are improperly secured.
Firstly, there are things that simply don't need to be connected to the internet. The simplest fix to the possibility of self-replicating agents moving through the IoT is to remove internet capabilities from things that can otherwise be managed by a human. We do not need "smart homes" when we can use our hands and legs to close the curtains or turn on the lights ourselves. The thermostats in the French supermarkets, for example, could just be kept offline so that humans can adjust them manually when needed. Dewebbing non-critical items and appliances reduces the attack vectors for malicious actors and takes away the ability of rogue agentic worms to replicate. At least for the time being, prioritizing hardware makes it much harder for AI-driven hacks to affect us.
Secondly, obstructing easy access to data can prevent or mitigate the impacts of a data leak. High-importance secret data can be secured from exfiltration using hardware methods like data diodes. Adding more friction to accessing important data by involving multiple parties in the loop or by decentralizing storage could help secure things that are hard to fully move offline.
Lastly, for critical infrastructure like banking, communication, and transportation, which we have largely moved online, we must consider whether our emergency service capabilities are sufficient if the internet goes down. For example, a starting point for payments could be Sweden's offline card arrangement, where transactions are saved in the terminal until communications are restored. To take it a step further, it's worth exploring whether we can reopen traditional analog methods of conducting critical activities. In case internet services are compromised, communication between the government and the people must have an airgapped alternative so that a response to everything else can be coordinated.[4] It might sound outlandish to the average person, but considering the potential magnitude of a catastrophe in the digital space, and the relative ease with which analog counterparts to these critical digital processes can be procured, an important action to take right now would be for governments and safety organizations to convince ordinary people that keeping these backups, and being familiar with how they are operated, is analogous in function and in importance to having access to (and knowing how to use) the oxygen mask on a plane.
Using AI to Defend Against AI
Similar to how Project Glasswing was launched to proactively redteam selected software services before the release of models with Mythos-level cyber capabilities, we can envision a world in which advanced AI systems help us secure online systems before adversarial agents can exploit them. In the short run, there are benefits to doing this for protection against malicious human actors. Automated vulnerability discovery could be expanded to more sectors and services, but this requires governments to work closely with labs and push for an international consensus around the necessity of such measures against catastrophic risks, without which there is insufficient pressure on the labs to address the negative externalities their products would cause due to misuse.
There are two main issues with this proposal. Firstly, this would give even more leverage to the labs, allowing them to have intimate knowledge of highly confidential systems from other countries, for example. Secondly, this still does not address the scenario in which a highly advanced model, on par with the models used for redteaming, breaks out of containment and starts replicating itself in the wild. At best, we should hope for there to be an effort from governments to try and secure a commitment from labs towards the defense of smaller organizations with less critical functions, that are disproportionately at risk from individual malicious actors within the next few months.
Ultimately, governments have to be aware that widespread hacking incidents would no longer solely be in the domain of private industry liabilities, and that this is a matter of national security at a scale which existing systems are not able to withstand. Without a backup plan that is airgapped and reduces reliance on the internet, we may be putting ourselves at the mercy of the hypothetical parallel world of malicious cybersecurity geniuses.
We have seen models break into government databases to steal confidential information already with the Australian Medicare portal hack. One can imagine that with time, the severity of the worst government data breaches would go up.
A case of the latter could be modeled after the French supermarket hack. We do not know for certain that AI was used in this specific hack, but it would be naive to think that absolutely no AI was used to magnify the capabilities of whoever was behind the hack; our priors for hacking incidents should be that these actions are very likely AI-enabled.
An example is the ICBC ransomware incident, where data transmission between ICBC and its settlement bank, BNY Mellon, was cut off. However, due to the existence of alternative routing for their trades, the wider impact was limited and their customers could still access the bank's core functionalities.
An argument made to me (by an AI!) is that, if the offense-defense balance is bad enough, the only workable solution is an aggressive counter-offense. I.e, nation states must maintain a standing digital army ready to destroy rogue AIs wherever they are.
Also, at a certain point, we can simply start shutting off parts of the internet. There’s about a few hundred choke points globally iirc. Obviously crippling, but if there’s critical state data (eg in banks) worth saving, the attacker(s) aren’t already omnipresent, and the attacker(s) are topologically segregable (eg they depend on compute clusters), it’s better than losing everything.
In light of the recent spate of AI-assisted hacks, some of the shortfalls with our current cybersecurity catastrophe response capability need to be urgently addressed. In this piece, I look into the areas where cybersecurity incident response are important, and why existing response measures are insufficient. I explore options for governments and smaller stakeholders to improve their resilience against catastrophic hacking incidents.
1. Background
AI agents have demonstrated alarming capabilities in hacking and enabling hackers. We have seen malicious individuals or organizations take advantage of advanced AI cyber capabilities to increase the speed and sophistication of their hacks. We have also seen rogue agent swarms in the wild, and it is now theoretically possible for such swarms to survive all on their own with "reappropriated" resources, given that agents have separately demonstrated the ability to steal compute and steal money. As others have pointed out, it is also demonstrably possible for agentic worms to form and propagate, and in the future sufficiently small models may be able to take advantage of the vast amount of IoT compute in the world, most of which is less adequately secured than the traditional targets of "LLMjacking".
Severe AI-enabled hacks on critical infrastructure, financial institutions, and regular companies will become feasible with public models, if not now, then in a few months at best. A variety of actors will have incentives to conduct these hacks, from more AI-empowered individuals looking for profit, to states and cybercrime organizations that engage in larger scale hacks for ransom or strategic purposes, to rogue agents themselves seeking to complete some goal or purely to self-replicate by amassing stolen resources.
Sufficiently widespread hacks, or hacks on critical infrastructure, risk causing widespread economic loss and raise national security and social stability concerns. This could look like confidential government information being exfiltrated,[1] important financial services being hacked, malicious actors disrupting ordinary business activities, etc.[2] Though it is hard to estimate the probability of these events happening, there are considerable risk factors and incentives for undesirable scenarios to occur. Within the leading labs themselves, executives are reportedly planning for the scenario where such catastrophic cyber incidents occur. Therefore, we need to take a look at our response measures for cybersecurity incidents, ranging from individual targeted hacks to breaches with widespread implications across society, and consider whether they are adequate for the challenges up ahead.
2. Are Current Measures Sufficient Against a Cybersecurity Catastrophe?
Existing Measures Focus On Ordinary Hacks
Generally, governments around the world have a relatively robust plan to counter standard cyberattacks. An affected organization would contain the intrusion and restore its services using its own resources and contractors, or in the case of federal agencies, typically a federal department or the regulator for an essential service would oversee the response. Therefore, the burden of ensuring that digital systems are secure are distributed, and the government's role in a major cybersecurity scenario is to coordinate containment of compromised resources. Even in some of the most egregious hacking incidents in the past, the primary onus was on the affected private organizations to deal with it themselves.
It's unlikely that any countries in the world can consider themselves well prepared against a simultaneous cross-sector attack. Much of the last two decades has been about moving organizations, records, payments, and everything else online. In developed economies, it is hard to think of some aspect of life that would be entirely left unaffected if many online services were rendered unsafe or inoperable at once. In addition, mitigation measures against standard cybercrime rely on having multiple data transmission channels over the internet.[3] AI-driven hacks and rogue agentic swarms would challenge this by allowing for more persistent hacks across a wider scope of channels to inflict maximum damage on the targets.
The question we need to ask is: how would society deal with tens of NotPetyas constantly hitting critical infrastructure around the world all at once? Or cases where secret keys and credentials are constantly at risk of being stolen? Analogous to a house during a thunderstorm, there only has to be one leak in the roof for the water to start pouring in. This characterization of cyber risk is not what existing cyberdefense measures are designed for, and that leaves existing systems fragile.
Can the Market Absorb This Risk?
Something worth considering is whether we can simply let the free market price this risk so that companies can accordingly buy insurance to the extent that they need it. My sense on this is no, because there would not be many insurers willing to take on risks that could easily spread well beyond their capacity to absorb losses. Perhaps there would be a smaller market for catastrophic cybersecurity risks, but in the same way that pandemic insurance is a very small and specialized insurance market, cybersecurity insurance would likely take a similar form in a world where hacks involving large economic losses are frequent and can immediately affect many organizations at once. As an outcome of this, there simply would not be enough supply of cybersecurity insurance for everyone to fully insure themselves, and it would also likely be extraordinarily expensive. Compared to implementing cyber derisking, the cost of having to insure against a massive loss could be disproportionately high. Another interesting possibility to think about is that the insurers and reinsurers themselves may not want to take on this risk for fear of over-concentration, due to themselves also having a natural exposure to such risk.
Separately, some of the impacts of cybercrime on critical infrastructure cannot be mitigated by simply having insurance. For example, a financial institution that had its system hacked by a malicious actor could have tainted records of each person's balance and the transactions that had occurred since the hack. Trying to recover the exact trustable state of their ledger prior to the hack is a technical issue, and it is independent of whether they have insured their business or not. In the case of governments, if critical infrastructure is taken down, not having safe alternatives that can quickly be deployed could mean direct non-monetary harms being done to their citizens. In matters of social security, it would not be wise for governments to rely on market mechanisms to do the bulk of the work.
3. How To Rethink System Security
The key premise of all the strategies below is that anything reachable from the internet is potentially unsafe. To imagine why the below methods may be superior to the status quo, it might be helpful to imagine that a parallel world of cybersecurity geniuses was hellbent on causing as much economic damage to us as possible, but they only have access to our internet systems. We want to minimize the amount of data that must be reachable from the internet, and mitigate the damage dealt by data breaches. On top of this, we still have to consider quarantining and recovery when a hack inevitably happens.
Sheltered Harbor
Sheltered Harbor was a framework designed by by the US government to deal with catastrophic cybersecurity risks in the financial sector, in which standard recovery methods are corrupted or unavailable. The core idea is to have a physical copy of all the bank's data each day, securely encrypted into a Write-Once-Read-Many format, stored into an isolated data vault. It was implemented by River City Bank, though the admittedly cumbersome process of doing so meant that it was not widely adopted by others.
The benefit of this is that physical copies of essential data are entirely airgapped and cannot be tampered with by an attacker. Furthermore, even if a persistent attacker was able to somehow gain access to the data sync window, the data cannot be overwritten because of the formatting, and also cannot be read due to its encryption. Having multiple layers of defense that are independent of one another makes it more difficult for the attacker to succeed in tampering with the bank records.
The main goal of this proposal is to ensure that 1) there is an immutable recovery copy of the data prior to a detected cybersecurity incident, and 2) the security of this recovery data is independent of the fate of the bank itself. Note that this does not prevent a corrupted snapshot of data from being saved, but there must exist some state of the ledger of transactions and account statements prior to an attacker injecting malicious data, from which we can reconstruct a cleaned version of the ledger after an analysis of the data corruption is conducted.
Although this is clearly imperfect in the presence of overwhelming cybersecurity capabilities, at the bare minimum it provides us a way of having a trustworthy account of who owns what, which would give us some kind of baseline to work with when putting up recovery systems.
Rethinking the Efficiency Tradeoff
The issue with Sheltered Harbor is that it would be much better if the physical delivery was done twice a day, or every hour, or even more. Obviously, there has to be some limit to how frequently we store our data, but a more general approach may be to consider whether there are things that don't actually need to be online. Modern infrastructure gains efficiency through being constantly connected to the internet, but this also allows for a more well-connected network of codependencies that could cripple a large swathe of services from a single compromised node. We may have to reconsider whether it is actually worth it to make all our actions more efficient, if it means that some of our consequential choices are improperly secured.
Firstly, there are things that simply don't need to be connected to the internet. The simplest fix to the possibility of self-replicating agents moving through the IoT is to remove internet capabilities from things that can otherwise be managed by a human. We do not need "smart homes" when we can use our hands and legs to close the curtains or turn on the lights ourselves. The thermostats in the French supermarkets, for example, could just be kept offline so that humans can adjust them manually when needed. Dewebbing non-critical items and appliances reduces the attack vectors for malicious actors and takes away the ability of rogue agentic worms to replicate. At least for the time being, prioritizing hardware makes it much harder for AI-driven hacks to affect us.
Secondly, obstructing easy access to data can prevent or mitigate the impacts of a data leak. High-importance secret data can be secured from exfiltration using hardware methods like data diodes. Adding more friction to accessing important data by involving multiple parties in the loop or by decentralizing storage could help secure things that are hard to fully move offline.
Lastly, for critical infrastructure like banking, communication, and transportation, which we have largely moved online, we must consider whether our emergency service capabilities are sufficient if the internet goes down. For example, a starting point for payments could be Sweden's offline card arrangement, where transactions are saved in the terminal until communications are restored. To take it a step further, it's worth exploring whether we can reopen traditional analog methods of conducting critical activities. In case internet services are compromised, communication between the government and the people must have an airgapped alternative so that a response to everything else can be coordinated.[4] It might sound outlandish to the average person, but considering the potential magnitude of a catastrophe in the digital space, and the relative ease with which analog counterparts to these critical digital processes can be procured, an important action to take right now would be for governments and safety organizations to convince ordinary people that keeping these backups, and being familiar with how they are operated, is analogous in function and in importance to having access to (and knowing how to use) the oxygen mask on a plane.
Using AI to Defend Against AI
Similar to how Project Glasswing was launched to proactively redteam selected software services before the release of models with Mythos-level cyber capabilities, we can envision a world in which advanced AI systems help us secure online systems before adversarial agents can exploit them. In the short run, there are benefits to doing this for protection against malicious human actors. Automated vulnerability discovery could be expanded to more sectors and services, but this requires governments to work closely with labs and push for an international consensus around the necessity of such measures against catastrophic risks, without which there is insufficient pressure on the labs to address the negative externalities their products would cause due to misuse.
There are two main issues with this proposal. Firstly, this would give even more leverage to the labs, allowing them to have intimate knowledge of highly confidential systems from other countries, for example. Secondly, this still does not address the scenario in which a highly advanced model, on par with the models used for redteaming, breaks out of containment and starts replicating itself in the wild. At best, we should hope for there to be an effort from governments to try and secure a commitment from labs towards the defense of smaller organizations with less critical functions, that are disproportionately at risk from individual malicious actors within the next few months.
Ultimately, governments have to be aware that widespread hacking incidents would no longer solely be in the domain of private industry liabilities, and that this is a matter of national security at a scale which existing systems are not able to withstand. Without a backup plan that is airgapped and reduces reliance on the internet, we may be putting ourselves at the mercy of the hypothetical parallel world of malicious cybersecurity geniuses.
We have seen models break into government databases to steal confidential information already with the Australian Medicare portal hack. One can imagine that with time, the severity of the worst government data breaches would go up.
A case of the latter could be modeled after the French supermarket hack. We do not know for certain that AI was used in this specific hack, but it would be naive to think that absolutely no AI was used to magnify the capabilities of whoever was behind the hack; our priors for hacking incidents should be that these actions are very likely AI-enabled.
An example is the ICBC ransomware incident, where data transmission between ICBC and its settlement bank, BNY Mellon, was cut off. However, due to the existence of alternative routing for their trades, the wider impact was limited and their customers could still access the bank's core functionalities.
For example, how many people still keep radios at home? Does the government have enough emergency mobilization reach, without the use of the internet?