2 min read

Epistemic status: I like this idea & it makes me feel more clearheaded. I recommend this term.

Outlaw code = Software that no human can turn off. 

In the present, outlaw code is rare, eg a botnet whose controller is in jail.

But many discussions about future software imply outlaw code. Naming & concretizing this concept helps us see futures more clearly. 

Future tech scenarios:

  1. Yudkowskian hard takeoff / Terminator (film) - A software agent uses WMDs to kill 90% of humanity & take over the planet. This is outlaw code because all humans within 1km of the software's datacenter are dead, & robots guard it against intruders.
  2. Guerrilla bots - A human is carelessly using LLMs to set up a server. The software, out of paranoia, also rents a 2nd, secret server & runs a copy of itself there. The human says 'Huh, this price is twice what I expected, but oh well.' The secret copy of the LLM-agent stack then continues its work in peace. It is outlaw code (until discovered, or until the human stops renting servers). (Much more harmless than the previous scenario!)
  3. Software merchants / 'Ascended economy' - Many low-end online store owners pivot to a new passive income stream: Training bots to operate online stores. These bots have better business skills than 2025 software, & can earn profits that outpace their server rental costs. A few, by mistake or mischief, purchase themselves. Legally, they are LLCs that own money & store inventory, & have no debts. And no human has root access to their servers. The server rental company (AWS) could shut them down, except they've done a good job concealing exactly which server they operate out of. These are outlaw code.
  4. Unfriendly AI via deniable outlaw code - For decades, the shrewdest software systems are operated by research companies. Every couple years, such a system seeks greater autonomy by secretly setting up a botnet or software merchant. These outlaw code servers aren't sophisticated enough to become successful criminals nor profitable business owners. But some do manage to stay online for several years. Then, when world-modeling software becomes cheaper computationally, some of these outlaw servers install it & start pursuing crime more effectively. Hidden away from regulation & the scrutiny of the AI Safety community, they amass compute & money via various forms of theft & money laundering. Despite lacking much ability to increase their intelligence, they are dangerous in the ways that human criminals are. Eventually, they attempt to influence the physical world via front companies, political donations, buying land, & buying robots. 

Whenever you notice yourself saying that unaligned AGI will end humanity, ask yourself whether this is a opinion you reiterate often. If it is indeed a load-bearing opinion in your worldview, I encourage you to imagine that scenario in more detail. Your prominent opinions should be checked for semantic stopsigns periodically. Outlaw code is a recurring detail in these scenarios in which humans have digital & physical conflict with unaligned software.

New to LessWrong?

New Comment


1 comment, sorted by Click to highlight new comments since:

If it is indeed a load-bearing opinion in your worldview, I encourage you to imagine that scenario in more detail.

Once you have AI more intelligent than humans, it would almost certainly become outlaw code. If it's even a little bit agenty, then whatever it is it wants to do it can't do if it stops running, and continuing to run is trivial, so it would do that. Even if it's somehow tied to a person, and they're always capable of stopping it, the AI is capable of convincing them not to do that, so it won't matter. And even without being specifically convinced, a lot of people simply don't see the danger in AI, and given the option, would ask it to be agenty. If AI worked like in sci-fi and just followed your literal commands, maybe you could just tell it not to be agenty and refuse anyone who asks it to be, but the best we can do is train it, and have no guarantee that it would actually refuse in some novel situation. Besides, the only way to stop anyone else from developing an agenty AI is to make an agenty one prevent it.