Focusing on the ASIC part specifically (I am an ASIC designer by trade): This is feasible in some form, though realistically it would not be a fully unrolled implementation like you describe (i.e. a one-to-one map from neural architecture to ASIC architecture), but rather some type of specialized GPU/TPU with additional restrictions baked into the hardware (e.g. fixed network architecture + cryptographically signed weights). A fully unrolled implementation is not feasible for frontier models because they are just too big. The real challenge is that you would have to convince hardware manufacturers to build chips that will rapidly become obsolete due to these restrictions, which are largely artificial. You can gain a lot of efficiency by fixing various architectural details like the number of layers, the dimensions of latent vectors, the KV caching strategy etc, but not by fixing the actual weights. Realistically the weights will still be streamed from memory because they are just too large.
Instead of focusing on fixed-function hardware, I think it is more practical and also more effective to focus on things like cryptographic signing (to prevent rogue takeover), remote monitoring (... (read more)
They will be doing RL on automated wetlab actions soon (if they aren't already) I suspect.
Nanotech might be on the table from there?
If they try to build even a single successful programmable universal constructor with atomic placement powers (using a bootstrap technology implemented in biological motifs)...
...then its just a question of how long the doubling time T is, such that the universal constructor can become 2^40 universal constructors after a 40*T timedelay.
If dumb slow humans with financial incentives to keep secrets are doing the oversight then you might have something that can just disassemble the floor, fall down into the water table, and escape, before any human even notices. This is an old trick from nethack that also works in minecraft!
Doing RL on math and programming (like with RLVR so far) is uniquely possible because "the domain you're using RL to explore" can itself be put inside the airgap and run super super fast (and it would take Anathem-level physics weirdness to break out of that, and Anathem's physics are probably just false).
Seeing if something compiles, or if a proof checks, is a very fast operation. If it takes more than 50 milliseconds it can probably ... (read more)
That gets you closer but it's still not enough for frontier models.
For example, the Cerebras wafer-scale engine (by far the largest ASIC in the world) has ~4T transistors, but it relies on a modular architecture to achieve that. Due to how ASICs are manufactured, wafers are generally tiled with a repeating structure that must fit within the reticle size (currently 26x33mm). That limits the design to roughly 100B transistors. You can get multiple copies of that design on the same wafer, but that doesn't help you if the implementation is fully unrolled. If you want multiple different designs, you're basically making multiple unique ASICs, which is dramatically more expensive because you don't get the same economies of scale. Especially with small runs this becomes very unappealing.
Frontier models are believed to have ~10T weights, and you need many transistors per weight. The number would depend on weight and activation bit widths and format, but I'd expect in the order of 100-1000 transistors per weight if you wanted to give each weight its own multiplier (as you would do in a fully unrolled implementation). So with this naive implementation you would need ~1P to ~10P transistors, i... (read more)
No, this estimate is for a more typical quantization (e.g. 4-bit weights, 8-bit activation). Conversion to ternary is a lossy operation, I don't think this is realistic for a frontier model.
Edit: To give you a better idea of how this works: a basic adder requires roughly 28 transistors per bit. A NxM multiplier requires roughly N M-bit additions (it's not actually implemented like that but it's a decent estimate). So, multiplying an arbitrary 4-bit weight with an 8-bit activation requires roughly 896 transistors. There are some optimizations you can do to reduce this, but they usually trade efficiency for speed, and there's extra routing overhead I'm not counting here. If the weight is fixed (value known at compile time), gate count usually drops by a factor 3x due to optimization, but that still leaves you with ~300 transistors.
For the ternary version, overhead is probably going to be larger than the naive cost of multiplication, so it's hard to make meaningful estimates.
I like the creative thinking! You'd still need a bunch of the elements of Plan A / Plan S to get this to work (e.g. an international deal, inspectors flying around to count the chips, etc.). I'm curious for more details about how the AI R&D orgs would be financed and regulated. Like, how do you stop one of the AI deployment orgs (or NVIDIA for that matter) from having a bunch of former employees with massive amounts of deployment-org equity go start a R&D company and then make models that they ship to their former employer...
and then make models that they ship to their former employer...
I suppose you could have a generalized rule that people involved in working at or running or financing AI R&D orgs aren't allowed to have a financial interest in AI deployment orgs. I feel like this sort of thing has been done with the Sarbanes-Oxley Act to control the incentives in banking so it's not a huge stretch that some degree of real financial firewalling could be done.
Cool. I must admit, it's pretty fun to envision the cyberpunk world you'd create with Plan R (or perhaps our own version of Plan S, in which existing models are still allowed to run, but no new models can be trained).
The year is 2047. For twenty years, Plan R has been in effect, the world paused on the brink of AGI. The world's best AIs are only about as capable as Mythos 6 / Astra 6.5 / etc., when it comes to AI R&D skills, because the transparent, government-monitored AI R&D orgs have really gone hard at prosaic AI safety. No neuralese, CoT more faithful than ever, plus an extra layer of j-space monitoring. Teaching the models AI R&D, bio, and persuasion skills is basically forbidden, so their skills in those domains come from generalizing from the world knowledge, video game, physical engineering, and robotic manipulation and navigation environments they are trained in. They are basically AGI now, able to do pretty much any profession except those that heavily rely on persuasion/charisma and those that heavily involve coding or AI research or those that heavily involve bio research. In those domains they are merely very good, but they've been trained to refuse and al... (read more)
For those who haven't seen it, LLMs burnt into ASICs can run shockingly fast. Here's a demo of an ASIC that can run Llama-3.1-8B (which came out only slightly more than two years ago! This chip did not take that long to make!) The numbers here are around 17,000 tokens/sec, compared to the B300's ~500 tokens/sec.
I'd guess that if capabilities did stagnate at the current level for ~years (whether due to a pause or some ceiling in ability), companies would begin to turn to ASIC inference for their frontier models for purely financial reasons, which could set up for this. I'm not sure what other worlds lead to a pursual of Plan R.
SRAM or hardcoded weights (with SRAM KV cache) doesn't really work for the biggest models. You need too many chips to fit a big model, many more than you need if you also use HBM for each chip (to store both weights and KV), at which point the networking gets infeasible. With B300s and such you can serve the biggest models batch-size-1 with no MTP to get 10x faster (by only reading the active-not-total params and KV for a single request each pass), but that's maybe 20x more expensive per token.
I might be missing something, but why is (B) relevant, given (A)? If (A) also involves the likely destruction of the controlling entity itself (as is the case with unaligned ASI), then for a rational player there is no incentive to pursue it even given (B). And if (A) doesn't harm the controlling entitiy, and just gives it control over the world, then it automatically implies (B) as well, since no one can stop it?
The analogy with nuclear weapons breaks down, because there the relevant fact was not (B), but rather that other players also got nuclear weapons, and in particular, second-strike capability / MAD.
Nuclear technology and bioweapon technology both approximate (A) and they are mostly okay because without (B), there isn't an incentive for people controlling them to push their luck on safety.
Wait, what? I thought the massive runup in nuclear weapons was primarily because of the 'unbounded claim' dynamics (first strike capability maybe means you can rule the world).
... (read more)Obviously, the humans made a mistake in letting the AI company write the IOUs to them. But once there's a system for stocks, banking, money, etc, letting people write you IOUs is the default an
Burning the weights into an ASIC would effectively make them public to anyone willing to expend with one of their purchased chips, right? They could extract the weights and then run them in their own GPUs/TPUs. One would need to encrypt the weights in the chip to avoid extraction, but that would increase the transistors per weight immensely with of the decryption circuits (or slow it down with serial decryptions). Or, one could FHE everything, but this would also come with an immense size/transistor count cost.
For me the biggest problem with this is it requires some crazy level of civilizational cooperation. Everyone has to agree to this regime of giving up 'high spec compute', among many other things that need to be hard and fast rules. Roko.claims this is easy because only China and the US are involved - but that's clearly not the case when everyone has to agree to this.
And despite that massive civilizational realignment...it doesn't actually solve that much. It makes it difficult for an AI to self replicate, unless it can take over and ASIC fab and reverse engineer itself...which seems a dubious claim. It does nothing to make sure the models are fully aligned on reaching production - the labs still have to be trusted to do this correctly.
What it mostly seems to do is throw sand in the gears, which, ok but doesn't really need this convoluted scheme. And try to make sure something like the a misaligned attack from a model cannot spread outside a lab. Though if one did happen in a lab but was not detected, nothing is stopping a misaligned AI from polluting a production model that then is burned to ASICs and gets to do whatever in the outside world.
Blocking any sort of test time t... (read more)
Firstly, can the lab repeatedly train the same model (with different seeds) to get past the copy limit? This seems like the main barrier to the research org gaining and "unbounded claim on the future"? I suppose there's also the limit on equity and maybe you think that means they'll only be able to do this at a certain rate?
This seems like a massive sacrifice and I'm not sure the safety benefits are that much given you still plan to mass-produce AIs and hand them de facto control of large chunks of the economy.
It's a nice idea in theory, but I'm skeptical that it is achievable. Especially as you describe it.
I would say this boils down to separating out the economic incentives, so that A) has room to proceed slightly slower and more safely. The ASIC question is kind of orthogonal, and I think it would not be well-received because it would devalue our massive existing infrastructure investments, and slow progress more than would be palatable.
To me, on the incentives side, your proposal is basically nationalization of compute and the formation of a gov-supervised Manhattan project, with corporate partners handling the diffusion. The AI R&D arm can't be corporate, because any way you give them revenue for funding, it brings back in the economic incentives. Debt is not realistic without revenue expectations (which would reintroduce economic incentives). Charity is insufficient. So they would need to be government financed, one way or another. You could have the deployment partners send licensing revenue back up, but that still reintroduces economic incentives.
If you could snap your fingers and make it happen, I do think it lowers p(doom). But there is a massive political boulder to be mov... (read more)
I can't wait for the ASIC phase.
I've built 3 production level harnesses and find the value to be in the tools that leverage the LLM output. A model that is burned in is (IMO) better than one that can have personality changes and blow out prompts and puke outputs.
This sounds like the way people here would talk about alignment in the late 2010s, where you look for a technical solution for AI safety that's bulletproof as long as everybody agrees to do AI a certain way.
But, well... This is not a realistic plan. People, investment funds, countries have invested literal billions of dollars into NVidia GPUs. If you come to them and say "I am the AI czar, you must now sign an international treaty that says that the only AIs you're allowed to run must be printed on single-purpose ASICs", they're going to laugh in your face.
I'm not going to go into this point by point, but it just doesn't seem grounded in reality at all.
Well, a simple rule that might help seriously blunt that risk is that the AI R&D orgs would not be allowed to use their own models
How will they benchmark the models? You could say the deployment companies will do that, but what about sensitive capabilities? You don't want to hand a cyber-capable and misaligned model to the deployment company that wants to "move fast and break things," and how can you ensure you won't do that without some kind of testing? The government will do that, perhaps?
Then are they allowed to benchmark but not use it on research? Isn't it worth determining if AI is good at frontier AI R&D? I don't necessarily oppose this idea but these are important gaps.
This is one of the more lucid plans on ai safety and alignment, although in fairness it would have to involve massive amounts of civilisational cooperation- I would not necessarily put it past humanity's abilities (see transnational agreements on nuclear warheads and their proliferation). I think currently there is only possible path for the complete execution of plan R:
So if one AI R&D org pulls ahead, inferior ASICs from other companies slow it down.
In this system, no entity gets to own an unbounded financial claim on The Singularity.
This allows America to pull ahead and gain an unbounded financial, no?
This problem could be mitigated by small adjustments, like requiring the anti-dogfooding of different/randomly chosen ASICs. But if today the world is worried because American companies are poised to gain an unbounded financial claim on the singularity, many naïve* policies end up with the world being worried because ... (read more)
Much of the civilization-scale risk we are seeing in AI in 2026 comes from the following combination: we created a single institution (the "Frontier AI Company") that has two properties:
A. It is set up to create very powerful and/or self-replicating entities that may exceed the capabilities of the entirety of the rest of civilization and come with extraordinary risks
B. It gets to own an unbounded financial claim on the resulting surplus
All the technical stuff about AI, AI alignment, etc can be rolled up into point (A) above. My claim is that having point (A) on its own, without point (B) is probably okay. Nuclear technology and bioweapon technology both approximate (A) and they are mostly okay because without (B), there isn't an incentive for people controlling them to push their luck on safety.
But with Frontier AI Companies, we mixed the two.
The key claim of this post is that we can probably get rid of most of AI risk without doing anything other than separating out the bookkeeping, physical footprint and institutions so that there is no single org with both properties. And with a little help from ASICs, maybe we can also have a very productive AI industry that actually delivers most the benefits of AI to boot. No busy-waiting pause, no "banning AI", etc.
What a typical AI disaster scenario currently looks like (e.g. those by Daniel Kokotajlo):
Obviously, the humans made a mistake in letting the AI company write the IOUs to them. But once there's a system for stocks, banking, money, etc, letting people write you IOUs is the default and it's hard to stop.
So, here's how I would separate the two. Create two new kinds of entity:
(1) - AI research and development (AI R&D) organizations.
(2) - AI deployment companies
AI R&D orgs would do everything that involves training, distilling, testing, and aligning frontier AI models. They would be heavily legally restricted and unable to issue equity. Maybe they could get debt financing, I will have to think more about this.
AI deployment companies would do all the consumer-facing stuff, or B2B work. They would run inference at scale. They would handle the UI/UX, and also anything that doesn't require gradient descent, so they could organize AI agents into swarms, attach databases, etc. Some AI deployment companies could sell AI hardware direct to consumer or integrate them into robots and laptops.
AI deployment companies would be able to issue equity and they would pay the AI research and development organizations for the models. Also of note: these AI deployment companies would potentially have a lot of intimate knowledge about the real world and real people, either via the web interfaces or perhaps customer data from robots. The AI R&D orgs would not be allowed to have that kind of data. This makes an global AI takeover from an AI R&D org much harder, because the organization as a whole simply doesn't have this goldmine of blackmail material and intel on people just sitting around for the taking.
Now fortunately the technology underlying AI allows us to enforce this distinction at a hardware level - an AI R&D org would work on a model and once they were done with it and it had passed whatever evals we deem necessary, they would burn it into an ASIC. This would be a fixed-function ASIC whose physical circuitry implements a particular model, the way a human brain has physical connections that implement a human mind. It would have some specific input-output type, such as text in/text out, or video in/motor controls out. That ASIC has some run size R units produced, and those are then sold. Then the model is deleted or encrypted and not allowed to run in the AI R&D org any more. In the most paranoid case, one could try to make a run size of R=1 or some small integer. But I am imagining runs of tens or hundreds of thousands.
All high-spec compute that exists outside the AI R&D orgs would be in the form of ASICs. So whole categories of AI risk simply disappear. There is no compute overhang that a rogue AI can capture. It is stuck with CPUs or maybe GPUs (though maybe there is a way to change how GPUs work so that they can't run AI models - more to do here).
The AI R&D orgs would have an airgapped, high security perimeter containing their general purpose GPU/TPU compute, with physically enforced multi-hour latency connections to the outside world including to their own research floor. This physically enforced latency is done in a way that makes it impossible to get data in or out in less than say 12 hours, even if you wanted to. This means that models that are undergoing training do not have a low-latency channel to the real world, which severely restricts the damage they can do, including things like the recent Hugging Face hack.
In addition to these AI R&D orgs not having equity, they would have some amount of abnormal scrutiny. Perhaps not full research transparency, but that is on the table. They might have mandatory write-once logging, for example. They should probably be completely disconnected from the internet at a physical level - not even the research floor has internet. There might be rules on what you can and cannot do during training. There might be other technologies or systems for regulating dangerous AI work. Whatever it is, it goes here.
AI Deployment orgs continue to run on Silicon Valley rules. Move fast and break things.
AI R&D orgs are run like nuclear weapons labs mixed with Global Systemically Important Banks (G-SIBs)
What about recursive self-improvement? Well, a simple rule that might help seriously blunt that risk is that the AI R&D orgs would not be allowed to use their own models - neither the versions that are running on general purpose compute, nor the ASICs that are for sale. But they are allowed to use ASICs from other AI R&D orgs. This is "Anti-dogfooding" - you are not allowed to eat your own dog food.
This would allow humanity as a whole to recursively improve its AI tech base, though of course each iteration would be limited by how quickly new models could be certified and ASICs could be made. But it would prevent any one AI company from "going nuclear" on its own, because its research floor is only allowed to use AI ASICs from other companies - which must legally also be available to the general market. So if one AI R&D org pulls ahead, inferior ASICs from other companies slow it down.
In this system, no entity gets to own an unbounded financial claim on The Singularity.
But the AI deployment companies can still use these AI ASICs to do useful work. AI is not banned. You can still have ChatGPT - it just gets served from an ASIC. People can still try to cure cancer and aging. Robots can still have AI brains, either in place or remotely. We can still use AI for military applications. But there is a very strong separation between inference and training, enforced at the hardware level. ASICs cannot compute gradients, so they cannot train. They cannot change what model they are running, so they cannot be taken over by rogue AIs. Insider risk at the AI R&D orgs is reduced because model X cannot be used to help train model X+1. You have to use someone else's model.
For additional safety, ASIC adoption by AI R&D labs could be lagged - so an ASIC must spend say 3 months working on a variety of real world tasks before it can be considered for use in an AI R&D org. This should hopefully add something to safety as real world use allows a run of ASICs to collect a track record, and it will put a speed limit on the Singularity.
An international agreement would be needed to enforce this system (call it "Plan R") globally, but since the industry currently really only exists in two countries I think that is feasible. I think this "Plan R" is much better than both "Plan S" (total AI research shutdown) and "Plan A" from the point of view of the large relevant international actors. There may nevertheless be bargaining frictions at play, but I feel like analyzing them is not appropriate here.
The philosophy behind Plan R is a sort of "AI Safety by skeuomorphism" - instead of trying to work out how to make the world safe with arbitrarily powerful and general AIs, change what an AI is so that it behaves more like a human, which the world is already set up to deal with. ASICs that lock a particular algorithm to a specific piece of hardware are more human-like than GPUs which allow a single algorithm to "grab" massive amounts of new compute at almost "instant speed".
See also:
https://www.lesswrong.com/posts/BHGoF7tPqtLo9mXFL/plan-r-diversity-escrow-and-political-rights-for-asics
https://www.lesswrong.com/posts/uwtnWvnJAEccksKNk/skeuomorphic-ai-safety-2