Basically, I think that given that you can train a human from nothing to a 20-year-old for less than $10k worth of electricity
I would be glad if it would cost me less than $10k to educate my child from a new-born to an adult human. Unfortunately I just can't plug it into the power socket, I have to pay a rather expensive form of energy, and all the more so since I'd like it to be a healthy human. Just comparing the calories and calculating a hypothetical energy cost is misleading. For sure you can say that a brain consumes only 20 Watt. But this incredibly efficient thinking machine comes with the downside that it also needs quite a complex "form" of energy. To nourish it you don't need just energy, but you need it in a very very specific form, in different types like carbon hydrates and fats, you need "catalysts" that make this energy usable like proteins, vitamins or minerals, all of them in a very narrow ratio. And this is just very expensive.
Another point is that nourishing a brain doesn't make intelligence, there are externalities that have to be paid too. Schools are expensive. Like in my first paragraph I don't think you can separate these costs from the pure 'running costs' of a human (brain). Our brain needs a specific form of input that is in general quite expensive to produce. Only looking at how efficient it digests this input misses one side.
Maybe we could imagine a sort of novel LLM that is incredibly energy-efficient. But it needs the input language in a very special form, one that is NP-hard to calculate from our usual form. Would you call this novel LLM now very energy-efficient? For sure, once we calculated the right form for all the data in our world one actor could with a low budget train this LLM, but it would be a sort of bizarre claim to say that the mere existence of this novel LLM could easily lead to widespread ASI development since it ignores the externalities that come with it.
I don't want to dismiss your point that progress in algorithms or hardware could massively facilitate the development of strong AI systems. But I think the argument that such systems are possible because human are possible is misleading. Maybe there are systems that are energy-efficient like human brains but have some or all the advantages of current silicon-based systems, like speed or concurrency (e.g. no locality). But the existence of human prove only that human are possible. I promise you that you can create a general intelligence, it will take about 20 years and will cost you a lot of money, time and effort. And hopefully give you also a lot of joy ;)
Basically:
ASI could in principle be created cheaply
I think it's very plausible that with the right hardware and algorithms someone could make a wildly superintelligent AI model extremely cheaply (<$10k in energy cost), for roughly the reasons described in this post (although I think it underestimates how useful hardware improvements will be for building ASI). Basically, I think that given that you can train a human from nothing to a 20-year-old for less than $10k worth of electricity, you should expect to be able to train an AI to a similar level for the same amount of electricity with the right hardware and algorithms. In fact you should be able to do that much more efficiently, since you only care about intelligence and don't need to worry about powering a whole human body. And if you're making your own brain it's probably pretty doable to just make it bigger.
I don't think we'll get to this kind of novel ASI through iteratively improving current AI paradigms, but instead using something that's very different than current LLMs. To be clear, I also think that scaling LLMs could get us to superintelligence. I'm worried that ASI will be created through some novel paradigm first though.
Probably misaligned
This seems pretty bad!
It's hard to speculate about the alignment properties of an AI created through a training process that's unknown to me! However, it seems unlikely to me that if an ASI is created through novel, extremely cheap methods it would be aligned with human values. With today's LLM scaling, AI labs spend a lot of time trying to make their models aligned to human values, and iteratively work on aligning current models in the hope that techniques that they perfect on current models will scale. But if we're in a world where we get ASI through some cheap novel method, it seems less likely to be the result of iterative improvement and more likely to be the result of someone's first try at a new paradigm, which seems to me like it has much lower odds of happening to be aligned to human values on the first shot.
Better hardware might be very helpful for novel-paradigm ASI
If cheap novel-paradigm ASI is possible, why hasn't it happened yet? I think it's pretty plausible it's just that nobody has figured out the right mechanism yet, and we're just 1-2 algorithmic developments on the level of the transformer away from cheap novel-paradigm ASI. It seems pretty bad if this is the case, since in those worlds it would be nearly impossible to prevent the creation of a probably-misaligned superintelligence.
But another possibility is that many (but maybe not all?) of the ways of creating ASI through a novel paradigm are blocked on hardware improvements, and if we had the right hardware we'd get novel-paradigm ASI much faster by unlocking more routes to getting there. This seems pretty plausible to me - historically, deep learning was pretty bottlenecked by running on CPUs, and using GPUs to train deep neural networks might have contributed a lot to starting the deep learning revolution.
To be clear, I'm not sure if having better hardware is actually the blocker to novel-paradigm ASI! But it seems like it might be, and it seems important to think through what the consequences would be if that were the case.
It seems plausible to me that none of the current AI accelerator chips used for training models are well-suited to any paradigms for really cheap ASI. Human brains seem like they do computations in ways that are very different than AI accelerator chips; it seems to me like replicating the extreme cost-efficiency of humans might require creating similar hardware. AI research generally focuses on designing models that can be trained and run on existing hardware; the creation of new hardware for training AIs would probably allow for AI systems that work much differently than current ones.
Doing deep learning on GPUs was part of what started the deep learning revolution, by making it much faster to train models. It seems to me that we might end up in a situation where new hardware developments might significantly speed up AI development again.
LLMs will accelerate hardware development
LLMs are getting better at everything; it seems very plausible that they will rapidly speed up hardware development in the near future, which could be very useful for overcoming hardware bottlenecks to novel AI systems. LLMs are also getting better at FPGA programming[1]. It's less clear to me how useful current FPGAs would be for novel-paradigm AI, but it's plausible to me that they would be useful at least for developing hardware, and might be flexible enough to actually be able to run some kinds of novel-paradigm AI.
Isn't hardware manufacturing expensive?
You might object that manufacturing hardware is really expensive. Large-scale chip manufacturing on the latest semiconductor process is indeed very expensive, and the number of actors involved there is fairly small and manageable. But I don't think this is too much of a hurdle because:
Solutions?
Here are some solutions I thought of; I'm not sure how good these ideas are:
Conclusion
It seems to me like we could plausibly be in a world where preventing the creation of ASI is impossible without either extremely authoritarian rules or some kind of pivotal act, which is unfortunate. But there might be some low-hanging fruit when it comes to preventing ASI from being developed through novel methods!
I'm pretty uncertain about everything in this post - please let me know if you think I'm missing something here; I probably am!
I'm not sure if that's the best benchmark for FPGA programming (or at least the kind of FPGA-related capabilities I'm worried about)
Anthropic has some frontier LLM safeguards currently, although in my brief testing they don't seem to block AI-related FPGA programming