[Epistemic status: I'm trying to make my thinking legible to myself and others, rather than trying to compose something highly polished here. I think I have good reasons for saying what I say and will try to cite sources where possible, but nonetheless take it with some grains of salt. As with my last post, I am lowering my standards so that this post gets out at all.]
I'm working my way through Harvard's AI safety club (AISST)'s modified version of the BlueDot Impact AI Governance Curriculum. I'm doing this because I am pessimistic about technical alignment on current AGI timelines, and so I am trying to extend timelines by getting better at governance.
I've already taken MIT AI Alignment (MAIA)'s version of the BlueDot Alignment course (MAIA's version is not publicly available), and I've taken MIT's graduate-level deep learning course, so I'll mostly be skimming through the technical details.
The purpose of this sequence is for me to explain what I learn, so that I internalize it faster, and so that I can actively discuss with people about my takes on the readings. Reading this is not intended as a substitute for actually doing any of the above-mentioned courses, but it might give you some new ideas.
I'm already familiar with a lot of the content by osmosis and by my more technical AI safety background, so this post will probably be shorter than some of the later ones. Even though the curriculum is organized into weeks, I don't plan on doing these posts weekly. I will do them as fast as I can, given my other commitments.
I've watched this video a handful of times in the past, and since then I've gotten significantly more technically skilled at AI stuff. I'm gonna skip out on this one. If you haven't seen it, it's an excellent explainer.
Technically the curriculum only says to read the executive summary. This is once again some basic technical stuff that I mostly skimmed through. The triad it describes is compute, data, and algorithms. Those are definitely important things, now what about them? In the rest of the first section it goes on to define other stuff like "machine learning" and "supervised learning" and stuff like that.
In section 2, it talks about how the three elements of the triad can serve as levers for policymakers to control AI development. In the context of the data, we see two main focuses:
Debiasing datasets: making sure that datasets are not representing harmful biases, and especially making sure of this for high-stakes systems like those making parole decisions. This mostly seems irrelevant to current alignment and governance work, not in that it's absolutely unimportant, but in that we have bigger problems to solve on our current trajectory.
Information security: how do we secure existing large datasets that are quite valuable and potentially dangerous if misused? How do government datasets get secured, and who gets access to them. (since the government has a lot of data, this is potentially valuable, they claim. This seems plausibly right, but I don't have strong intuitions for how big the internet is relative to how big government records are. I'd guess that the internet is much bigger, but the government data has a higher density of useful information.)
In the context of algorithms, they talk about talent pipelines, visa control, and worker retraining, mostly from the context of doing capabilities research. I don't have strong priors on this, but bringing in a lot of extra technical skill seems like it will help capabilities more than it will help safety, by default. Still, weak overall opinions here.
In the context of compute, they mostly talk about supply chain regulations. This seems straightforwardly really important for regulating scaling, although I feel like they're probably missing some other parts of compute governance.
Overall, even though this paper is mostly focused on capabilities research, it talks about some useful policy levers. I think I already had a decent number of these concepts in my head, but it's good to make them more explicit.
This article shares some pretty standard graphs and quotes, mostly from Epoch, as well as one from ContextuaAI. While Epoch doesn't directly forecast AGI timelines here, I think these are still pretty important for showing that things are just continuing upwards. Seems sound and correct.
This gets into some really good stuff about chip manufacturing that I mostly didn't know before! They get really into the weeds with TSMC and NVIDIA numbers, which I won't copy here. The tl;dr is that they forecast an increase in compute of the largest training runs of about 4 OOMs by 2030, with some decent-sized uncertainty given the large number of constraints playing together. Figure 1 has a really nice explainer of their different estimates of important constraints (check it out on the website, since it's interactive and has multiple slides).
I think their mainline prediction should really account more concretely for the unprecedented economic growth that AI is going to bring over the next few years, and the unprecedented demand for AI chips that this creates by default as soon as this growth becomes widely apparent. Their predictions are being very conservative in this respect, and mostly not accounting for AI speeding up economic growth as far as I can tell, nor are they accounting (in their mainline prediction) for weird discontinuities in demand to TSMC as people realize just how big AI is becoming.
I don't know how I made it this far without much understanding of the synthetic data generation process, other than "just have the model make data." I'm a bit disappointed that this paper doesn't include synthetic data in their prediction of dataset growth, but I understand that they don't have robust ground truths to base their predictions off of, like in the other domains they investigate. However, this is another reason to suspect that they are underestimating the trends, since synthetic data will likely play an important role. Since LLMs are much better at evaluating the quality of data than generating high-quality data, they can just generate a bunch of raw synthetic data and filter it. They mention concerns of model collapse due to too much synthetic data, but once again don't incorporate this. I think that, in the age of o3 and other thinking models incentivizing companies to go hard on getting a lot of compute, it might be a lot easier to get a lot of synthetic data using their large clusters while they're not actively training models. It seems like the straightforward way to turn an excess of compute and a bottleneck of data into a balance of the two.
However, the paper doesn't predict that we'll end up in a low-data and high-compute scenario, given the other concerns about compute supply chains, but it doesn't rule that situation out either. It predicts that energy and compute will be the primary bottlenecks. I think that both of these are flexible given the economic upset that I hypothesized above. Their conclusion: the 4x training compute increase per year can likely continue until at least 2030, and labs are incentivized to do so.
Energy bottlenecks seem the tightest, and the most feasible scaling strategy is to build data centers in a lot of different places so that they draw on different power grids. This is apparently worth the latency, which seems reasonable.
Conclusion
I've heard a lot already about compute governance, talent pipelines and the like already. One thing that the last reading revealed as possibly important is energy governance of AI. Maybe people are talking about this and I'm just not hearing it, but if that's the tightest bottleneck on development, then it's a powerful lever indeed. These sources feel a bit outdated, since the landscape has massively shifted even in the time since these articles have come out (o1, then o3 and deepseek v3). I don't think we know how much compute o3 took to train, but it's giving me the impression that OpenAI pushed above the trend line in terms of all the different things we're trying to predict here, and so we have to adjust further.
I still feel like all this talk of "scaling to 2030" is a bit misguided, since I'm ready for AGI to be here sooner than that. It is, however, further evidence that we're probably not going to run out of resources before AGI.
[Epistemic status: I'm trying to make my thinking legible to myself and others, rather than trying to compose something highly polished here. I think I have good reasons for saying what I say and will try to cite sources where possible, but nonetheless take it with some grains of salt. As with my last post, I am lowering my standards so that this post gets out at all.]
I'm working my way through Harvard's AI safety club (AISST)'s modified version of the BlueDot Impact AI Governance Curriculum. I'm doing this because I am pessimistic about technical alignment on current AGI timelines, and so I am trying to extend timelines by getting better at governance.
I've already taken MIT AI Alignment (MAIA)'s version of the BlueDot Alignment course (MAIA's version is not publicly available), and I've taken MIT's graduate-level deep learning course, so I'll mostly be skimming through the technical details.
The purpose of this sequence is for me to explain what I learn, so that I internalize it faster, and so that I can actively discuss with people about my takes on the readings. Reading this is not intended as a substitute for actually doing any of the above-mentioned courses, but it might give you some new ideas.
I'm already familiar with a lot of the content by osmosis and by my more technical AI safety background, so this post will probably be shorter than some of the later ones. Even though the curriculum is organized into weeks, I don't plan on doing these posts weekly. I will do them as fast as I can, given my other commitments.
But what is a neural network? (3Blue1Brown, 2017)
I've watched this video a handful of times in the past, and since then I've gotten significantly more technically skilled at AI stuff. I'm gonna skip out on this one. If you haven't seen it, it's an excellent explainer.
The AI Triad and what it means for national security strategy (Buchanan, 2020)
Technically the curriculum only says to read the executive summary. This is once again some basic technical stuff that I mostly skimmed through. The triad it describes is compute, data, and algorithms. Those are definitely important things, now what about them? In the rest of the first section it goes on to define other stuff like "machine learning" and "supervised learning" and stuff like that.
In section 2, it talks about how the three elements of the triad can serve as levers for policymakers to control AI development. In the context of the data, we see two main focuses:
In the context of algorithms, they talk about talent pipelines, visa control, and worker retraining, mostly from the context of doing capabilities research. I don't have strong priors on this, but bringing in a lot of extra technical skill seems like it will help capabilities more than it will help safety, by default. Still, weak overall opinions here.
In the context of compute, they mostly talk about supply chain regulations. This seems straightforwardly really important for regulating scaling, although I feel like they're probably missing some other parts of compute governance.
Overall, even though this paper is mostly focused on capabilities research, it talks about some useful policy levers. I think I already had a decent number of these concepts in my head, but it's good to make them more explicit.
4 charts that show why AI progress is unlikely to slow down (Henshall, 2023)
This article shares some pretty standard graphs and quotes, mostly from Epoch, as well as one from ContextuaAI. While Epoch doesn't directly forecast AGI timelines here, I think these are still pretty important for showing that things are just continuing upwards. Seems sound and correct.
Can AI Scaling Continue Through 2030?(Sevilla et al., 2024)
This gets into some really good stuff about chip manufacturing that I mostly didn't know before! They get really into the weeds with TSMC and NVIDIA numbers, which I won't copy here. The tl;dr is that they forecast an increase in compute of the largest training runs of about 4 OOMs by 2030, with some decent-sized uncertainty given the large number of constraints playing together. Figure 1 has a really nice explainer of their different estimates of important constraints (check it out on the website, since it's interactive and has multiple slides).
I think their mainline prediction should really account more concretely for the unprecedented economic growth that AI is going to bring over the next few years, and the unprecedented demand for AI chips that this creates by default as soon as this growth becomes widely apparent. Their predictions are being very conservative in this respect, and mostly not accounting for AI speeding up economic growth as far as I can tell, nor are they accounting (in their mainline prediction) for weird discontinuities in demand to TSMC as people realize just how big AI is becoming.
I don't know how I made it this far without much understanding of the synthetic data generation process, other than "just have the model make data." I'm a bit disappointed that this paper doesn't include synthetic data in their prediction of dataset growth, but I understand that they don't have robust ground truths to base their predictions off of, like in the other domains they investigate. However, this is another reason to suspect that they are underestimating the trends, since synthetic data will likely play an important role. Since LLMs are much better at evaluating the quality of data than generating high-quality data, they can just generate a bunch of raw synthetic data and filter it. They mention concerns of model collapse due to too much synthetic data, but once again don't incorporate this. I think that, in the age of o3 and other thinking models incentivizing companies to go hard on getting a lot of compute, it might be a lot easier to get a lot of synthetic data using their large clusters while they're not actively training models. It seems like the straightforward way to turn an excess of compute and a bottleneck of data into a balance of the two.
However, the paper doesn't predict that we'll end up in a low-data and high-compute scenario, given the other concerns about compute supply chains, but it doesn't rule that situation out either. It predicts that energy and compute will be the primary bottlenecks. I think that both of these are flexible given the economic upset that I hypothesized above. Their conclusion: the 4x training compute increase per year can likely continue until at least 2030, and labs are incentivized to do so.
Energy bottlenecks seem the tightest, and the most feasible scaling strategy is to build data centers in a lot of different places so that they draw on different power grids. This is apparently worth the latency, which seems reasonable.
Conclusion
I've heard a lot already about compute governance, talent pipelines and the like already. One thing that the last reading revealed as possibly important is energy governance of AI. Maybe people are talking about this and I'm just not hearing it, but if that's the tightest bottleneck on development, then it's a powerful lever indeed. These sources feel a bit outdated, since the landscape has massively shifted even in the time since these articles have come out (o1, then o3 and deepseek v3). I don't think we know how much compute o3 took to train, but it's giving me the impression that OpenAI pushed above the trend line in terms of all the different things we're trying to predict here, and so we have to adjust further.
I still feel like all this talk of "scaling to 2030" is a bit misguided, since I'm ready for AGI to be here sooner than that. It is, however, further evidence that we're probably not going to run out of resources before AGI.