Review
Progress is mostly illegal or at least extremely hobbled by regulations. When people are free to innovate there is usually quite rapid progress. There are a few exceptions like fundamental physics but existing physics explains things we actually interact with fairly well except at extremely high energies. I have all sorts of very viable and productive business ideas that would help people. But they are either illegal or regulations make them tooo expensive to start up. This is the norm. If you want to improve things you need to look for areas where innovation is still doable without incurring gigantic costs.
Perhaps this is also related to how education works. Schools are designed for the average children. The smartest kids can start learning at their own speed at university. That is a lot of wasted time. If that time was spent more usefully, perhaps the new discoveries would have come at an earlier age.
Not sufficiently different, in my opinion... but I don't have enough data here, especially about other countries.
For example, I have attended a math-oriented school, but it wasn't anything like a math-maximizing school. More like an average school with 30% extra math. Most of our time was spent learning the standard curriculum, using standard textbooks.
Re: average age of authors/laureates and average team size
Are these data adjusted for demographic changes? i.e. Aging populations in most western countries, and general population growth.
We see similar patterns in the transition from Newtonian to relativistic mechanics or discursive Greek geometric algebra to symbolic Arabic equations or superstitious alchemy to physically grounded chemistry. There are thousands of other examples. It is not a general rule that all the past knowledge must be learned to create something new. Often, past knowledge is completely supplanted by a new discovery and progress can continue without increasing, and often decreasing, the necessary educational investment.
The Ptolemaic model is an extreme example. I doubt that we can actually find thousands of other huge corpus of accumulated knowledge who were later utterly trashed. Yes, students today do not learn the Ptolemaic model, because it was plain wrong. But the transition from Netwonian to relativistic mechanic is definitely not in the same reference class, since high school students today are still starting physics courses studying Netwonian mechanics, which are a very good approximation of relativistic mechanics in simplified conditions. You can't just dismiss Newton as superseded by Einstein (and I also doubt that Greek geometry is dead).
Moreover, if you sit in the frontier of knowledge developing a shiny new model, you can't just blatantly ignore the current model (even if it's wrong): that luxury will belong to future scientists. The developer of the new model must also master the old one in order to explain why the new one is better.
Several posts back I wrote about the 2020 paper “Are Ideas Getting Harder To Find?” by Nicholas Bloom, Charles Jones, John Van Reenen, and Michael Webb. In the post, I explain the paper with a metaphor: Our economy is like a car. Jones et al point out that our fuel use (R&D investment) has been growing fast for 100 years but our acceleration (productivity growth rate) hasn’t budged. They explain this by positing some inherent drag on idea exploration that gets more burdensome as we learn more. Something like inertia which makes it harder to double the speed of a car that’s already going fast than a car that starts slower.
But without knowing the underlying kinetics of science and economic growth, this inertia model is just a guess. There are lots of other explanations which are consistent with the observation of diverging fuel use and acceleration. Our car could be going up a hill, or over a rough and rocky road. Or our engine could be depreciating or using the extra fuel inefficiently. Similarly, the ideas we produce might face growing barriers before they can materialize as physical products and buildings which affect productivity. Or our institutions of science are squandering the extra resources they receive with inefficient institutional designs.
The title of my post was “Something Is Getting Harder But It's Not Finding Ideas” but I really only end up proving that Something Is Getting Harder And We Aren’t Sure What. This post gets closer to fulfilling my original promise by addressing one of the most common arguments for why ideas really are getting harder to find: the burden of knowledge.
In Favor of The Burden of Knowledge
The burden of knowledge claims that new ideas inherently get harder to find because you have to spend more time learning more old ones before you’re ready to expand the frontier.
The argument in favor of this claim is pretty intuitive and convincing. It’s hard to imagine a model of scientific progress without accumulating knowledge. Some of this accumulated knowledge is necessary for expanding the frontier, but no one starts knowing it, so you need some process of investment in education. If the cost of education increases the more you need to learn, then we’ve already created a model with the burden of knowledge.
Outside of theory we find more support for the burden of knowledge. Knowledge does seem cumulative. Our encyclopedias are much longer. Scientists today know about more species, more fundamental particles, and more archaeological artifacts than scientists in the past. Education also seems like necessary pre-requisite for expanding the frontier of knowledge. Scientists often spend decades in training before they start making original research contributions.
But to explain the observation of diverging R&D investments and productivity growth rates over the past several decades, the burden of knowledge must have increased over this time. There is empirical evidence for this too.
The average age of authors in academic journals is increasing, as is the age when Nobel laurates do their best work. This can be explained by an increasing burden of knowledge which takes more time to get through before new contributions can be made.
Source: Matt Clancy
Relatedly, more journal articles and patents are being written in larger teams. Larger teams are a response to an increasing burden of knowledge as researchers specialize in narrower topics to get to the frontier of knowledge faster and then collaborate to combine all the necessary elements for a new expansion.
Source: Matt Clancy
It is also becoming less common and less rewarding for researchers or inventors to contribute to multiple fields of study, again suggesting specialization in response to a greater burden of knowledge.
All of this evidence is parsimoniously explained with a burden of knowledge model. It’s still far from clear that these effects are enough to explain all or a significant portion of the divergence between R&D inputs and productivity growth outputs but it does tell compelling story.
There are several problems with this story though.
Knowledge is Often Not Cumulative
The central piece of the burden of knowledge model is accumulating knowledge which simultaneously represents the fruits of progress but also an obstacle for future researchers. But the history of scientific and technological progress shows countless examples where this assumption in false.
My favorite is orbital mechanics. For thousands of years the standard model for astronomical prediction was Ptolemy’s geocentric model. Earth was at the center and all the celestial bodies rotated around us. This worked well but things got complicated because, from our perspective, lots of these celestial bodies would switch directions and orbit the opposite way at various times throughout the year.
Ptolemy and his astronomical ancestors explained these “retrograde” motions with the extra loops you see in the map above called “epicycles.” By the 15th century astronomers had accumulated centuries of meticulous measurements and incorporated them into complex orbital paths, matching their observations. Learning these models and taking enough measurements to improve one of them took an entire lifetime of monastic devotion to studying the stars. The burden of knowledge was immense.
But then, Copernicus came along with the heliocentric model which, in its simplest initial form, made worse prediction than the tuned-up Ptolemaic model. But the burden of knowledge was dissolved in an instant. Improving the Copernican model meant shifting orbital paths from perfect circles to ellipses. It had nothing to do with the epicycles and perihelions of the Ptolemaic model and none of that burdensome knowledge was necessary to expand the frontier anew.
We see similar patterns in the transition from Newtonian to relativistic mechanics or discursive Greek geometric algebra to symbolic Arabic equations or superstitious alchemy to physically grounded chemistry. There are thousands of other examples. It is not a general rule that all the past knowledge must be learned to create something new. Often, past knowledge is completely supplanted by a new discovery and progress can continue without increasing, and often decreasing, the necessary educational investment.
Specialization Does Not Always Make Innovation Harder
Another prediction of the burden of knowledge is that researchers will respond by specializing in a narrower field which allows them to get to the frontier faster. This is fine, but if this specialization then requires them to collaborate with larger groups and this drags on innovation, it could explain some of the divergence between R&D inputs and productivity growth outputs.
It’s definitely true that specialization is a common response to an increased burden of knowledge. But the most common way for this specialization to manifest is not larger teams of academic scientists collaborating on a long and complicated paper. It is scientists and technologists specializing and trading for complex tools which they could not recreate themselves but which nonetheless help them expand the frontier of technology.
Consider a tech entrepreneur in the 2020s. If they want to expand the frontier of technology in software there is, in some sense, a massive burden of knowledge standing before them. The history of theoretical computing, computer engineering, all the science and engineering behind Moore’s law, networking, and the internet. But in fact, the entrepreneur does not need to invest any time in learning about these things to create a new technology using a computer. They can just buy a Macbook and offload all the burden of knowledge onto the people who make it.
Similarly, a modern scientist is highly specialized in that they use dozens of extremely complex tools which they could not recreate themselves, e.g electron microscopes, super-computers, and protein synthesizers. The burden of knowledge required to deeply understand all of these tools is far too great, so the scientist must specialize in their research and remain a mere user of these tools. But this doesn’t drag on their ability to advance knowledge, in fact it does the opposite.
The modern world has accumulated much more knowledge than at any time in the past. Specialization is a necessary response to this accumulation, but this is not a drag on innovation. Access to the fruits of this accumulated knowledge through specialization and trade accelerates progress and discovery.
The Empirical Evidence is Explained by Institutional Decay
The previous two arguments are intuitive and theoretical but they do not address the significant empirical evidence which the burden of knowledge can explain. If new fields and new tools are cutting through the burden of knowledge, why are researchers getting older, their teams getting larger, and their fields getting narrower and more permanent?
One explanation for this evidence is an increasing burden of knowledge. Idea space is just structured such that it takes more time, effort, and more narrow specialization to get to the modern frontier of knowledge.
Another explanation which equally explains our observations is that the institution of academia is depreciating. Inward looking networks of grant applicants and reviewers reward a fixed cohort of researchers that gets older every year. Risk averse funders and reviewers that reward incremental, labor intensive research. A business model based on exclusivity that requires more and more hurdles as the initial pool of applicants grows.
Institutional decay in the academy is obvious to anyone who looks and can explain our observations of aging, narrowing careers in academia. The burden of knowledge claims that these trends are inevitable responses to an unavoidable cost of progress. But an explanation based on on institutional decay suggests a solution: metascience. Redesign our academic institutions and we can reverse some of these trends.
The burden of knowledge is an intuitive explanation for ideas getting harder to find. Even extremely simple models of scientific progress have the burden of knowledge in them and there is some suggestive empirical evidence for the burden of knowledge in the real world. But new fields and new tools cut against the burden of knowledge. It is unlikely both theoretically and empirically that the burden of knowledge explains a significant portion of the divergence between R&D inputs and productivity growth outputs that “Are Ideas Getting Harder To Find?” points out.