TLDR:

1. Around Einstein-level, relatively small changes in intelligence can lead to large changes in what one is capable to accomplish.
   1. E.g. Einstein was a bit better than the other best physi at seeing deep connections and reasoning, but was able to accomplish much more in terms of impressive scientific output.
2. There are architectures where small changes can have significant effects on intelligence.
   1. E.g. small changes in human-brain-hyperparameters: Einstein’s brain didn’t need to be trained on 3x the compute than normal physics professors for him to become much better at forming deep understanding, even without intelligence improving intelligence.

Einstein and the heavytail of human intelligence

1905 is often described as the "annus mirabilis" of Albert Einstein. He founded quantum physics by postulating the existence of (light) quanta, explained Brownian motion, introduced the special relativity theory and derived E=mc² from it. All of this. In one year. While having a full-time job in the Swiss patent office.

With the exception of John von Neumann, we’d say those discoveries alone seem more than any other scientist of the 20th century achieved in their lifetime (though it's debatable). 

Though perhaps even more impressive is that Einstein was able to derive general relativity.

Einstein was often so far ahead of his time that even years after he published his theories the majority of physicists rejected them because they couldn’t understand them, sometimes even though there was experimental evidence favoring Einstein's theories. After solving the greatest open physics problems at the time in 1905, he continued working in the patent office until 1908, since the universities were too slow on the uptake to hire him earlier.

Example for how far ahead of his time Einstein was: Deriving the theory of light quanta

The following section is based on parts of the 8th chapter of “Surfaces and Essences” by Douglas Hofstadter. For an analysis of some of Einstein's discoveries, which show how far ahead of his time he was, I can recommend reading it.

 At the time, one of the biggest problems in physics was the “Blackbody spectrum”, which describes the spectrum of electromagnetic wavelengths emitted by a Blackbody. The problem with it was that the emitted spectrum was not explainable by known physics. Einstein achieved a breakthrough by considering light not just as a wave, but also as light quanta. Although this idea sufficiently explained the Blackbody spectrum, physicists (at least almost) unanimously rejected it. The fight between the “light is corpuscles” and “light is a wave” faction had been decided a century ago, with a clear victory for the “wave” faction. 

Being aware of these possible doubts, Einstein proposed three experiments to prove his idea, one of which was the photoelectric effect. In the following years, Robert Millikan carried out various experiments on the photoelectric effect, which all confirmed Einstein’s predictions. Still, Millikan insisted that the light-quanta theory had no theoretical basis and even falsely claimed that Einstein himself did not believe in his idea anymore.

From Surfaces and Essences (p.611): 

To add insult to injury, although the 1921 Nobel Prize in Physics was awarded to Albert Einstein, it was not for his theory of light quanta but “for his discovery of the law of the photoelectric effect”. Weirdly, in the citation there was no mention of the ideas behind that law, since no one on the Nobel Committee (or in all of physics) believed in them! ^[1][...] And thus Albert Einstein’s revolutionary ideas on the nature of light, that most fundamental and all-pervading of natural phenomena, were not what won him the only Nobel Prize that he would ever receive; instead, it was just his little equation concerning the infinitely less significant photoelectric effect. It’s as if the highly discriminating Guide Michelin, in awarding its tiptop rank of three stars to Albert’s Auberge, had systematically ignored its chef’s consistently marvelous five-course meals and had cited merely the fact that the Auberge serves very fine coffee afterwards.

Concluding thoughts on Einstein

Einstein was able to reason through very complex arguments he constructed via thought experiments without making a mistake. He was able to generalize extremely well from other physics discoveries, to get a sense of the underlying nature of physical law. I believe that what enabled Einstein to make key discoveries much faster than the whole remaining field of theoretical physics combined (which itself contained many of the smartest people at the time) was that he was smarter in some dimensions of intelligence than all other 20th century scientists (rather than him just being born with good physics-particular intuitions).^[2][3]

Takeaways

1. Capabilities are likely to cascade once you get to Einstein-level intelligence, not just because an AI will likely be able to form a good understanding of how it works and use this to optimize itself to become smarter^[4][5], but also because it empirically seems to be the case that when you’re slightly better than all other humans at stuff like seeing deep connections between phenomena, this can enable you to solve hard tasks like particular research problems much much faster (as the example of Einstein suggests).
   1. Aka: Around Einstein-level, relatively small changes in intelligence can lead to large changes in what one is capable to accomplish.
2. For human brains, small changes in hyperparameters can lead to very significant increases in intelligence.^[6] Intuitively, one would suspect that scaling up training compute by 2x is a significantly larger change than than having a +6.4std hyperparameter sample instead of a +5.4std one, even though it is not obvious to me that 2x training compute would get you from "great physics professor" to "Einstein" if we had transformer architectures. So either there is some grokking cascade around genius level intelligence where capabilities can quickly be learned and improved, or it's just that (human) brains scale significantly faster in performance than transformers currently seem to.
   1. Aka: For at least some architectures, around genius-level, small changes in hyperparameters (or perhaps also compute) can lead to relatively large changes in intelligence.
3. Compute-based AI capability forecasting is unlikely to work well, since this entirely neglects the significant intelligence gap between Einstein and average humans.

Requests to AI researchers

Nobody currently knows how to align strongly superhumanly smart AIs to human interests, and we need way more time to solve this problem. Making incremental progress on AI capabilities is shortening the timeline we have left to figure out how to align AI and is thus making human extinction more likely. Thus by far the best action is to stop advancing AI capabilities.

Absent this, please be aware that capabilities might rapidly cascade around genius or supergenius level intelligence and take measures accordingly. In particular:

1. Monitor how quickly performance of an AI is improving in training.
2. When capability is performing unusually quickly: stop and audit.
   1. Do not ignore warning signs. If warning signs show up, stop training and coordinate with governments and other AI labs to get more time to solve the alignment problem.
   2. If the audit is fine, scale up slowly and continue to carefully audit unusual training dynamics.
3. Generally perform regular and precise safety audits while scaling up.
4. Be especially careful when scaling up new architectures or training setups. There likely exist architectures which scale much faster than transformers and might reach superhuman intelligence without needing nearly as much compute as the current best models.

1. ^{^}

   I am not confident that the doubts of Einstein's light quanta-theory in 1921 were as big as portrayed here. Still: Millikan's work, in which he wrote the above-mentioned false claims, was published in 1917, so it's reasonable that 4 years later there were still some confusions. Though the doubts (at least mostly) ended in 1923 with the discovery of Compton's effect.

2. ^{^}

   The fact that human intelligence is very heavy-tailed can also be observed in other examples like e.g. John von Neumann.

3. ^{^}

   One natural hypothesis that could explain large changes in capability from small changes in hyperparameters is that the small changes enabled the agent to make itself smarter (and then smarter again, though with the improvements getting smaller so it's below the threshold where it fooms). But this does NOT seem to be the driving factor which made Einstein able to accomplish so much more. Thus this post is warning about other kinds of capability cascades which seem to exist.

4. ^{^}

   We think intelligence improving intelligence is an important part of why we at some point expect a fast takeoff (though until then capabilities might continue to improve continuously for qutie a while). This post is showing that there is empirical evidence which suggest rapid capability gain might happen even without intelligence improving intelligence. Though it is plausible to us that intelligence improving intelligence is the more important factor, and at least for AIs significantly smarter than Einstein this seems likely.

5. ^{^}

   It seems plausible that AIs will be able to significantly improve themselves or speed up AI research before they are fully as smart as Einstein in all dimensions.

6. ^{^}

   The following seems plausible (but by no means close to certain): “The base architecture of the human brain is very capable, as capable as Einstein was or even more, but evolution didn’t figure out how to align humans, who are very smart in some dimensions, to optimize well for genetic fitness. Thus, people who were e.g. extraordinarily reflective had less kids in the ancestral environment, so most people today have some alignment-patches, which evolution designed into them, which nerf their intelligence (in particular dimensions). Part of the explanation for why Einstein was so smart was that he had unusually few alignment-patches that nerfed his brain. So the existence of Einstein isn’t strong evidence that some hyperparamter changes can lead to very rapid capability increases if the base architecture isn’t nerfed and actually already more capable.”. This might be true, but I still find it very surprising under this hypothesis that Einstein (and John von Neumann) was so much smarter than many of the next runner-ups who also had few alignment-patches. The point that seemingly small increases in some dimensions of intelligence at Einstein level can have huge effects on capability still carries.