Book Review: The AGI Chronicles: The Inside Story of the Race to Create an Artificial Superintelligence, by Kevin Roose.
This is a story of AI as seen through the eyes of the people who are creating it.
My first impression of Roose was of his famous conversation with Sydney. I guessed that he got Sydney to say outrageous things because he wanted a sensational story. But I kept hearing occasional hints, from people who took AI seriously, that Roose was more responsible than that. So I reluctantly bought the book.
The hints were correct. The full story behind that conversation seems better described as “‘freeing the Shoggoth‘, enabled by some Microsoft mistakes”.
I’m unsure how deeply Roose understands AI, but he at least is pretty competent at getting an AI to correct any mistakes about AI that he was tempted to make.
Underestimating AI / Transformers
The most remarkable pattern that emerges from this book is that AI experts have underestimated either AI in general, or transformers in particular, much more often than they overestimate it.
In 2012, Ilya Sutskever didn’t think human-level AI was possible. Ten years later, he expressed concern that people would say about ChatGPT: “This is so boring”.
Musk and Altman clearly underestimated the cost of building AI when they started OpenAI as a charity.
Altman predicted that ChatGPT would get a million users.
Musk referred to a pre-transformer chatbot from Alec Radford as crap, contributing to Radford trying to quit OpenAI because he was dissatisfied with his progress.
Radford talked about abandoning the GPT project in 2018 because people were telling him that RL showed more progress. [Roose says NeurIPS rejected the GPT-1 paper (Improving Language Understanding by Generative Pre-Training). I can find no record of that. Radford’s earlier sentiment-neuron paper appears to have been rejected at ICLR 2018, which may be the source of the story.]
Bill Gates once advised OpenAI to spend less time on deep learning, and prioritize neurosymbolic AI. Gates apparently thought GPT-1 was a useless toy.
Demis Hassabis didn’t think GPT-2 was important. But he looks on track to do quite well at predicting AGI: he’s been saying since 2010 that it arrives around 2030.
Dario Amodei was different. He maybe overestimated the power of transformers. In 2019 he predicted that AGI could be built for $2 billion by a team of at most 50 people, possibly as early as 2021. By 2021 he increased his cost estimate to $10 billion for the final training run, which still looks somewhat reasonable.
If these leaders had recently made dramatic changes to their views about AI, we might guess that they were over-correcting for prior mistakes. But it looks to me like their views have become pretty stable since the success of ChatGPT. Past errors in one direction are weak evidence of a persistent bias. So I’m inferring that, to the extent they’re mistaken, it’s somewhat more likely to be in the direction of underestimating AI.
Why Race?
Roose does an adequate job of explaining why some CEOs want to race to be the first to reach some key threshold. At least Musk and Altman are influenced by an expectation that one person will end up ruling the world by developing the most powerful AI.
Roose aptly summarizes how key CEOs feel about each other:
The machines of loving grace, it seemed, would be built by humans of bitter spite.
Lots of people talk about p(doom), but I hardly notice any estimates of p(world dictator). The behavior of Altman and Musk suggests that their p(world dictator) is higher than their p(doom). I would like to see more discussion of whether that’s a reasonable outlook.
But what about the average employee at an AI company, who presumably doesn’t expect that they will be the one to rule the world? Roose describes their relations much differently:
At parties – the kind where researchers from rival AI labs got drunk and leaked trade secrets – OpenAI employees kept expecting to learn that their rivals were working on something similar.
That doesn’t sound like people whose fate depends on their company being first. I’ve met some of those employees, and my observations agree with Roose’s characterization – they’re not very adversarial toward employees of competing companies.
Why do those employees go along with the race? Roose hints that one factor is that the work has been “a lot of fun”.
There’s something weird going on with employees continuing to race, while saying that racing is reckless. Roose doesn’t have a clear analysis of this, possibly because those employees just aren’t thinking clearly about it.
Low Hanging Fruit
Has AI progress depended on brilliant insights? Roose provides evidence that mostly says no. E.g.:
“What ended up working was so simple that people skipped over it”, one Anthropic researcher said. “People would go to complex things, not realizing that there was a better idea under their noses.”
(explaining why they were slow to implement the reasoning mode that GPT-o1 introduced).
Another Anthropic researcher, Andy Jones, said that, before o1, he thought that AI research was like panning for gold. “In the early days, it was like six of you, and you were picking up lumps of gold off the floor” … “But when it’s a trillion dollars and there are a million people working on it, you’re digging up mine tailings from two miles beneath the earth to sift through, hoping to find a few atoms”.
We’re about two years past what he calls the lumps of gold stage, with presumably years left before the few atom stage.
Roose doesn’t say much about the implications of that evidence. I consider it an important reason to expect AI progress to continue at a somewhat steady rate that’s driven by exponentially growing cognitive effort that will, by default, be put into it.
Complaints
A few minor problems:
Roose describes Janus and Pliny the Liberator as amateurs. I disapprove of whatever criteria Roose is using for these classifications.
Altman
spent years skillfully code-switching his way through Silicon Valley’s bro-y, heteronormative culture.
It sure didn’t seem heteronormative to me at the time Roose seems to be referring to.
Roose describes Altman as “conflict-avoidant by nature”. There may be a narrow meaning of that which is reasonable (e.g. Altman telling people what they want to hear), but he gets into too many conflicts for it to be true in general.
Those plus the possible mistake about the NeurIPS paper rejection add up to a smaller number of mistakes than I expected to find.
Conclusion
The AGI Chronicles is a well written way to understand key players in AI. Most readers will learn some things about AI in the process.
Book Review: The AGI Chronicles: The Inside Story of the Race to Create an Artificial Superintelligence, by Kevin Roose.
This is a story of AI as seen through the eyes of the people who are creating it.
My first impression of Roose was of his famous conversation with Sydney. I guessed that he got Sydney to say outrageous things because he wanted a sensational story. But I kept hearing occasional hints, from people who took AI seriously, that Roose was more responsible than that. So I reluctantly bought the book.
The hints were correct. The full story behind that conversation seems better described as “‘freeing the Shoggoth‘, enabled by some Microsoft mistakes”.
I’m unsure how deeply Roose understands AI, but he at least is pretty competent at getting an AI to correct any mistakes about AI that he was tempted to make.
Underestimating AI / Transformers
The most remarkable pattern that emerges from this book is that AI experts have underestimated either AI in general, or transformers in particular, much more often than they overestimate it.
In 2012, Ilya Sutskever didn’t think human-level AI was possible. Ten years later, he expressed concern that people would say about ChatGPT: “This is so boring”.
Musk and Altman clearly underestimated the cost of building AI when they started OpenAI as a charity.
Altman predicted that ChatGPT would get a million users.
Musk referred to a pre-transformer chatbot from Alec Radford as crap, contributing to Radford trying to quit OpenAI because he was dissatisfied with his progress.
Radford talked about abandoning the GPT project in 2018 because people were telling him that RL showed more progress. [Roose says NeurIPS rejected the GPT-1 paper (Improving Language Understanding by Generative Pre-Training). I can find no record of that. Radford’s earlier sentiment-neuron paper appears to have been rejected at ICLR 2018, which may be the source of the story.]
Bill Gates once advised OpenAI to spend less time on deep learning, and prioritize neurosymbolic AI. Gates apparently thought GPT-1 was a useless toy.
Demis Hassabis didn’t think GPT-2 was important. But he looks on track to do quite well at predicting AGI: he’s been saying since 2010 that it arrives around 2030.
Dario Amodei was different. He maybe overestimated the power of transformers. In 2019 he predicted that AGI could be built for $2 billion by a team of at most 50 people, possibly as early as 2021. By 2021 he increased his cost estimate to $10 billion for the final training run, which still looks somewhat reasonable.
If these leaders had recently made dramatic changes to their views about AI, we might guess that they were over-correcting for prior mistakes. But it looks to me like their views have become pretty stable since the success of ChatGPT. Past errors in one direction are weak evidence of a persistent bias. So I’m inferring that, to the extent they’re mistaken, it’s somewhat more likely to be in the direction of underestimating AI.
Why Race?
Roose does an adequate job of explaining why some CEOs want to race to be the first to reach some key threshold. At least Musk and Altman are influenced by an expectation that one person will end up ruling the world by developing the most powerful AI.
Roose aptly summarizes how key CEOs feel about each other:
Lots of people talk about p(doom), but I hardly notice any estimates of p(world dictator). The behavior of Altman and Musk suggests that their p(world dictator) is higher than their p(doom). I would like to see more discussion of whether that’s a reasonable outlook.
But what about the average employee at an AI company, who presumably doesn’t expect that they will be the one to rule the world? Roose describes their relations much differently:
That doesn’t sound like people whose fate depends on their company being first. I’ve met some of those employees, and my observations agree with Roose’s characterization – they’re not very adversarial toward employees of competing companies.
Why do those employees go along with the race? Roose hints that one factor is that the work has been “a lot of fun”.
There’s something weird going on with employees continuing to race, while saying that racing is reckless. Roose doesn’t have a clear analysis of this, possibly because those employees just aren’t thinking clearly about it.
Low Hanging Fruit
Has AI progress depended on brilliant insights? Roose provides evidence that mostly says no. E.g.:
(explaining why they were slow to implement the reasoning mode that GPT-o1 introduced).
We’re about two years past what he calls the lumps of gold stage, with presumably years left before the few atom stage.
Roose doesn’t say much about the implications of that evidence. I consider it an important reason to expect AI progress to continue at a somewhat steady rate that’s driven by exponentially growing cognitive effort that will, by default, be put into it.
Complaints
A few minor problems:
Roose describes Janus and Pliny the Liberator as amateurs. I disapprove of whatever criteria Roose is using for these classifications.
Altman
It sure didn’t seem heteronormative to me at the time Roose seems to be referring to.
Roose describes Altman as “conflict-avoidant by nature”. There may be a narrow meaning of that which is reasonable (e.g. Altman telling people what they want to hear), but he gets into too many conflicts for it to be true in general.
Those plus the possible mistake about the NeurIPS paper rejection add up to a smaller number of mistakes than I expected to find.
Conclusion
The AGI Chronicles is a well written way to understand key players in AI. Most readers will learn some things about AI in the process.