Review
This is a linkpost for https://www.semianalysis.com/p/google-we-have-no-moat-and-neither
My prior, not having looked too carefully at the post or the specific projects involved, is that probably any claims that an open source model is 90% as good as GPT4 or indistinguishable are hugely exaggerated or otherwise not a fair comparison. In general in ML, confirmation bias and overclaiming is very common and as a base rate the vast majority of papers that claim some kind of groundbreaking result end up just never having any real impact.
Also, I expect facets of capabilities progress most relevant to existential risk will be especially constrained strongly by base model quality. I would agree that open source is probably better at squeezing stuff out of small models, but my model is that wrt existential risk relevant capabilities progress this is less relevant (cf the bitter lesson).
I have. It seems pretty good (not obviously worse than ChatGPT 3.5) at short conversational prompts, haven't tried technical or reasoning tasks.
Are you implying that it is close to GPT-4 level? If yes, it is clearly wrong. Especially in regards to code: everything (maybe except StarCoder which was released literally yesterday) is worse than GPT-3.5, and much worse than GPT-4.
I've tried StarCoder recently, though, and it's pretty impressive. I haven't yet tried to really stress-test it, but at the very least it can generate basic code with a parameter count way lower than Copilot's.
Does the code it writes work? ChatGPT can usually write a working Python module on its first try, and can make adjustments or fix bugs if you ask it to. All the local models I've tried so far could not keep it coherent for something that long. In one case it even tried to close a couple of Python blocks with curly braces. Maybe I'm just using the wrong settings.
I feel like the title of this post is slightly misleading. It's not like there was any kind of official Google statement saying they have no moat. It was some random Google Researchers whose private document leaked.
A better title would be "Unknown Google researcher says Google has no moat, and neither does OpenAI". I think people seem to be taking this a lot more seriously because they don't know who wrote it, and are treating it as some kind of consensus within Google.
Thanks for making this comment. I had a similar comment in mind. You're right nobody should assume any statements in this document represent the viewpoint of Google, or any of its subsidiaries, like DeepMind, or any department therein. Neither should be assumed that the researcher(s) who authored or leaked this document are department or project leads. The Substack post only mentions that a researcher leaked the document, not that any researcher authored it. The document could've been written up by one or more Google staffers who aren't directly doing the research themselves, like a project manager or a research assistant.
On the other hand, there isn't enough information to assume it was only one or more "random" staffers at Google. Again, nothing in the document should necessarily be taken as representative of Google, or any particular department, though the value of any insights drawn from the document could vary based on what AI research project(s)/department the authors of the document work on/in.
That might not be a useful question to puzzle over much, since we could easily never find out who the anonymous author(s) of the document is/are. Yet that the chance the authors aren't purely "random" researchers should still also be kept in mind.
Firstly, it suggests that open-source models are improving rapidly because people are able to iterate on top of each other's improvements and try out a much larger number of experiments than a small team at a single company possibly could.
Widely, does this come as a surprise? I recall back to the GPT2 days where the 4chan and Twitter users of AIDungeon discovered various prompting techniques we use today. More access means more people trying more things, and this should already be our base case because of how open participation in open source has advanced and improved OSS projects.
I'm worried that up until now, this community has been too focused on the threat of big companies pushing capabilities ahead and not focused enough on the threat posed by open-source AI. I would love to see more discussions of regulations in order to mitigate this risk. I suspect it would be possible to significantly hamper these projects by making the developers of these projects potentially liable for any resulting misuse.
I have no idea how you think this would work.
First, any attempt at weakening liability waivers will cause immediate opposition by the entire software industry. (I don't even know under what legal theory of liability this would even operate.) Remember under American law, code is free speech. So...second, in the case you're somehow (somehow!) able to pass something (while there's a politicized and deadlocked legislature) where a coalition that includes the entire tech industry is lobbying against it and there isn't an immediate prior restraint to speech challenge...what do you think you're going to do? Go after the mostly anonymous model trainers? A lot of these people are random Joe Schmoes with no assets. Some of the SD model trainers which aren't anonymous already have shell corporations set up, both to shield their real identities and to preemptively tank liability in case of artist nuisance lawsuits.
The fact that they can squeeze more performance out of models with smaller datasets and less compute points to a current computing overhang, which reduces the odds that Altman's strategy of short timelines/slow takeoff can be done safely.
Aren't a lot of the doomy scenarios predicated on a single monolithic AI though? (Or multipolar AI that all agree to work togrther for naughtiness for some reason)
A bunch of them being tinkered on by lots of people seems like an easier path to alignment and as a failsafe in terms of power distribution.
You have lots of smaller scale dangers introduced but they certaintly dont seem to me to rise to the level of x or s risk in the near term.
What have we had thus far? A bunch of think tanks using deductive reasoning with no access to good models and a few monoliths with all the access. Seems to me that having the capability to actually run experiments at a community level will neccesarily boost efforts on alignment and value loading more than it assists actual AGI being born.
This reminds me of stories (maybe exaggerated) about how Microsoft became internally terrified of GPLv3 and Linux in the late 90s and early 2000s.
Yeah, I honestly expected open-source to win out by now.
My guess would be:
a) Programmers are more enthusiastic about hacking for free than designers are about designing for free
b) There's no-one to do the boring tasks that need to be done
I think the more basic question is: why has open source software even gotten to the point that it has? Actually running a startup or a software company is really tough. To do it competently requires (as a starting point) constantly swallowing lots of bitter lessons about what your customers actually care about vs. what is technically interesting. Why expect open source software developers to do all of that hard work for free?
You can think of it as “dangerous capabilities in everyone’s hands”, but I prefer to think of it as “everyone in the world can work on alignment in a hands-on way, and millions of people are exposed to the problem in a much more intuitive and real way than we ever foresaw”.
Ordinary people without PhDs are learning what capabilities and limitations LLMs have. They are learning what capabilities you can and cannot trust an LLM with. They are coming up with creative jailbreaks we never thought of. And they’re doing so with toy models that don’t have superhuman powers of reasoning, and don’t pose X-risks.
It was always hubris to think only a small sect of people in the SF bay area could be trusted with the reins to AI. I’ve never been one to bet against human ingenuity, and I’m not about to bet against them now that I’ve seen the open source community use LLaMa to blaze past every tech company.
"Everyone in the world can work on alignment in a hands-on way" only helps us if they actually do choose to work on alignment, instead of capabilities and further accelerating the problem. Right now, people seem to be heavily focusing on the latter and not the former, and I expect it to continue that way.
I would like to turn that around and suggest that what is hubristic is a small, technical segment of the population imposing these risks on the rest of the population without them ever having had any say in it.
Except "aligned" AI (or at least corrugibility) benefits folks who are doing even shady things (say trying to scam people)
So any gains in those areas that are easily implemented will be widely spread and quickly.
And altruistic individuals already use their own compute and GPU for things like seti@home (if youre old enough to remember) and to protein folding projects for medical research. Those same people will become aware of AI safety and do the same and maybe more.
The cats out of the bag , you can't "regulate" AI use at home , I can run models on a smartphone.
What we can do is try and steer things toward a beneficial nash equilibrium.
I guess where we differ is I don't believe we get an equilibrium.
I believe that the offense-defense balance will strongly favour the attacker, such that one malicious actor could cause vast damage. One illustrative example: it doesn't matter how many security holes you patch, an attacker only needs one to compromise your system. A second: it is much easier to cause a pandemic than to defend against one.
I think this speaks of a move from focusing on building the largest models possible, to optimising smaller models for mundane utility. Since smaller models have a hard ceiling on capabilities, this is good news as it pushes off AGI.
I have a list of specific technical advances in algorithms/architecture that I suspect will push these open-source LLMs further in capabilities when integrated. Especially increasing effective agency & strategic planning. I think these advances, if like 80% of them are discovered and are as successful as I expect, are enough with like a 65 billion or so param LLM, to begin a successful RSI process. Slow at first, but able to accelerate since I still think there's a lot of algorithmic efficiency gain sitting on the table. What I don't know is what to do about that scenario. If this is the case, then the world is going to look pretty crazy a couple years from now.
The Substack post only mentions that a researcher leaked the document, not that any researcher authored it. The document could've been written up by one or more Google staffers who aren't directly doing the research themselves, like a project manager or a research assistant.
Nothing in the document should necessarily be taken as representative of Google, or any particular department, though the value of any insights drawn from the document could vary based on what AI research project(s)/department the authors of the document work on/in. This document is scant evidence in any direction of how representative the statements made are of Google and its leadership, or any of the teams or leaders of any particular projects or departments at Google focused on the relevant approaches to AI research.
This gives me the title for season 1 episode 1 of Uncanny Valley, the spinoff of Silicon Valley focused on AI alignment: "I Have No Moat and I Must Scream". (It's a Harlan Ellison reference.)
I think this is absolutely accurate and here's why:
1. open-source will surely beat the big guys. Openai and google do NOT have a monopoly on smart people, there's far more of us that do not work for them. Any smart person who is not allowed on the winning team will simply make their own team win. Therefore openAI and google will be left in the dust in 2 years flat. And I don't see anything stopping researchers from leaving and then just giving their truly awesome ideas to open-source after they take home the big bucks.
2. They have no moat, at least not one that is going to be there for long.
- First mover advantage is not an advantage in every game. America has been king the world ever since they joined WW2 late. And the first movers got the reputation they deserved. As they say, sometimes it's better to wait and let the competition draw a bit of blood before you jump into the fray.
- data moat? nope we got open data. network effects moat? nope, we got github, technology moat? apparently not a very thick one. Entrenched customers? yeah for openAI. But once you're paying for multiple Ai subscriptions and you have to get rid of one... maybe it's byebye openai.
The only moat they have is that they have a lot of money to pay researchers. And they can pour millions into training models. So if they can win by spending more money then that's how they can do it. OpenAI can win by building on top of their user base. But openAI shows no signs of doing that, they are just becoming microsofts toy, and google can pour as much money as they want wherever they want it's not going to buy them something better than open source.
⚫ What is the cost of even Cat level A.I. (CLAI)?
The Information had a good scoop recently. They said recently:
⚫ OpenAI’s losses roughly doubled to around $540 million last year as it developed ChatGPT and hired key employees from Google, according to three people with knowledge of the startup’s financials.
The previously unreported figure reflects the steep costs of training its machine-learning models during the period before it started selling access to the chatbot. Furthermore reports online indicate that ChatGPT costs approximately $700,000 a day just to run, or more.
Even as revenue has picked up—reaching an annual pace of hundreds of millions of dollars just weeks after OpenAI launched a paid version of the chatbot in February—those costs are likely to keep rising as more customers use its artificial intelligence technology and the company trains future versions of the software.
⚫ Reflecting that capital drain, CEO Sam Altman has privately suggested OpenAI may try to raise as much as $100 billion in the coming years to achieve its aim of developing artificial general intelligence that is advanced enough to improve its own.
⚫ A leak out of Google claims that they have no moat against open-source LLMs that could one day overtake them. (Read semianalysis on Sustack for this scoop). simonwillison and LessWrong have blogged about this too.
What is clear is few of even the best Generative A.I. startups today, will even be around in 2030. They will have failed to make money. Which makes all the hype and hubris a bit of a head fake.
Microsoft's investment in OpenAI isn't likely to age well given the pace of innovation in LLMs. Very few companies or even Nation states can easily bear the costs of developing this stuff and they will get disrupted by new tech here rather easily. That won't be a problem for the Beijing Academy of Artificial Intelligence (BAAI).
Furthermore reports online indicate that ChatGPT costs approximately $700,000 a day just to run, or more.
If there are two million subscribers for ChatGPT plus that would pay for those $700,000 a day.
This (purported) leak is incredibly scary if you're worried about AI risk due to AI capabilities developing too fast.
Firstly, it suggests that open-source models are improving rapidly because people are able to iterate on top of each other's improvements and try out a much larger number of experiments than a small team at a single company possibly could.
Secondly, it suggests that data size is less important than you might think from recent scaling laws, as you can achieve high performance with "small, highly curated datasets". This suggests a world where dangerous capabilities are much further distributed.
Thirdly, it proposes that Google should open-source its AI technology. Obviously, this is only the opinion of one researcher and it currently seems unlikely to occur, but if Google was to pursue this path, this would lead to a significant shortening of timelines.
I'm worried that up until now, this community has been too focused on the threat of big companies pushing capabilities ahead and not focused enough on the threat posed by open-source AI. I would love to see more discussions of regulations in order to mitigate this risk. I suspect it would be possible to significantly hamper these projects by making the developers of these projects potentially liable for any resulting misuse.