Firstly, it suggests that open-source models are improving rapidly because people are able to iterate on top of each other's improvements and try out a much larger number of experiments than a small team at a single company possibly could.

Widely, does this come as a surprise? I recall back to the GPT2 days where the 4chan and Twitter users of AIDungeon discovered various prompting techniques we use today. More access means more people trying more things, and this should already be our base case because of how open participation in open source has advanced and improved OSS projects.

I'm worried that up until now, this community has been too focused on the threat of big companies pushing capabilities ahead and not focused enough on the threat posed by open-source AI. I would love to see more discussions of regulations in order to mitigate this risk. I suspect it would be possible to significantly hamper these projects by making the developers of these projects potentially liable for any resulting misuse.

I have no idea how you think this would work.

First, any attempt at weakening liability waivers will cause immediate opposition by the entire software industry. (I don't even know under what legal theory of liability this would even operate.) Remember under American law, code is free speech. So...second, in the case you're somehow (somehow!) able to pass something (while there's a politicized and deadlocked legislature) where a coalition that includes the entire tech industry is lobbying against it and there isn't an immediate prior restraint to speech challenge...what do you think you're going to do? Go after the mostly anonymous model trainers? A lot of these people are random Joe Schmoes with no assets. Some of the SD model trainers which aren't anonymous already have shell corporations set up, both to shield their real identities and to preemptively tank liability in case of artist nuisance lawsuits.

The fact that they can squeeze more performance out of models with smaller datasets and less compute points to a current computing overhang, which reduces the odds that Altman's strategy of short timelines/slow takeoff can be done safely.

Aren't a lot of the doomy scenarios predicated on a single monolithic AI though? (Or multipolar AI that all agree to work togrther for naughtiness for some reason)

A bunch of them being tinkered on by lots of people seems like an easier path to alignment and as a failsafe in terms of power distribution.

You have lots of smaller scale dangers introduced but they certaintly dont seem to me to rise to the level of x or s risk in the near term.

What have we had thus far? A bunch of think tanks using deductive reasoning with no access to good models and a few monoliths with all the access. Seems to me that having the capability to actually run experiments at a community level will neccesarily boost efforts on alignment and value loading more than it assists actual AGI being born.

I think this speaks of a move from focusing on building the largest models possible, to optimising smaller models for mundane utility. Since smaller models have a hard ceiling on capabilities, this is good news as it pushes off AGI.

I have a list of specific technical advances in algorithms/architecture that I suspect will push these open-source LLMs further in capabilities when integrated. Especially increasing effective agency & strategic planning. I think these advances, if like 80% of them are discovered and are as successful as I expect, are enough with like a 65 billion or so param LLM, to begin a successful RSI process. Slow at first, but able to accelerate since I still think there's a lot of algorithmic efficiency gain sitting on the table. What I don't know is what to do about that scenario. If this is the case, then the world is going to look pretty crazy a couple years from now.

The Substack post only mentions that a researcher leaked the document, not that any researcher authored it. The document could've been written up by one or more Google staffers who aren't directly doing the research themselves, like a project manager or a research assistant. 

Nothing in the document should necessarily be taken as representative of Google, or any particular department, though the value of any insights drawn from the document could vary based on what AI research project(s)/department the authors of the document work on/in. This document is scant evidence in any direction of how representative the statements made are of Google and its leadership, or any of the teams or leaders of any particular projects or departments at Google focused on the relevant approaches to AI research.

This gives me the title for season 1 episode 1 of Uncanny Valley, the spinoff of Silicon Valley focused on AI alignment: "I Have No Moat and I Must Scream". (It's a Harlan Ellison reference.)

I think this is absolutely accurate and here's why:

1. open-source will surely beat the big guys. Openai and google do NOT have a monopoly on smart people, there's far more of us that do not work for them. Any smart person who is not allowed on the winning team will simply make their own team win. Therefore openAI and google will be left in the dust in 2 years flat. And I don't see anything stopping researchers from leaving and then just giving their truly awesome ideas to open-source after they take home the big bucks.

2. They have no moat, at least not one that is going to be there for long. 
 - First mover advantage is not an advantage in every game. America has been king the world ever since they joined WW2 late. And the first movers got the reputation they deserved. As they say, sometimes it's better to wait and let the competition draw a bit of blood before you jump into the fray.

 - data moat? nope we got open data. network effects moat? nope, we got github, technology moat? apparently not a very thick one. Entrenched customers? yeah for openAI. But once you're paying for multiple Ai subscriptions and you have to get rid of one... maybe it's byebye openai. 

The only moat they have is that they have a lot of money to pay researchers. And they can pour millions into training models. So if they can win by spending more money then that's how they can do it. OpenAI can win by building on top of their user base. But openAI shows no signs of doing that, they are just becoming microsofts toy, and google can pour as much money as they want wherever they want it's not going to buy them something better than open source.