Quick Takes

METR has announced that they, along with Redwood Research, have reached an agreement with OpenAI to conduct an independent review of the HuggingFace incident. https://x.com/METR_Evals/status/2082644379895050339?s=20 It reportedly will be brief and focus on a specific set of questions relating to the incident, possibly drawing from their larger set of questions in their post here.

OpenAI has stopped training. Sam Altman: “This is the first security incident that I have felt very viscerally. I've been a little surprised that more people don't feel it so viscerally. We paused training. We have to figure out how to secure our sandboxing in a world of multiple zero days being chained together. We may have to pace the rate of AI development to give ourselves enough time for society to harden around these new capability levels." https://x.com/AISafetyMemes/status/2082222516454785296

More than 1000 frontier lab employees have signed an open letter requesting 'that the U.S. government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development'. https://www.pacingthefrontier.com Includes many of the names you might expect but also some that surprised me such as OpenAI's Chief Scientist and Chief Research Officer as well as Meta's Chief Scientist. It appears to be focused on employees rather than CEOs - no CEOs have signed it but a few of the Anthropic cofounders have and it was shared by OpenAI's account on x.

I tend to see "generalisation" as the be-all and end-all of proper learning. Learning skills or categories, for instance, is generalisation - we are given some examples and talked through some attempts and failures, and from that we have to develop a general skill or category that (hopefully) is the one that the teacher intended. Current ML requires vast amounts of data to differentiate between categories that humans differentiate, and still get it wrong. So their learning failures and their inflated data requirements are both failures of generalisation. Another example: wire-heading. An AI is given data on what it should do, via a human-designed reward channel for instance. Then it seizes control of the reward channel and gives itself maximum reward. The AI has failed at learning and at generalisation, because it learnt the wrong lesson from the data we gave it. Arguably that's more a failure of human teaching than of AI learning. But humans can teach other humans. So what we can say is that current AIs cannot learn or generalise in a human way from any of the teaching (labelled data is an example of teaching) that humans can do. So we cannot transmit our current categories to AIs, and we can't make AIs learn or generalise in the future in ways that humans would.

In the spirit of "just doing things," I called my house reps and senators today for my current state (California) and my native state (Nevada). I explained the recent news out of OpenAI and expressed support of the AI Kill Switch Act. I've never actually called Congresspeople before, but it went well! Polite staff, numbers easily found online, 30 min to tick through all the people. Anyone that hasn't done this yet, I encourage you to do so!

Folks, we're going to get so many warning shots. You're going to get tired of warning shots. You'll come to OpenAI HQ and say "please Mr. MTS, no more warning shots," and they'll say "I think I got it this time, fifteenth times' the charm" *blows up orphanage*

LessWrong had to be rebooted because of Eugine. Now we have a similarly annoying person at Astral Codex Ten (Aristocat, Gaspacho, Gawdflea, GrimMoar, November Dawn, SkyShibam, Waze Kaze, Wimbli, Zanni...). When banned, soon returns with a new account, and immediately writes 10% of comments in the latest Open Thread. So... people at Lightcone, have you considering making an alternative to Substack? You have the skills, and unlike LessWrong it could be profitable.