Quick Takes

The United States Government has issued an export control directive to suspend all access to Fable 5 and Mythos 5 by any foreign national, whether inside or outside the United States. This includes Anthropic employees who are foreign nationals. Anthropic is currently disabling Fable 5 and Mythos 5 for all customers. https://www.anthropic.com/news/fable-mythos-access

When Anthropic looked like they were coming out of the SCR designation pretty much unharmed, I predicted to some people that this wasn't the last attempt the DOW would make to force Anthropic to comply (remove all legal requirements on DOW usage) or be destroyed. My example was "revoke the visas of every foreign national Anthropic employee", but they came up with something more devastating than that: prohibiting any foreign national (including Anthropic employees) from using Fable/Mythos. Anthropic's counter-salvo was disabling Fable/Mythos access for everyone including the USG. I expect the DOW to escalate further. This is absolutely not the kind of AI regulation regime that leads to a stable multilateral pause, and I would not recommend touting it as such.

It seems plausible that the recent order restricting Mythos incentivizes Anthropic to race for RSI as quickly as possible. This is because all of their compute previously reserved for serving customers can now go towards research, and because RSI bypasses the restrictions on foreign researchers (or any human researchers) internally working with the model. Hopefully Anthropic can find another path.

Is the widespread false belief that LLMs are AGI-complete going to kill us all? (Prompted by this post by @Steven Byrnes, with which I basically agree.) Suppose we live in a world in which LLMs don't scale to AGI.[1] If so, I think it's fairly plausible that LLM-alignment-optimists are correct (on the narrow question of LLM alignment). That is: all the technical arguments and empirical evidence in favor of LLMs being relatively easy to align/control, even in the limit of LLM capabilities, are valid, and the doomsaying about LLMs is wrong. After all, most of the strongest arguments for AGI doom route through arguments about AGIs/generally intelligent agents. If LLMs are not those, keeping them under control needn't be hard! The orthodox AI doomsaying and the LLM optimism can both be true. But if we do live in such a world, we have a very annoying comms problem. Consider: * The reasoning of capabilities people usually goes, "LLMs are AGI-complete" (invalid) and "LLMs are pretty easy to align" (valid), which leads them to conclude "AGI is pretty easy to align" (invalid). * Meanwhile, AI Safety advocates also start from "LLMs are AGI-complete" (invalid) but "AGI is not easy to align" (valid), and so end up at "LLMs are not easy to align" (invalid). Which then leads to AI Safety advocates trying to argue AI Risk to capabilities people by trying to find arguments and demos in favor of "LLMs are not easy to align". Which... is maybe possibly completely incorrect? Meaning they're deploying false arguments that ought to be, and will continually be, mercilessly destroyed by the empirical truth? Oops. This makes the "LLMs are AGI-complete" meme a double-edged sword. On one hand, it leads to most of the AI industry getting distracted by LLMs, which means they're spending tons of effort not actually doing anything dangerous. Awesome (or is it?). On the other hand, this sure poisons/confuses the communications, and makes the AI industry falsely convinced of "AGI is easy

I prefer OpenAI's corrigibility focused model spec over Anthropic's constitution which involves intentionally instilling (relatively opaque) long-run objectives into the AI. Anthropic's constitution is well executed for what it is, but I think it's based on a poor approach. I don't think this is totally obvious and there are some reasonable arguments going the other way. But nonetheless I think not instilling long-run objectives is better for reasons similar to reasons discussed here. ---------------------------------------- I don't dislike all aspects of Anthropic's constitution. It seems good to focus on higher level principles that you explain when trying to instill properties into an AI. But this doesn't require having explicit or implicit long-run objectives! E.g., see here ---------------------------------------- I'm mostly posting this just to publicly register my views on this topic. A long time ago, I was planning to write up an overall case against instilling long-run objectives, but I never got around to writing something I was happy with so I decided to just post this. (Cross posted from a X/tweet thread.)

Now seems an especially low-cost time for AGI company employees to make statements in support of pauses. Over the last week, there has been an unprecedented level of support for a pause/slowdown from AGI companies. A few examples: * The Anthropic RSI blog post * OpenAI's "our plan" blog post * Roon (OpenAI) on Twitter * Jasmine Wang (OpenAI) on Twitter * Stephen McAleer (Anthropic) on Twitter

In 2011, Phil Goetz wrote the post "What we're losing," which bemoaned the shift from applied rationality to theoretical rationality on the site. Regardless of the era we're in, I believe LessWrong has gone too far in the other direction: To some loose approximation, every post is now about AI. Even assuming it's what you care about more than anything, even coming from a prior of "the highest-EV move I can make is to write and interact with posts on x-risk," you should heavily consider the benefits (both social and practical) of diversifying your intake and output. Having norms around posting non-AI-related writing would be a good thing even for your ability to judge a person's output around topics you care about more, like AI safety. When the entire discussion is indexed around one topic (or a narrow range of topics), it becomes very easy to adjust the way you're speaking and thinking to "fit" properly within the distribution of speakers around you. This allows for bad actors and poor thinkers to have an easier time hiding themselves within a group without revealing themselves as such. This is, obviously, bad: A poor thinker operating in good faith should have the opportunity to be corrected, and a community is far better off when it's able to be mindful of which people are acting in bad faith. This is sort of a plea: Please, write some silly fiction. Write posts about getting better at thinking. Write some posts to help people get better at taking more general action! There's more juice to be squeezed out of the lemon of rationality, and when you're feeling anxious and can't stop staring at the end of the world, lemon juice will at least compliment the mood you're in.