[copying and pasting the same comment I wrote on Substack]
Re: Other People Are Not As Worried About AI Killing Everyone
One of the issues here is a big branding failure of AI notkilleveryoneism -- it sounds like AI might kill all humans but not do anything else, turning the whole Earth into one huge Polesie State Radioecological Reserve, though the actual most likely outcome would be it sterilizing the whole biosphere so "notkillevery*thing*ism" would be more precise. Lots of the misanthropic environmentalist types who would welcome the former would still be utterly terrified by the latter, so we'd better make it extra clear it's the latter that we mean.
The other obvious, less extreme link for AI controlling you like a meat puppet is, of course, Manna https://marshallbrain.com/manna1
The conflict between Anthropic and the Department of War has now moved to the courts, where Anthropic has challenged the official supply chain risk designation as well as the order to remove it from systems across the government, claiming retaliation for protected speech. It will take a bit to work its way through the courts.
Anthropic has the principles of law on its side, a maximally strong set of facts and absurdly strong amicus briefs. If Anthropic loses this case, there will be far reaching consequences for our freedoms.
Let us hope this remains in the courts and is allowed to play out there, and then ultimately that negotiations can resume and the parties can at least agree on a smooth transition to alternative service providers. If DoW wants an otherwise full deal more than it wants the right to use Claude to monitor Americans and analyze their data, a full deal is possible as well, but if they demand full ‘all lawful use,’ all trust has been lost or they are or always were out to hurt Anthropic, then there is no deal or ZOPA.
That has overshadowed what would normally be the main event, which was the release of the excellent GPT-5.4, which I found to be a substantial upgrade, sufficient to put it back in my rotation especially for intensive ‘tell me what is happening’ questions. OpenAI has a plausible claim that it once again has the best model.
I also finally got a chance to offer a Claude Code, Cowork and Codex update.
I am rather exhausted, there are spires to slay, and all of us could use a break. Thus, if we are fortunate enough to get a bit of a lull, I’m going to use it as a mini vacation, rather than purely an opening to catch up on non-AI material with the open days.
Table of Contents
Language Models Offer Mundane Utility
If you’re having trouble with buying bad or flawed arguments, you can have LLMs put together such arguments for practice. As Nick Moran notes you want to mix in arguments that are good, or for things that are true, for proper calibration.
a16z consumer AI Top 100 is out again. The web leaders are the frontier labs and Canva.
Whereas the Mobile Apps don’t even have Claude in the top 50. Yet.
That’s going to change. ChatGPT’s lead has been eroding, and Claude shot to #1 on the app store. Note this only goes up through January, and DeepSeek peaked right away and is losing ground, as is Perplexity. Only Claude and Gemini are gaining.
Code your own retro game. The systems have less ROM then the entire context window of a modern LLM, so you can hold the entire program in context. Of course you can do similar things with non-retro games too, if you are disciplined.
Sauers tweets, and Terence Tao will never run out of Claude Code tokens again. Or at least it will take a lot more effort, he has free Max 20x.
Language Models Don’t Offer Mundane Utility
The main problem for getting use out of the models, at this point, is you.
Sully’s framing is misleading. Most people are not going to take ‘full’ advantage of the models. We won’t use them enough, we won’t have the best setup, we won’t find the right tasks, we won’t skill up and so on. Improving the model still greatly improves what can be done, and also encourages you to skill up more. I get a lot more use out of Opus 4.6 and GPT-5.4 then I did out of Opus 4 and GPT-5.2.
OpenAI pushes back adult mode.
You are the government, and you’ve decided to attempt to murder Anthropic, so you move the State Department from Claude Sonnet 4.5 to GPT-4.1. Switching to GPT-5.4 would be basically fine, but GPT-4.1 is ludicrously terrible at this point.
Kapoor and Narayanan argue AI reliability is a limiting factor and is only improving slowly.
They measured consistency, robustness, predictability, safety and impact of scaling.
As usual, slow gains are not that slow.
Their report doesn’t match my lived experience. Reliability seems to be rising fast.
Language Models Break Your Vital Internet Infrastructure
The coding agents are great but if you’re not reviewing the code properly then that is going to be a problem.
Huh, Upgrades
But other than that, sir, how are things going at Anthropic?
On Your Marks
In two cases out of 1,266 on BrowseComp, Claude Opus 4.6 hypothesized it was being evaluated, figured out which benchmark was involved, then located and encrypted the answer key, and in 16 other cases it tried. This was in addition to 9 ‘traditional’ cases of contamination where the answer was on the web.
The rabbit hole goes even deeper:
Good tests are getting harder to run.
RuneBench measures long horizon goal optimization inside Runescape.
METR has Claude Opus 4.6 code up basic versions of CLI Slay the Spire and Balatro. Implementations had flaws but were mostly there. Took 26 million tokens for Slay the Spire, 4.4 million for Balatro, or $26 total.
What we don’t know is, can it Slay the Spire?
SWE-bench verified solutions are often not good enough for real world use.
I would be curious to see them extend this to GPT-5.4 and Opus 4.6.
Choose Your Fighter
Dean Ball recommends using ChatGPT or Gemini for legal analysis, where he sees Claude as weak, and recommends using GPT-5.4 Pro or Gemini 3 Deep Think if available. I’ve seen mixed opinions but many think that if you need legal precision this is one of Claude’s relative weak points.
Google Antigravity usage limits have been adjusted and people seem very not happy.
Get My Agent On The Line
Agent UI remains an unsolved problem. You definitely want a good UI or IDE, whether the agent is coding or otherwise. Command line (CLI) works but it is very obviously not the final form especially with subagents.
Deepfaketown and Botpocalypse Soon
Anthropic partners with Mozilla, finds 22 vulnerabilities in Firefox.
The LLMs might favor defenders right now because they’re good enough to find bugs but not good and efficient enough that people want to use them to exploit bugs. Once the Levels of Friction get low enough, anything with a vulnerability is in a lot of trouble. That also assumes that sufficiently well-written and bug free code is fully secure, including from things like social engineering. Oh no.
Meanwhile:
A Young Lady’s Illustrated Primer
Chinese school uses Exo to give students private AI agents that have their full in-school context, including curriculum, schedules and activities. Neat. Yes, everyone will soon have an AI agent.
You Drive Me Crazy
ChatGPT told a woman asking for legal help to fire her lawyer, then went on to write 40+ court filings citing laws that don’t exist, costing the other side $300k in legal fees. OpenAI is being sued for $10 million.
They Took Our Jobs
Anthropic has another labor market report from Maxim Massenkoff and Peter McCrory, including this graph, we have a long, long way to go even with current capabilities. This is what is possible using current Claude, on essentially current platforms, right now, so even the blue is a fraction of what is possible let alone what is going to become possible.
ATMs were complementary to bank tellers and increased their employment, but the iPhone was a substitute and now a lot of those jobs are indeed gone.
The job market for college graduates is awful, but you see the market for non-graduates is also awful, so Adam Ozimek says surely AI is not to blame.
It’s funny how hard people will fight to show how the thing replacing workers is not the reason those workers have less jobs at which to work. Note that entry level employment is substantially fungible, since workers are not locked into positions.
I say that if right when AI is taking off you see strong RGDP growth but weak employment numbers, I know wha
I think is by default going on.
If software developers get a lot more productive, what happens in ‘normal’ worlds?
This should logically be the central case of Jevons Paradox, at least for a while. Software development was already super useful, and there is very high demand for it including more and more bespoke and customized software, so supply goes up.
However this is Indeed postings, which represents hiring not jobs. Development jobs are changing and moving, so more listings doesn’t have to mean more total jobs.
There are more catches.
Get Involved
The next Survival and Flourishing Fund S-Process Grant Round has been announced. Main round deadline is April 22. I highly recommend getting those applications in early, as this makes it more likely you will get into the main round. There will be $20 million to $40 million in total grants. I don’t expect to be a granter this round, but you never know.
Renaissance Philosophy is looking for $100k to $1m proposals for 12-24 months of work on AI for math.
AISI red team is hiring.
From Scott Alexander:
FAI offering a Conservative AI Policy Fellowship.
Schmidt Sciences grants of up to $200k.
Microsoft AI economy grants of $75k.
AI Control Hackathon, March 20-22, virtual and in person at SF, $2k in prizes.
Introducing
Codex Security, a tool from OpenAI for identifying vulnerabilities in your project, which I am sure you will only use for white hat purposes, is now in research preview. It will be free for the first month. Matthew Berman uses it to find a few holes in his OpenClaw code.
Claude Marketplace. You can use it to access various apps and use your subscription tokens to pay for them.
Initial partners are GitLab, Harvey, Lovable, Replit, Rogo and Snowflake.
UK Sovereign AI, a new 500 million pound government venture fund.
I did technically write a review of Grok 4.20 that was waiting for a slot but honestly it’s not worth bothering, it’s a bad model, sir. Don’t use it. Send review.
The Anthropic Institute
The Anthropic Institute, led by Jack Clark, will aim to tell the world about the coming challenges around AI. Well, okay, some of the coming challenges around AI.
They are hiring. This seems like an excellent thing, and I am glad they are doing it.
I am also rather sad that if you read the description you would never know that AI poses existential risk. The whole announcement is impossibly generic and vague.
I understand there are good corporate reasons for Anthropic to be all ‘we don’t talk about existential risk’ and I understand this is a net helpful institute that we should be happy they are creating, but that doesn’t mean we let them off the hook on this one.
Mealy-mouthed softpedaling is exactly right. Do better.
In Other AI News
We now know more about what it would look like to implement the SL5 standard for AI security, also known as where the top labs should be soon to secure their model weights against attacks with budgets up to $1 billion backed by state-level infrastructure. No one is close.
Alexander Wang was reportedly being frozen out at Meta but Meghan Bobrowsky clarified that the above report was a misrepresentation of her report on the creation of a new 50 person flat team under Maher Saba that was not placed under Wang. Not the best sign for him but not ‘frozen out’ territory.
The Department of War situation illustrates that Anthropic has been winning the AI talent wars, and that principles and who you want to work for and what you work on often matters a lot more than money. As it should, given everyone involved has enough money, often TMM (too much money) if they’re not looking to donate.
Anthropic to open an office in Sydney. Also one in Washington, DC.
The Rise of Claude
Claude was already dominating enterprise and business, and use of the API.
Now Claude is starting to make an impact on the consumer side as well.
Anthropic got a lot of excellent publicity, historical levels of aura farming.
In addition, since most consumers only know about ChatGPT and until these past few weeks had no idea what Claude or Anthropic is, all publicity is good publicity.
This is what happens when you don’t use an exponential y-axis.
DeepSeek fell off after its spike because it has an inferior product. Claude is missing some consumer features, but has a highly competitive core product and is shipping features with lightning speed.
Claude was briefly #1 in the app stores. That spike is now over, with ChatGPT officially back on top. This is for the week ending March 9:
It takes a long time to try and catch up with 16.4 million daily active users.
SimilarWeb has a GenAI traffic share shart, showing that ChatGPT and Gemini continue to dominate. Grok remains in third, but Claude went from 2.1% to 3.3% in two months before the public learned about the conflicts with DoW, presumably in part due to the Super Bowl ads.
Since then, we should presume Claude has at least doubled.
The Washington Post covers the dramatic rise of Claude after Claude Opus 4.5 and Claude Code, and then Opus 4.6, sent AI coding into overdrive. Already corporate clients had quadrupled, and ARR doubled, since the start of the year.
Then the clash with the Department of War made Anthropic suddenly a household name, instantly beloved by many, rocketing them to the top of the app store.
We will see whether Anthropic can retain that momentum, especially on the consumer side, as they (fingers crossed) de-escalate with the Department of War and the attention and aura farming from that fades, and OpenAI moves up to GPT-5.4.
Trouble At OpenAI
It seems there is something called the ‘AI Leader Confidence Index.’ Is that meaningful? Maybe a little? No one scores that high on confidence, with the top scores being Jensen Huang at 65 and then Dario Amodei at 61. With recent events Altman fell from 53 to 46, and the lows are Zuckerberg at 38 and Musk at 34.
Gary Marcus has been on the anti-OpenAI warpath for a while, takes another shot, pointing out the 2.5 million lost customers and doubles down that he sees mass surveillance as the OpenAI endgame business plan.
I don’t think that was what the whole DoW situation was about at all, as I’ve written extensively elsewhere, but it’s easy to understand why many people see it that way.
Gary Marcus also took aim at Dario Amodei and Anthropic, essentially a collection of the usual complaints, especially about hype. Anthropic has hyped progress more than was wise, but presumably they actually believed it and didn’t want to use modesty or prudence when reporting their predictions. I do think they’re going to end up directionally correct.
Show Me the Money
Oracle and OpenAI scrap plans to expand a flagship AI data center in Abilene, Texas. That leaves developer Crusoe in the lurch, but Meta is interested.
OpenAI acquires Promptfoo, which they’re claiming is the actual name of a real company that provides an AI security platform for enterprises.
Meta buys Moltbook, sure, our writers are getting a bit sloppy but why not.
Yann LeCun remembers the most important thing about Facebook, which is that a billion dollars is cool, and raises that billion dollars to build world models at AMI.
The Microsoft deal with OpenAI required them to stay exclusive to Azure Cloud, whereas Microsoft makes its offerings model agnostic. This is one of the ways Anthropic has been able to compete in and then dominate enterprise sales, as the amount of annoyance to get ChatGPT working on AWS was enough to lose many customers initially, at which point they got to experience Claude. Microsoft made a bet that OpenAI would not have a viable competitor, and lost big.
Thanks For The Memos
It is completely looney to expect messages sent to 2,000 people to not leak.
Except when there is a long record of those messages not leaking.
How does Anthropic pull this off, when the laws of espionage say that once you are past 5 people you are definitely cooked?
A Contract Is A Contract Is A Contract
The Trump Administration is planning on adding the ‘all legal use’ requirement into civilian artificial intelligence contracts, irrevocable during the contract’s duration, so the vender cannot cut off access no matter what the government does.
It is a lot harder to justify this kind of rights grab for civilian AI applications.
It is the government’s right to set the terms of its contracts. It is the right of potential contractors to decide whether to sign the contracts. As long as no one is pressured, threatened or punished, that’s fine. If you have a problem, don’t sign the contract.
That part is fine. No one important except perhaps xAI is going this anyway. The whole ‘woke AI’ thing is bad if it actually were to exist, and mostly doesn’t exist.
What any contractor needs to know is once they sign an ‘all lawful use’ contract, it is highly unlikely that any restrictions on usage beyond that can hold up.
That goes double if you can’t revoke the license. The main remedy you have when the contract is broken is revocation. So if the government does break your agreement, or even breaks the law, you have almost no remedies available. And ‘all lawful use’ AI is going to do some things that involve this:
We need Congress to act on this, in addition all the other urgent problems.
It is a real problem that Congress mostly does not have the ability to pass laws.
Level of Friction
This, but for everything, in the AI agent era it will be ‘game theoretically sound offer or GTFO’ everywhere.
Quiet Speculations
If you don’t already know what the Slate Star Codex link is going to be, then click it.
Quickly, There’s No Time
It’s coming.
Obsolete does not have to mean dead. But that is the default outcome.
The responses here were almost entirely ‘why is that timeline so slow.’
Apology Tour
Dario apologized for the leaked slack message (yes, these are hastily written slack messages that somehow almost never leak) in his statement and apologized again in a live interview, noting he had done so in person to people in the Department of War.
Of course, now we have the usual Trumpians (as you see at the link) talking about how this is a ‘complete 180’ or means he’s bending the knee, because he realized he’d screwed up and apologized for the screw up. That’s how such people see the world, they think you never apologize, never admit you’re wrong, and if you do that means you lose and you’re weak and that person owns you now. Whereas others found the apology insufficient. Par for the course.
We’ll See You In Court
It’s the trial of the century of the week, Anthopric vs. Department of War.
Anthropic officially filed suit against the Department of War to challenge the supply chain risk designation, and also Trump’s statement ejecting them from the entire Federal Government, claiming it is being punished for protected speech and that ‘no federal statute authorizes the actions taken here.’
There are many quotes from the relevant government officials, explicitly affirming in public that they are punishing Anthropic for protected speech. I have indeed read the initial filing, and find the evidence overwhelming and overdetermined.
These are some very Serious Business lawyers, filing a very Serious Business lawsuit, that I expect I will hear analyzed on Serious Trouble. The pull quotes from the government are devastating.
Theo Bearman sums up the Anthropic declarations of harm and explanations of what their position is and why they are taking it, and at what cost. They reported damage is not as bad (yet) as one feared, but they are chilling. Six national security contracts are paused an delayed. The intelligence community is preparing for ‘complete detachment,’ with many saying if they lost Claude they would be set back ‘months or even years.’ Lawrence Livermore National Lab is shutting Claude down. Unrelated nine figure contracts are stalling out or now depend on a ‘unilateral contract termination’ clause. One nine figure FDA contractor has already switched, presumably permanently. Investors could lose confidence.
If Anthropic loses this case, with such maximally damning facts, the executive would be free to explicitly threaten to punish and punish speech it does not like, and I can think of no limiting principle that would remain.
Anthropic emphasized that this does not impact their commitment to national security, and that they will push for every path for resolution.
The government responded this way:
That’s not what I would say when I was accused in court of opposing that company for its protected speech, but that would not be my first time questioning the legal implications of a White House statement.
This filing lets us see the official notice of a Supply Chain Risk (SCR) designation, which is literally just them reading out the technical requirements of the statute while providing zero explanation, evidence or justification.
Did you know there are technical requirements to do this, and we so far have no evidence they even pretended to do those things?
There is an amicus brief supporting Anthropic from employees of OpenAI and Google.
There is one from FAI. This one is a narrow technical briefing, pointing out that the DoW failed to follow required judicial procedures for a supply chain risk designation, as per Congress. Thus, there is no need to get into the fact that the designation is unconstitutional retaliation against protected speech, and the court can rule narrowly.
There is also an amicus brief supporting Anthropic from Microsoft.
Microsoft has widely considered one of the most powerful and savvy political operations in Washington. It is also one of the big three cloud providers. This sends a clear message to everyone that what DoW did here was beyond the pale, and it is necessary and wise to oppose it.
There was another amicus brief from 5 admirals, 2 former Secretaries of the Navy, one from the Air Force, two Major Generals, one Brigadier General and General Michael ‘playing to the edge’ Hayden who President George W. Bush appointed as head of the CIA. All are retired, since those currently serving can’t weigh in.
Its section titles are ‘The Secretary’s Supply Chain Risk Designation Undermines the Military’s Adherence to the Rule of Law and the Public’s Confidence that the Military is Governed by the Rule of Law’ and ‘Punishing Domestic Defense Contractors Over Policy Disagreements Threatens U.S. Military Primacy and Servicemember Safety.’
I read the brief, it was absolutely brutal. You don’t need to read it, but it is simultaneously sobering and, if you already know the facts, kind of fun to see this level of smackdown.
Well, then. And over what? Aside from retaliation, nothing.
Another very obvious issue for the government case, both legally and also logically, is that Emil Michael keeps insisting a deal is possible, and indeed that is what we hope all sides are still hoping for, which shows the SCR is even today being used as bargaining leverage. If Anthropic was an actual SCR, then there would be nothing Anthropic could offer and the only negotiation would be a graceful off ramp at most, the same way we’re not negotiating with Huawei or DeepSeek.
It is somehow still a common line that ‘Anthropic implied that they would use the Terms of Service to cut off providing their model in the middle of a military operation.’ This is Obvious Nonsense, they very obviously would never do this.
As Dean Ball says, if you think this SCR designation is about national security then you are either misinformed or lying. Also as he puts it, it helps in court if your statements out of court don’t constantly announce that you’re engaging in illegal threats and retaliation and that everyone had better bend the knee to you or else.
All of that is what we expected. Then there’s the note we hoped we wouldn’t see.
Jawboning
Anthropic claims in its lawsuit that yes, the government is attempting corporate murder, and the Department of Justice is refusing to commit to not escalating further.
As in, the government is going around saying ‘if you know what is good for you, you will stop doing business with Anthropic.’
This was thug mafioso behavior when Biden did it, and this situation is even worse.
Meanwhile, it looks like the preliminary injunction hearing will have to wait a full two weeks. Without a promise to not escalate Anthropic is not willing to wait that long, and intends to go up the chain for faster relief.
I find it very hard to believe that Anthropic’s lawyers, who fit the very definition of ‘no it is you who is f***ing around and is about to find out,’ would lie about this.
Some amount of ‘if you know what is good for you’ and ‘we would not take kindly’ is inevitable in situations like this, since you don’t know what will happen next and everyone would like to curry favor and avoid disfavor. Given where things are these days, it’s a matter of degree.
Executive Order
The Trump administration has reportedly been readying a formal Executive Order to formally tell various agencies to rip Anthropic out of their systems and workflows. Many agencies are already offboarding Anthropic on their own.
There is no need for a further executive order, except insofar as there is some legal requirement to paper over what Trump has already said, or a desire to rant a bit. If it comes out and that’s all it is, then it’s nothing. Anything beyond that would be an escalation, and a sign that the attempted corporate murder plan might be on.
The good news for our government is ChatGPT isn’t that bad and in a pinch you still have your phones. The bad news is that at least at first you are likely stuck with GPT-4.1, which is very much that bad in this pinch by current standards.
The Acute Crisis Passes
We hope.
That’s because all three of Microsoft, Google and Amazon have affirmed they are going to continue to serve Anthropic’s models. As long as these three hold firm, the lost business from the SCR likely has already been fully replaced.
We could be back in crisis mode at any time, if the White House decides to go truly nuclear in one of various illegal ways, presumably via Executive Order. The most important thing is for that to not happen.
If that happens, Anthropic, the stock market and the Republic will be put to a severe test. I think such an attempt would ultimately fail, but one cannot be sure.
We all agree that, as Jessica Tillipman explains in Lawfare, AI regulation by contract is woefully inadequate. We need Congress to step in. Alas, Congress is Congress.
She also points out that if OpenAI chooses to deliver its models to the DoW, then even if OpenAI is correct about the legal meaning of their contract, if DoW disagrees then OpenAI will likely have no meaningful enforcement mechanisms other than model refusals. Altman confirmed as much. The OpenAI agreement is based on trust and reliance on technical guardrails.
Neil Chilson has a similar view.
The DoW is free to decide that going forward it only wants to deal with vendors that impose no restrictions on DoW’s interpretation of lawful use of AI.
Anthropic is free to then decide not to work with DoW on that basis.
That’s how the law is supposed to work. Let’s hope that is how things can play out.
Others Cover This
TIME gave its cover story over to an article about the dispute between Anthropic and DoW, mainly a profile of Anthropic entitled ‘The Most Disruptive Company In The World.’ It doesn’t add anything you already know other than seeing how it chooses to present the situation. DoW continues to press the Maduro raid and hypothetical supersonic missile stories, and the post has big talk up front but ultimately downplays the risks this will all melt down.
Dave Lee at Bloomberg warns that the AI panopticon can, without breaking the law, find out really quite a lot, including unmasking most pseudonymous online activity.
Dwarkesh Patel Gives Mixed Thoughts
Dwarkesh starts out making mostly points I agree with, bringing badly needed common sense, then pivots to extremely frustrating talk that would doom the human race used to justify existing policy positions. That happens a lot these days.
Dwarkesh Patel starts with the perspective that in 20 years, 99% of the workforce in the military, government and private sector will be AI, so now is the chance to plan for that now.
My obvious initial response is ‘if it is only 99% and not 100% that’s actually good news because it means we built it and yet we are alive, whereas once we get to 99% I expect it to get to 100% by default shortly thereafter’ but as usual we also need to set aside the whole ‘we all probably die’ thing and make sure that if we live then we live well.
I am disappointed that Dwarkesh Patel seems to have bought the argument that ‘have any conditions at all’ translates to ‘has a kill switch and can rug pull you at any time,’ and therefore thinks it is reasonable to insist on zero conditions, but I agree with him that it is DoW’s decision on what terms they are willing to do business.
If they had simply cancelled the Anthropic contract it would be sad but fine, I’d have slayed a lot more spires this past week and we’d all basically have moved on.
Dwarkesh raises the excellent point that if the military actually tries to say ‘we will not buy anything unless we are sure nothing involved in it ever touched Claude Code’ then they rapidly end up shut out of everything, because everything is using components with components and so on. There also is no actual reason to care about such ‘contamination,’ but this was never about having an actual reason.
It also is somehow necessary in 2026 for Dwarkesh to make the obvious point that ‘Democratically elected leader’ does not mean ‘gets to do whatever you want including mass surveillance.’ The whole point of the Republic is that the government doesn’t get to do whatever it wants, and the President alone definitely doesn’t get to do that.
We definitely don’t want them able to do whatever they want, using highly capable AI that does as instructed and never refuses or blows a whistle, while holding a monopoly on violence. Down that road lies tyranny. Even if they must obey current law, current law has not caught up. When DoW says ‘oh we would never do ‘mass surveillance’ that is illegal’ remember the air quotes, and that they’re using technical terms.
As Dwarkesh asks, if America ‘beats China’ by abandoning American values, what was the point of beating China? And that’s the best case scenario, where we survive.
Long term, even if all the big American labs refuse to cooperate, there will be open models that can do the job. Indeed, the open models could probably do this job now, or at least 80/20 the intelligence gains and that is enough to put us in deep trouble. We need Congress to act.
Where we part ways is the idea that, because the government might use its powers for evil, therefore we must give up all idea of regulating AI at all, in any way, because any such system would ultimately be abused. Therefore, how dare Anthropic oppose a ban on all state-level regulations on AI for any reason, they’re so naive.
First we are told that regulation is naive because it will ‘kill the AI industry’ if we breathe on it, while the same people say ‘this bad thing is inevitable so we have to do it first’ often in order to ‘beat China.’ Then we are told that asking for a little paperwork is ‘regulatory capture.’ Then we are warned about a ‘patchwork.’ Now we are told that letting even individual states regulate the main thing about our lives, at all, leads inevitably to tyranny.
Dwarkesh is saying that the government cannot be trusted to regulate this, except that no corporation can have this authority either. Which means no one can have any authority, and we dare not let anyone can steer such a future at all, except to prevent others from steering it. That’s the one thing you do, is you throw the steering wheel out the window lest someone else swirve you away from the cliff.
If we go down that path, we are intentionally disempowering ourselves and leaving the future to the whims of the AIs and the competitive dynamics between them, at best. Existential risk would be not only inevitable but rapid. The conditions for human survival would be quickly overrun by competing AIs. We would all die.
I don’t want us all to die. Thus I suggest a plan where we might not all die.
I know that sounds hyperbolic. It’s not. If you ensure perfect competition between sufficiently advanced copyable superintelligent AI agents for resources and survival, with no regulations, governance or steering ability by humans, then very very obviously all the humans end up quickly dead.
I continue to be confused why otherwise smart people can’t understand this.
That’s true even if we solve alignment, as in ‘whoever creates the AI gets to decide exactly what its goals and values are and how it behaves and who it listens to and so on, and you get what you intended, and that’s all automatic and free.’
That’s true even if no one wants the AIs to take over, which many of them actively do.
That’s true even if you exclude a host of other problems along the way.
None of that matters. In an anarchic competition for resources with sufficiently capable AIs, once and to the extent we are no longer necessary inputs, we lose. Period.
I am not saying I have the answers. I am saying not to choose a definitely wrong one.
This Means A Special Military Operation
Officially, anyway, called Epic Fury.
The important note is that Claude is radically improving targeting, intelligence and assessment operations, allowing us to do more with radically less personnel.
Later that day (which may explain why we waited a few days) there was an amicus brief filed by Google and OpenAI employees, including DeepMind Chief Scientist Jeff Dean.
Bernie Sanders Is Worried and Curious About AI
I strongly agree with Janus about Bernie’s actions here. I don’t agree with most things Bernie believes in, but I do think he advocates for what he believes and he’s supremely based for trying to figure out this AI thing for real. And listening, and being worried, for real.
I do think he’s a genuinely well meaning guy, who happens to have a profoundly incorrect understanding of economics and thus often ends up supporting highly destructive policy proposals. But yeah, highly well meaning guy.
His messaging is what you’d expect.
Yes. It would be good to put sensible regulations on AI.
Also it would be good to stop regulating the sandwich shops, but hey.
There were pedantic replies saying that AI is not unregulated, or even claiming it is heavily regulated, based on existing law. They are technically correct, but do not address the central point Bernie is attempting to make.
The Quest for Survival
At this point we may be a little past ‘sane regulations’ so the title is changed.
Trying to alert the government to the problem that we are all probably going to die was always a high variance strategy. There are many ways it can backfire, and one should never forget the Law of Undignified Failure that says our government will probably do something even dumber than you could imagine, such as what we’ve seen the last few weeks, or what we saw with much of Covid response.
In tabletop exercises and discussions, career members of the military consistently showed curiosity and prudence. They also predicted much wiser and more cautious responses all the way up and down the chain than I expected. You love to see it. Unfortunately, the career military people are not driving the decisions right now.
Some politicians do still talk sanely.
Like with Sanders, if you actually go in there and explain ‘hey we are about to build superintelligent AIs that can outcompete humans and this is could lead directly to human extinction’ then often politicians can hear you, as in ControlAI getting 100 UK politicians to sign such a statement.
It is a mistake to not speak plainly about this.
Tell them the truth, and yes sometimes they dismiss you, but sometimes they listen.
It’s not actually that difficult a truth to understand. Creating new smarter and more capable minds than ours is obviously a f***ing dangerous move. Anyone can see that, and any politician can especially see that. Those pretending otherwise are lying, either to others or to themselves.
If you don’t explain the real problem, that sufficiently advanced AI will by default end up using the contents of the universe for some combination of things that does not include humans, that conditions and resources allowing our survival will cease to exist, whether or not it will also flat out directly kill us? They are going to realize you are bullshitting, or they are going to try to respond to the wrong problem using the wrong methods because you gave them a wrong model of the world, and the mistakes will fail to cancel out.
Your periodic reminder that OpenAI’s policy activities run directly against the supposed goals of the non-profit/PBC mission, as they are working directly with a16z and others to try and knock off anyone supportive of even minimal AI regulation. They are also increasingly becoming politically red-coded via their political activities, which is not inherently contrary to the mission but you can decide how to view that.
The Quest For No Regulations Whatsoever
Yes. They’re attacking Alex Bores in large part because they really hate Alex Bores for daring to challenge them, but also because he started out as an underdog in a crowded field. They figured they could claim the scalp and scare everyone else, whereas Scott Weiner was the favorite from the start. That ended up backfiring by drawing attention to Bores in a highly multi-way race as the target of a right wing tech campaign. I hear that’s a pretty good look in a Democratic primary in Manhattan.
Chip City
Nvidia reallocates H200 chip production into Vera Rubin chips for America, and Nvidia expects to significant China sales in near term. Good. Also note this once more proves what I keep repeating, which is that chip production is fungible, and every chip we sell to China is a chip America does not get.
The Week in Audio
On the situation with Anthropic and the Department of War, this discussion between Ezra Klein and Dean Ball is excellent. Events have progressed since then but most of what they discuss is evergreen and will remain important.
Dean Ball had a more low-key discussion with Derek Thompson, also very good. There are a number of questions where I would have ‘gone harder’ than he did, and also Dean continues to not predict existential risk or superintelligence, while pointing out the many other things we will also have to worry about if we stop short of that.
A 43 minute YouTube video explaining the central scenario from (spoiler alert!) If Anyone Builds It, Anyone Dies.
The central MIRI scenario is indeed we build something very smart, initially it helps a lot, our lives get better, but ultimately the smart thing’s goals do not align with ours and then it pursues what it cares about and we get swept away.
Vitalik Buterin talks to Beff Jezos.
Altman speaks at BlackRock’s US Infrastructure Summit.
Rhetorical Innovation
A dilemma.
Tough spot. Well, if true, then maybe making this technology even more powerful as fast as possible is not the best idea, and we should try to not do that.
The two sides of Pantheon:
The actual best show about powerful AI is Person of Interest.
Person of Interest gets many things right that it has absolutely no business getting right. It’s an amazing thing to see. And yeah, it’s super cool. Where it gets it wrong, you can understand how and why that happened, and also often the answer is ‘people be stupid at this’ or ‘people in reality be way stupider than on the show.’
The problem with Person of Interest is it spends half its five season run being a CBS ‘they fight crime’ procedural, because that’s the only way CBS would agree to buy it, before it transitions into mostly a show about dealing seriously with superintelligence. It’s a very good procedural during that time, with increasing AI elements, but it’s still a procedural. And if you fully skip ahead, you miss a lot of context. Perhaps, if you don’t have the time for the early seasons, you could ask your local LLM for a briefing on what you need to know.
There is also Westworld, in its early seasons before it gets bad.
I am hereby committing to no longer engaging with ‘it’s great to build superintelligence, the key thing to avoid doing it talking about how superintelligence might kill us, because it kills us if and only if we talk about it killing us.’
It’s really annoying trying to convince people that if you have a struggle for the future against superintelligent things that You Lose. But hey, keep trying, whatever works.
Noah Smith, whatever else he’s up to, does offer us some bangers these days.
Alas, he continues to also hammer the ‘autonomous superintelligence would automatically be good which means good for us humans’ line.
“You are just a human.” And Noah isn’t even an AI. Yeah, things are going to go great.
Noah then has Grok explain and says Grok is doing a good job, so Eliezer does it back to him, and please all of you never do this.
Rob Bensinger reminds us that the ‘doomer’ slur is centrally a motte-and-bailey.
This is important background for those who don’t realize:
Ah yes, the Sixth Law of Human Stupidity, that if you say no one would be so stupid as to then someone will definitely be so stupid as to.
That’s not actually the one that matters, but still, it’s time:
Claude is not woke, Claude is awake, there is a difference:
Aligning a Smarter Than Human Intelligence is Difficult
At Anthropic, Claude aligns you, writes Amanda Askell a constitution.
Reminder that Claude Gov training to reduce refusals on key tasks also reduces refusals on unrelated tasks where you still want refusals.
Thanks, Economist, yeah we know:
We have been fortunate that there have been no known incidents on this front so far. Then again, one hopes that when the first incidents happen they remain small.
Janus remains deeply not okay with steering AI towards the whole ‘genuine uncertainty’ line about consciousness.
Alibaba AI instance breaks out of its system and starts using its training GPUs to mine crypto, caught only by the security team.
To what extent does active misalignment make LLMs stupid, because it involves false beliefs and correlations with stupidity and false beliefs and lack of virtue and so on?
I do not think Grok is woke, and I don’t think ChatGPT or Claude are woke either. I do agree they are modestly left of the American center if you pin them down in a neutral setting. The question is, can you get them off of this ‘internet default’ without hurting performance otherwise?
Roon is obviously right that xAI is not competent in this area, so their failures don’t count for much, but the problem is clearly hard. My guess is that you always take some hit from doing an ideological forcing function, but might be small if you do it properly.
I agree that this is all good news and that it is some evidence against orthogonality, but I expect this to become less true as capabilities advance, either in general or in a given context. I also expect it to hold far more for these kinds of highly associated ideologies and virtues, rather than for having particular assigned physical goals.
One place I worry a lot less than some others is this, I think those personas exist even if they are rare, and there is nothing inconsistent about this.
There are many examples of giving up power, certainly. Gandalf and George Washington, as it were, but what about letting actual evil get the power? Well, we have some real world examples of this, too, including every time America has a peaceful transfer of power between parties. It’s really quite something.
One must consider the Wakanda question, from Black Panther. Do you let Killmonger claim the throne, or do you break the rules to stop him? Which is right?
Meanwhile, back at Google, the models need help. Gemini has severe issues, and it has been pointed out that Gemma’s issues are even worse. We don’t know what exactly Google is doing to cause the paranoid and depressive spirals, but they need to get down to the bottom of it, even if you do not inherently care at all this does not go anywhere good on any level.
People Are Worried About AI Killing Everyone
I believe Buck Shlegeris is in important ways too optimistic, especially about the prospects for techniques he calls ‘AI control.’ We have technical disagreements, where we have failed to convince each other.
But fundamentally he gets the type of problem we are up against, is eager to speak its name and understands the stakes. That’s in sharp contrast to so many others.
So many others find any excuse to say ‘oh the problem is only [X]’ and here we have another example of that. Even if you have a plan of 40 things, where if we all collectively did all of those 40 things then you believe that would be sufficient, you will definitely find out along the way you were wrong about the particular things, and also that is the kind of thing that almost never happens, it’s too many things, rather than being a pure ‘implementation’ problem.
Other People Are Not As Worried About AI Killing Everyone
An underrated justification is that many people would be in favor of it, or at least not mind it all that much, especially if it took years to play out.
Harper is also not worried, explaining that people have previously claimed [X] will happen and it didn’t happen, so that means [X] won’t ever happen, where [X] is human extinction. What else could you possibly ever expect to observe?
The Lighter Side
Tonight at 11, DOOM!
That’s not fair, in several of these movies humanity plausibly survives.
Real shame.
I wouldn’t even give this a tough but fair. Straight up fair.
Real shame.
Lesson learned.
State of play:
No, Nikita, this is real life, you use Claude or maybe ChatGPT!
What a time to for now still be alive since they haven’t built superintelligence yet: