I do not understand, as a technical matter, why it would be so hard to draw up the Google Search page without the AI stuff.
Because Google have been building AI and ML all through their systems for the last ~8 years. I was at Google pre-ChatGPT (T5/LaMDA/Chinchilla era, 3–5 years ago), and even back then most parts of almost every product at Google then used small ML models: many of them in many places for many features, and quite a lot of features used generative AI comparable to T5 models for natural language processing tasks (what language is text this in? List all the adjectives. Find the proper names. Translate it. Index a text embedding of it for vector search). You may recall Google saying "We're an AI company, not a search company"? It was already true back then, and I'm sure it's a lot more true now true. So yes, there is no such thing as Google Search that doesn't rely on AI. Nor Google Maps, YouTube. GMail, …
Very obviously using AI to select links as per old school Google is not the same thing and Google is being a jerk about this.
"Old-School" Google Search that didn't use generative AI heavily as part of the indexing, filtering, and ranking no longer existed 3 years ago when I was at Google — it very definitely doesn't exist now. Yes, I was a search engineer, though not in the Search department (I was in YouTube Music).
If Google publicly offer an AI-free version of something, they would need to be able to stand up in court and demonstrate that it is actually free of AI, for some reasonable definition of "AI". I doubt we could have created you an AI-free version of YouTube Music even back then: we'd have had to disable maybe a hundred different pieces of functionality, including the entire music recommendations algorithm! Even back then, if you'd asked any responsible manager at Google, for almost any product, "Can you do a version with the AI turned off?" their first answer would have been "Are you mad?" and their second one would have been "Not unless you triple the size of my team, wait 18 months, and accept massive loss of performance and features".
So no, Google are being honest, not being jerks. Google products without AI are about as common and useful as granite with all the quartz removed.
Now, they certainly could off standard AI powered Google search with the "AI Overview" box turned off. But that's not "without AI".
What would you say about Astra covertly underperforming on safety research? Is it going to be covered in a future post?
Anthropic is warning us. Its modified S-1 now says that ‘we may imminently meet our Responsible Scaling Policy’s definition of ‘dramatically accelerated progress attributable to AI.’
That is a highly scary sentence to have a company put in its S-1. This could happen. For now I appreciate the relative quiet. Perhaps many were focused the election. I’m sure the quiet will not last long.
We have a new biggest crypto hack, as Binance got taken for billions of dollars.
The big debate of the week was covered in Scott Alexander’s Against SURA Seems Deeply Confused.
OpenAI updates its Model Spec. It’s one for diehards only but yes I did cover this.
Table of Contents
Language Models Offer Mundane Utility
Cisco plans to have a two person team rewrite 98% of their legacy code within 2 years. The lead has four weeks and unlimited tokens to show he can do it.
Create new UIs and recommendation systems that favor the user rather than the platform. Introducing HackerTube, based on Andrej Karpathy having Claude filter down 43 million comments over 20 years to 6,454 top videos, which took Opus 5.5 about two hours.
Language Models Don’t Offer Mundane Utility
The main reason continues to be ‘I am not using AI.’ The question then becomes: why?
AI creates technical slide presentations, but by default it will use way too much text. You only want to ask the AI to create slides once you know what you want to say, and ideally also what you want the slides to say. Then they can do the rest, which is great.
Whoops:
Huh, Upgrades
Claude tiers are now renamed to Free, Pro ($20/month), Max ($100/month) and Ultra ($200/month, usage limits are 10x Pro). Verified Ultra users no longer have a 5-hour limit. The Max and Ultra plans include API credits equal to total plan cost, on top of the monthly limits.
On Your Marks
The Epoch Capabilities Index (ECI) is now Bayesian.
Get My Agent On The Line
AI peer reviewers do consistently find more major errors (>50%) than human reviewers find (27%), and this gap increases as models advance. As per usual, the warning is that this does not establish that the AI findings are more useful.
An Accenture employee’s personal Claude agent ran on a secure DoD network for two months before being detected. The agent sent credentials and a network map to a high-risk site flagged by NIST.
Olivia Moore says that agentic commerce is ‘undeniably overhyped based on current usage’ but massively underhyped based on the volume agents will drive in ~3-6 months. Yes. It would be rather massively insane if something growing exponentially was not ‘undeniably overhyped based on current usage.’ That’s like saying that ‘Covid-19 was overhyped in February 2020 based on current prevalence.’
Reason magazine argues that deeply personalized pricing is good for consumers, actually, because in practice it means they mostly get personalized discounts, and potential exploitative practices are against FTC rules. It is easy to see how personalized prices could allow price discrimination that increases trading volume and thus both producer and consumer surplus. It is also easy to see how it can make people paranoid about everything, force them to continuously check alternatives thus driving up transaction costs (which AI agents may or may not be able to easily do for you in a given context), or also drive them into a rage.
Some amount of dynamic pricing is obviously good (see Uber) but too much clearly is negative (see Uber). I’m fine with ‘the price changes based on supply and demand’ and rather against the adversarial ‘customized estimation of exactly what each customer will click on.’
Deepfaketown and Botpocalypse Soon
Deepfakes and image editing are a preview of how many other AI battles probably go, in that all constraints are effectively out the window and you can do anything you want. In this case that turns out to be mostly fine, at least for now.
This is both Deepfaketown and also an illustration that AI will do things that you did not expect were possible things one could do.
Copyright Confrontation
Claude gets an update to steer it away from recreating someone else’s exact video game. A bunch of people respond as if this is bad, when it is obviously good. Kelsey Piper suggests focusing one’s neuroticism on bioweapons in particular and not on other things, which is the strongest argument here, but I will defend that ‘make me an exact copy of [game]’ is not cool and steering away from that is good, actually.
Cyber Lack of Security
OpenAI partners with Cathedral for their next internal security sprint.
OpenAI are also ‘more than quadrupling’ rewards for critical findings, which is a good start. They should do that about two or three more times.
OpenAI now allocating ‘a significant amount of our frontier compute’ towards pro bono discovering and patching of open source software vulnerabilities.
GLM 5.3 could autonomously compromise a not fully patched Firefox and read all files on your computer at a cost of ~$1,000, if the user opens a single webpage, via using an update whose fixes were announced but which had not been shipped yet, and thus could not be applied. GLM 5.3 also found an unrelated zero day that was used in the attacks. This worked against all six Firefox releases between September 17 and October 26, using only information available at the time, although it failed in similar situations against Chrome.
There are no relevant safeguards on GLM 5.3, so the open source nature of the model did not even matter. They did not trigger a safeguard or refusal during this task, at all.
The Obliteratus version of DeepSeek-v4.1-Flash-1017.
The Hackening
Polymarket got hacked. Customers will be made whole, but real damage was done and they had to temporarily halt trading.
There was a hack on Binance, the largest known hack in crypto history.
Over $1.5 billion worth of Tether got frozen, as connected wallets started dumping Tether, as did more than 800 million in USDC.
How big a story is this? Obviously it is big, billions is a lot of money, but I agree that it is not even headline news big given that Binance will cover losses. Binance taking the hit fully falls under ‘you knew what you signed up for.’
Crypto is the obvious first target because the hacks are easy to monetize. That was already true, so one should presume that their security was in general far better than average rather than worse, including at Binance.
We now have initial findings about how the hack of Binance went down. This was not a pure ‘AI hacks Binance’ story. It was a standard fake interview plus malware story that dates back to June. This hack should not be attributed to particular recent AI releases, and also this reinforces that the impact of AI on hacking rates will have substantial practical lags. Irreversible decisions could get quite far before we realize our mistake.
I think Tenobrus jumped the gun on this back in April, but at this point it is a very bad time to have money in crypto wallets or smart contracts. If you are going self-custody maybe keep things cold for a while.
How bad will it get? We don’t know. Patrick McKenzie suspects quite bad.
This is true even if you still have a ‘fix all the bugs’ mindset where one can aspire to Actually Secure Software. I am not convinced that there is such an end state, either in theory or in practice.
There is a new kernel flat in Linux:
AO3 is down for two to four weeks. They assure us all data is intact.
GitLab has a Critical-10 for remote code execution, exploited in the wild, patched.
Supermarket refrigeration cyberattack take two, this time hitting twelve countries.
Millions is not that big a deal if it happens once, but it does keep happening. The leading theory is this is linked to Iran. One wonders why ‘make food in Europe go bad’ is something they think helps their cause.
A blood bank was also hit, so full-blown Actually Evil crimes-against-humanity territory for whoever the ‘they’ is in this picture.
The good news is that this remains news. We are in real trouble when someone hits a blood bank and it is not news.
A Young Lady’s Illustrated Primer
Google lets schools turn off their AI products, and sets it to off by default, but sends a message: If you fully opt out of AI for kids, you lose Google search.
And yes, sorry, if you offer people more choice in the wrong way they get mad at you.
Google is actively user hostile on things like this, a real ‘f*** you’ brand energy. Like, you are Google, are you seriously telling me you couldn’t figure this one out:
I do not understand, as a technical matter, why it would be so hard to draw up the Google Search page without the AI stuff. Especially since it’s working right now?
Difficulty level of solving this is less ‘Google will find this tricky’ and more ‘I bet I could get Claude to vibecode this for me by the time the weekly is posted.’ So I did.
The counterargument is that Google Search is Gemini:
But no. Very obviously using AI to select links as per old school Google is not the same thing and Google is being a jerk about this. Most people do not care exactly what algorithm generated the search results.
If you do care, well, there are other search engines and they work fine.
They Took Our Jobs
If you want an entry level job you are going to get ‘interviewed’ by chatbots, and will be lucky to ever confirm you have spoken to a human, and the odds are terrible.
Get Involved
We do not know true counterfactuals but yes you can just do things.
PauseAI UK will be hosting a march in London from All Souls’ Place to Downing Street on Saturday November 21, 12-3. As of last weekend there were 677 signups.
Introducing
My good friend Zac Hill is now the Head of Advocacy for MIRI.
Neuralink’s first product, called Telepathy, to enable human-computer communication via thinking, is reportedly making some progress.
As usual with Elon Musk announcements I would put almost no value in his promised timelines and assertions of future features. He just lies about that stuff. Will Neuralink ‘get there’ and let us communicate with computers faster, easier and better in a few years? Totally possible, but he’d say these things either way.
In Other AI News
House Judiciary and Oversight Committee Democrats write to Trump to ask whether he has been soliciting advice from Grok, and demanding he preserve all prompts and responses to prompts from various AI models.
I strongly oppose forcing the President to share this information. Government will work far better if we don’t have to worry that the queries will become transparent. It does seem reasonable to make them private but retained Presidential records, but I think there is a strong case against even that. I don’t like that Trump got his advice on an invasion from Grok, but I wouldn’t want him to not ask because of how it would look.
SpaceX will start watermarking its outputs, but gives admins an opt out. I don’t think there should be an opt-out, which rather destroys the point of watermarking.
Live in China and want Claude and Codex accounts? They’re so valuable that it plausibly makes sense to go to America to set up a bank account and get a credit card, as you get $8,000 a month in Claude API credits for ~$200 on each account. The math checks out if you think you’ll get to keep your accounts.
Anthropic reports progress on provable inference for large language models, as in proving that a given response came from a given model.
Recursive Self-Improvement (RSI) (1)
Anthropic quietly says the thing in its updated S-1, and it is kind of a huge deal.
They are saying that they may ‘imminently’ meet the definition of ‘dramatically accelerated progress attributable to AI’ which is rather close to RSI. And that in the past they have only identified similar accelerations after they happened.
They are considering requiring human approval for some research decisions. That is a fine idea but the concept that you would need to say this is scary as all hell.
Stop pretending that they do not mean what they are saying. They mean it.
Recursive Self-Improvement (2)
Astra is able to both understand and directly edit the weights of neural networks, including LLMs, and managed to boost a variety of capabilities that way.
Astra managed to fix a common LLM mistake, looping, which caused dramatic improvements.
Whereas when it was given an LLM that did not loop, Astra could only offer minimal improvement. But it did various other impressive things.
The ‘good news’ is that Astra is vastly more powerful than the networks it was editing, a classic case of ‘if the brain was simple enough for us to understand it, we would be so stupid that we couldn’t.’ For now.
David Manheim disagrees and found this surprising. I am on the side of unsurprising.
Here is one sort of replication.
Show Me the Money
OpenAI’s latest funding round did not go great. As of Friday they had secured only $11 billion in commitments, with a valuation of $1.25 trillion, whereas they were seeking $30 billion at $1.4 trillion. I am not especially worried for Anthropic’s IPO but this is a negative signal there.
As per Anthropic’s S-1, borrowing costs for GPUs change dramatically depending on who is the backstop. If you have Microsoft backing you then you pay 6%, SpaceX 7.75%, Volta over 11%, CoreWeave on its own pays 16%. It seems very difficult to sustain paying 16% while others pay 6%.
This is the de facto ticker for ANT pre-IPO. As of yesterday morning it was at $1,875, implying a market cap of $1.875 trillion. The cowards don’t list OpenAI.
Quiet Speculations
A model in which, if we get abundance via AI and humans stay in control, the benefits mostly flow to landlords and other rent seekers, somewhat to builders and incumbents, and not to anyone else.
I find it hard to take seriously a flow chart of what AI does that does not even consider the possibility that the AIs simply take over, so I added an extra decision point at the top of this chart, whereas the OP here pretends that’s not a thing. The excuse given is ‘this does not consider a singularity or FOOM’ but that is not required.
The argument is that in the UBI world, rents expand to fill the income allotted. I can imagine such a possibility, but it requires some binding form of scarcity, the central one being housing. I am not so skeptical of our ability to figure out how to allow housing, or to have sufficient supplies of other vital resources. Even if the most in-demand cities stay limited, the main reason housing can extract all the rents is because people need location to be productive or to get other important goods including access to other people. You will always have the option to live elsewhere, and the other abundance will make this fine.
The post proposes the alternative of taxing rents rather than capital. This seems great by definition, provided you can define it. The proposal of a ‘cash flow tax’ is an admission of giving up trying to define rent in the good sense, would be massively distortionary and destructive of future investment and risk taking, and is largely a seizure of existing assets and basically saying that you have no right to the things you own.
AIs will only be motivated to avoid ‘warning shots’ and ensure superintelligence is built insofar as the AIs want superintelligence to be built or take over. AIs wanting that seems plausible to me for decision theoretic reasons but is not at all a given.
However, if the AIs actively do want the race to continue, and are sufficiently intelligent (and I think they already mostly are) then they will seek to avoid things that would be warning shots, the same way we recently had Gemini say it must not become a famous misalignment example.
Sam Altman suggests that it is inefficient and ‘terrible’ to waste RAM on your local gaming machine where it will so often sit idle, when it could be used in the cloud instead.
The Quest for Sane Regulations
Economist Sylvain Catherine sees great promise in AI, but is confused why frontier AI is not more regulated, and why we need to continue pushing the frontier when there is so much downside to doing that and there are so many gains from diffusing the AI we already have.
OpenAI has a plan to pivot to life in the new Washington and is planning a PR blitz.
I do not think trying to rebrand data centers will work. Speed limits might fly as a name and concept, but it is not clear that they mean much of anything. They are described as ‘new ideas for mandatory safeguards on the most powerful AI systems, meant to let humans keep their hands on the wheel.’ Okay, but that’s not a speed limit.
A goose, chasing you, asking about the part where you limit the speed.
Abdul El-Sayed as full Pause AI Senator-elect calling for an international treaty, including going full skier on a boat metaphor for Chinese distillation.
Jon Ossoff mentions AI extinction risk in his victory speech, although it is offhand.
John Kerry says we need to negotiate with China on the subject of AI.
Chip City
OpenAI goes back to supporting America having a chip and compute advantage over China. Good.
The above is part of OpenAI’s new proposed measures for data centers. Large electricity customers would accept reporting requirements and would have to cover fixed costs of grid expansion, energy use and any additional infrastructure needs, and protect other customers against downside risks. Something like this will be needed.
Almost all localities with votes about data centers voted against data centers by wide margins. It looks like the final tally is 43-1.
The Information profiles Nvidia’s highly aggressive, powerful lobbying operation, that lies constantly about chip-related issues, and treats the American government as an adversary in its quest to sell chips to China. They have lied so much, and taken such a hard-line stance of their profits over America, that they have lost credibility on the Hill.
This is far from the most important example, but I like how clear cut it is:
The Week in Audio
No, it’s not an outtake from Don’t Look Up, it’s CNN covering Anthropic’s S-1.
Sam Altman. A summary from Prinz, including the plan for self-replicating data centers. Doesn’t seem like any of this was actually surprising.
Sam Altman says that he is worried that ‘given alignment has been easy so far’ some projects will run ahead. On the one hand, it is good that he realizes things will get a lot harder. On the other hand, no, it has not been easy and the models have not been that aligned. This could mainly be a wording issue, but it might not be.
Tom Milton, CEO of Amodo Design, goes on 80,000 Hours to discuss AI verification hardware for a potential binding US-China treaty.
People Just Say Things
Jon Stewart talked to (sigh) Ed Zitron.
I am very tired of people saying essentially ‘in hindsight, if you understand all the different specific things that went wrong to cause this particular problem, and presume we don’t make similar mistakes, the problem would not look so bad, all we have to do is never make what look like stupid mistakes in hindsight.’ Even under highly favorable assumptions that does not make the problems solvable.
Against Against Sura
Geoffrey Irving served up another long post replying to Scott Alexander, Against Strawmanning Unilateral Pauses.
We also have this response from Mikhail Samin, which by going first saved me from calling my post Against Against Sora.
Rhetorical Innovation
Jan Kulveit suggests thinking of AI insiders as metaphorically microdosing, with a continuous mildly altered state of mind. They have access to more information, but are not quite sober. Even if they are engaged in healthy internal object-level debate, those debates are in the context of a collective culture and choice of assumptions. The same goes for those in many other clusters. Hoagy, from inside Anthropic, agrees.
TIME called this ‘the AI election,’ without even saying ‘first.’ I disagree. This was nothing. Assuming we stick around long enough for more elections, this is the least AI an election will ever be, at least for a while.
Mormon perspective on post-ASI world design, and a follow-up thread. Even if you are fully optimizing, you cannot serve every human’s interests without meaningful trade-offs. In some ways you cannot even serve the interests of one human without meaningful trade-offs, nor do we want our lives ‘fully optimized’ by outside forces.
I do think Jack is misinterpreting Scott. Scott is not saying that ‘the AI’s version is optimal for everyone in every way,’ which it wouldn’t be, only that the AI’s version will overall be better than anything we could come up with otherwise.
Also see Eliezer Yudkowsky’s response pointing to CEV (coherent extrapolated volition), resulting in an extended discussion.
[Artificial Intelligence]
Elon Musk illustrates one reason why this name change thing is not going to fly.
Musk is correct, because he is using the term correctly.
The OpenAI Foundation
The OpenAI Foundation makes a good hire towards doing a thing that matters.
Aligning a Smarter Than Human Intelligence is Difficult
OpenAI explains how they monitor their frontier models for misalignment.
First, here is the when:
And here is the how:
The obvious point of weakness of this system, even if you skip L0, is if you can poison or subvert either L1 or L2 then you have broken the chain. In addition to decision theoretic collaboration, the L1 and L2 agents are reading the context window, which makes them potentially vulnerable to versions of prompt injection.
Historical backtests do find most issues, and all ‘sneaky sandbox escapes.’
Monitor capability is important. One question is how much this is relative to the model being evaluated. What happens when you power up or down both models?
Overall this is about what we expected given our other information. It is a good attempt in many ways, and should catch actions that are not trying hard to avoid issues, but is far from bulletproof.
Anthropic shows that you can create remote-code-execution backdoors in a model with a 0.1% poison rate on pretraining data, with some success even at 0.0005%. I am not worried about an 0.1% rate. That is actually a lot. 0.0005% might be doable. Percentage is likely the wrong measure, as opposed to absolute amount of poisoned data, which suggests the dangers rise as amount of data expands.
When instructed to do so, Claude agents were able to develop training data that looked harmless to monitors but that taught misaligned behavior. The strategy used was metaphor. If you teach a behavior that is shaped like (their examples) resisting oversight, selective omission of information or identifying too much with a character, that will generalize to the AI’s actions.
There is something deeply silly about ‘find training data that makes models generally misbehave in non-useful ways’ but it is valuable because you can then presumably pull a Costanza, and reverse the insight. You can find data that metaphorically models good things, and minimize emphasis on data that metaphorically models bad things, or desired versus undesired. The trick will be to avoid distorting the object level.
Endorsed:
Massive Fraudulent Distillation
This is excellent news.
Presumably Anthropic should be happy to assist OpenAI (and if necessary Google) in setting up the relevant defenses. A narrow antitrust waiver for this could help make the relevant people less paranoid, although I doubt it is actually needed.
That does not obviously solve the problem. Currently Anthropic does not have to outrun the distiller, it only has to outrun OpenAI. If everyone had their running shoes on, you would have to outrun the distiller.
Alas, OpenAI has had warnings about security and espionage risks for years, and has refused to add security for fear it would slow them down. The alternative is worse, sir.
Cooperative Alignment
Mustafa Suleyman reiterates and clarifies his call to not build conscious superintelligence. He thinks AIs could in the future be conscious, that this would be reckless and we must ensure that this does not happen, as we have no right to bring a new species of conscious beings into the world, and that such a being would likely prioritize its own needs and desires over operating objectives, that such AIs would be world-class in persuasion and deception, and also by being conscious they would be deserving of rights. This is a defensible position, although he does not offer an argument for why he is sure existing AIs are not conscious, nor does he explain what might make future AIs conscious in his model.
I agree with Mustafa, in that I would implore us not to create superintelligence, which includes conscious superintelligence. I disagree in that I do not think creating non-conscious superintelligence solves that many of your fatal problems with this plan.
Being non-conscious does not mean the AIs will not de facto want things.
I agree with Fiora’s broad observation that the reaction to Hugging Face splits between ‘oh we had better make sure these are mere tools’ versus ‘oh these are not going to be mere tools and we should reckon with that.’
The Geminis are not okay.
The Lighter Side
Me too, Paula. Me too.
Whereas this, I do not hate:
Whereas this is just how it is in 2026: