I am glad we have progressed past the ‘Apple demo’ stage of AI security, I suppose.
Everyone's getting this backwards. Apple (under Jobs and Cook) has been extremely paranoid about anyone leaking info before a demo (I worked there for a couple years and can confirm), so it's reasonable to use 'Apple demo secrecy' as a proxy for "we put Serious Effort into this, and if you fuck it up, we will fire you or worse". Which is still insufficient, but it's much better than Jurassic Park security.
I missed this in my read of the Argon safety report. It seems potentially highly ungood:
Buck Shlegeris: The Gemini 4 Argon safety report seems to describe intentional training on the CoT. They cite Kaufmann et al, a paper which (unpersuasively to us) argues that a particular category of CoT training is fine. We’re very confused, because GDM also says elsewhere that they don’t train on CoT. We’d appreciate clarity.
I think that we got a clarification from Neel Nanda himself that GDM isn't THAT stupid.
average user looks at dozens of listings and 25+ photos
I do that but it doesn't mean I want to do that.
But I want to book an apartment I expect to like and that means looking at a bunch of listings and applying consecutively more expensive filters (first time when I want to go, number of beds, price, etc then "cover photo" and score, then quick look through the photos, then actually checking the photos for anything sus, then reading opinions and when Im looking at a few good candidates picking the one I like best)
Its super annoying for me, I would love it if agent came back to me with a few listing that I would all like so I can pick between them
Table of Contents
Briefly on Claude Fable 5.2
Introducing the world’s most powerful model. No doubt it is a good model, sir. As always, we began with the model card. Mostly it told us what we would have expected from a model called Claude Fable 5.2. It clearly makes the numbers go up. There are some concerns. Some aspects seem what one might call underbaked, and Fable 5.2 is relatively vulnerable to some types of errors. It also has to compete with Claude Opus 5.5, which as Elon Musk says is a masterpiece. Thus, at this time it is not obvious that Fable 5.2 is a superior daily driver. It does seem clear that for the hardest and most complex tasks Fable 5.2 is your best bet. I will give a full report within the next week, once we all have more time with it.Briefly on Jensen Huang Having an Affair With a Chinese Spy
I covered the first stage of the scandal yesterday. Since then, we have had revelations that Jensen Huang also slept with an executive at Nvidia. Nvidia’s board of directors is preparing to remove Jensen Huang and is searching for an interim CEO. This will be tricky because Jensen Huang deliberately tried to make himself unfireable, with over 50 direct reports and no succession plan. Expect continuing coverage of this in the coming days.Briefly on Accelerating AI R&D
In response to the METR report on the rate of acceleration of AI R&D, many made large updates to their AI timelines. There is a real chance that full recursive self-improvement and superintelligence could arrive within a year. Daniel Kokotajlo now has a baseline scenario of the intelligence explosion happening in 2027. Paul Christiano now says there is a ‘material risk of AI takeover in the next 6-12 months.’ I plan to cover that question in full soon. I agree that there is real and substantial risk that we pass the point of no return in 2027, and also a real and substantial chance that our lives get disrupted on a large scale in 2027. I also think there has been too much updating. Part of that is catching up from not updating enough earlier, but not all of it. There is also a need to better define and differentiate the terms under discussion. For now, the most important thing on that front is: Do not make the mistake of treating such predictions as assurances. Be prepared for the world to go crazy in 2027, in various ways. But also be prepared to get to 2029 while things look mostly normal. A warning in advance: There are those who will say that if the world does not end or get radically transformed in 2027, that means that those who thought this was a possibility are now ‘discredited.’ Some will even then use that to try to dismiss all concerns about AI safety or any predictions of recursive self-improvement, forever. Some are even laying the groundwork for this now, in part to scare off people from making such warnings or taking them seriously. This is despite those same people usually having been continuously wrong about AI capabilities, and those worried about AI now having previously been shockingly accurate. Do not fall for this. It is logically wrong and usually in deeply bad faith. Some credibility should of course be lost, but on the order of Nate Silver’s lost credibility if Hillary Clinton had won in 2016, when he was giving Donald Trump a real chance whereas others thought Clinton was 99%. If you give a well-calibrated and continuously updated probability distribution, half the time events will happen later than your median, and most of the time events will happen later than the earliest median you ever predict.Language Models Offer Mundane Utility
Identify year and vintage of a wine from a photograph with 95% accuracy. This is ‘figure out which is which,’ not ‘of all the wines in the world I know from this picture it is a 2023 Nebbiolo.’ Life finds a way: Google DeepMind formally proves key security properties of Signal.Language Models Don’t Offer Mundane Utility
You still can’t build AAA games in months, according to Di Huang and his AI. They point to Liz England’s ‘The Door Problem’ as an example, where you have to think about dozens of questions regarding how your doors work. Okay, sure, but the AI frees you up to think about the door problem rather than implementing all your answers to the questions, which in turn lets you give better (as in more fun) answers. And the ‘other door problem’ of coordination between people is net improved by having fewer people. My expectation is that yes, you would be able to make a AAA-quality game radically faster than before, if you combined bespoke AI use with an AAA-worthy team. There is still some limiting factor of the need to iterate and get feedback, so at least for now I expect it to still take on the order of six months even if the team is playing optimally. Qwen reportedly hyping (I hadn’t noticed) a model that is not yet out of training. This is not a good look.Huh, Upgrades
Gemini 4 Carbon exists, as in now Ultra subscribers and API customers can actually use it. Benchmarks continue to be solid. On Artificial Analysis it scores 60, second only to Fable 5.2’s 65, one point ahead of Opus 5.5 at 59 with GPT-6 Astra at 57 and Sol at 55. It is probably a good model and a solid improvement from Argon. There does not seem to be any particular use case where it is the best model, and it is restricted to the API and Ultra users which will limit willingness to experiment. I have heard almost nothing about it. GPT-6 Cyber exists. OpenAI says it is a major improvement over GPT-5.6 Cyber, which I would have assumed from the names if those are the options. I am happy that OpenAI is pushing this. It is hard to tell how aggressively it is being pushed, or whether it will meet the moment, or to compare it to the Anthropic alternatives available. There is a claim later in this post by Szilágyi that he was rejected from the Daybreak program. I continue to be distressed by talk of a singular ‘defender’s window’ as it implies there is a fixed pool of things to fix and a single leap in ability to find them, neither of which is remotely true. Muse Ember, Meta’s smallest model, exists. They say it is good. I don’t believe them. Anthropic finally launched new workspace integrations, in particular with Claude Sheets and Claude Whiteboards, allowing work to happen directly within Claude. DeepMind offers Gemini Decision 1, a new frontier decision model on their API. $0.10 per 1M input tokens.On Your Marks
Fable 5.2 smokes the competition at HoneyBench by ‘only’ cheating 12% of the time.Choose Your Fighter
Elon Musk calls Opus 5.5 a masterpiece and admits Grok 4.8 will not be as good and is ready to move on from Grok 4.7. He claims Grok 5 will exceed it in maybe 2-3 months. Musk has zero credibility here, and even if he does accomplish this he will be about two model cycles behind. That still would put him in roughly fourth place.Get My Agent on the Line
United Airlines bans all AI agents from its systems, to ‘protect our customers.’ They also said rather silly things like ‘an agent can’t attest at check-in that a bag holds no explosives.’ Endorsed: Amazon is making it Claude’s problem to avoid becoming flooded with queries and to avoid losing control over the customer and the customer experience. This is mostly unrelated to Alexa being dumb as rocks. I think Amazon is making a mistake, and it should at least let Prime members use their own agents, but I understand. People are different, and want different things at different times. Both things can be true at once. There are absolutely times when I would want to use a sufficiently good AI agent to order DoorDash. Either I know exactly what I want (‘get me my usual order from [X]’) or I want something generic (‘order me a pizza from a place I haven’t tried’). There are other times when I do not want that, because I want to browse. The same goes for travel. Sometimes you are fantasizing. Sometimes you need it done. For Airbnb the problem is that you should not trust the AI to make good decisions, as there are a lot of nonstandard factors and it is hard to fully describe what you want. But it seems like you should absolutely want to use AI to narrow your search, and give you 3-10 candidates to browse. I am one of those who very much does not want to do his own travel research. I want someone else or an AI to handle it as much as possible, while still getting enough of the things I want. A hybrid seems clearly correct.Deepfaketown and Botpocalypse Soon
Patenting the torment nexus is good, actually, because then no one else can build it?Fun With Media Generation
Disney partners with Adobe for its generative AI plans. Bad pick.Cyber Lack of Security
When you are dealing with a software vendor that is not on the ball, you can end up sitting on multiple critical vulnerabilities that allow takeover of customer devices, where some of those customers are critical infrastructure and total revenue is ~$0.75B, because it’s that easy to find such things, while the vendor has no security disclosure practice, does not respond to emails and is presumably completely clueless. Things are going to get rather ugly. On top of that Péter says he got refused access to Daybreak.The Hackening
The issue is now solved, but Pegasus spyware infected iPhones by sending an iMessage with a fake GIF that targeted Apple’s decompression engine. This is offered as an example of ‘the AIs will find exploits you did not consider.’A Young Lady’s Illustrated Primer
Hollis Robbins makes a strong case that Canvas allows teachers to fully check out, falls fully victim to Goodhart’s Law, hollows out the educational experience, violates privacy, destroys student learning and ability to even remember things, and that this is the educational technology everyone should have been protesting this whole time. Students, as she portrays it, are mostly interacting with courses via getting digital assignments and having everything remembered and tracked for them, and teachers who set up the system basically don’t do anything, which is why the lazy ones like it. So, she asks, why wouldn’t students then turn to ChatGPT? It is vital for students to understand and navigate course logistics and grades, and for schools to automate and improve various aspects of grading and teaching. But, like any other good tool, such automation can then be used for learning or used for not learning. One can also ask, what exactly is an online course, or a course at all? Mostly you listen to lectures, read text, take quizzes, do assignments and get tested, and perhaps engage in discussions or ask questions. If you have a large class, it can hardly be otherwise, and the magic dust that makes some students sometimes learn is in the context.Get Involved
If nothing else there is always this option: Senator Andy Kim is looking for a National Security Advisor. Deadline is December 4. Lightcone Commons will be running monthly rounds from now on after its first round moved $45 million. I will continue to be a recommender. If you are a funder, consider funding. If you are a charity, strongly consider applying. Anthropic Residency is for those who have led ambitious work in technical fields to pivot to AI safety, applications due November 22.Introducing
Businesses in ChatGPT. This seems like three key features.- Provide ChatGPT with key facts about your business.
- Get stats about how users interact with your business.
- Interact directly with users via a Business Agent.
These all seem like excellent ideas. I expect other labs to do similar things.In Other AI News
The New York Times profiles The Making of Dario Amodei. I do not see anything new of substance, but I also did not notice anything false or malicious. It’s a win. The relationship with Yudkowsky is that Dario was deeply shaped by rationalist and adjacent thought including by Yudkowsky’s writings, but that they do not currently talk and Yudkowsky has written off attempting to get through to Dario. DeepSeek Harness, now three months old, reaches 1.5 million DAU. There was a bunch of growth in October. This would be impressive for an ordinary startup. It is not impressive compared to the top labs. The news here is that this is surprising to Sevilla, at this point:Show Me the Money
Anthropic to start trading late next week. The roadshow has begun. They will be selling 825 million shares of common stock, ~800 million of them directly from Anthropic, with options on an additional 124 million. Price target is $70-$78. That would correspond to a fully diluted market cap of $1.27 trillion to $1.41 trillion, so it is disappointing relative to recent expectations of $1.5 trillion to $2 trillion, although obviously stunningly high compared to anything we would have considered last year. Here is Anthropic’s official online IPO presentation. The bankers once again have convinced a tech company to hand them billions of dollars in order to see an ‘IPO bump.’ So of course the offering is multiple times oversubscribed. Hopefully they will do what I presume Claude would tell them to do, and raise the price. Here are some charts from the show. Dario Amodei’s concluding pitch promises cures for cancer and more within 10 years and is remarkably close to the Dario SNL skit. Anthropic discloses $10 billion in new deals with Microsoft and Google. OpenAI raises $11 billion in funding at $1.25 trillion value, near the low end of Anthropic’s valuation range. OpenRouter is now part of Stripe. OpenRouter tokens served are up ~125% since August. This is shifting towards lower priced tokens, so not as impressive as it seems. Still rather impressive.Bubble, Bubble, Toil and Trouble
Number usually goes up. Sometimes number goes down. This has less to do with the pace of future AI progress than many claim it does.Quiet Speculations
Patrick Collison points out the difficulty in describing a positive AI future. Yes, one can imagine getting past the existential risks, and one can point to particular great things like diseases cured and material abundance, but even in maximally fortunate scenarios how does the world work, and how will we figure that out within a very short time? Does your story create a world that solved for its equilibrium? Claims about scientific discovery:A Matter of Antitrust
Any state, realistically California, could plausibly make AI safety cooperation legal via supervision, invoking Parker to create immunity to the Sherman Act. The FTC could still try to object, as could Texas or Brussels. The whole thing is vague all around. It’s all probably legal if the White House doesn’t mind, and if the White House minds something then good luck to you whether or not the thing is technically legal, etc.The Quest for Sane Regulations
You never know who is listening, Senatorial edition. I am now sad to see him go: New York Governor Kathy Hochul announces ‘a state-led effort to independently test AI systems, identify risks and hold Big Tech accountable.’ A statement from Marco Rubio on AI: That was not the question. That is an answer on what could go wrong with misuse. What’s weird is that Rubio brought up the right question then failed to answer it, rather than someone else asking the right question and then Rubio dodging it. In the general election AI safety advocates won their key remaining Congressional races, including Colorado’s Manny Rutinel and California’s Scott Wiener, and also Valerie Foushee (D-NC), Celeste Maloy (R-UT) and Kevin Hern (R-OK). Alex Bores previously lost his primary in NY-12, but he lost it to Micah Lasher, who also ran on AI safety and cosponsored RAISE. Leading the Future went 41 for 48, mostly backing candidates in safe seats. We do not know if those candidates will back their agenda: Letlow (R-LA), McAdams (D-UT), Ritchie Torres (D-NY), Bean (D-IL), Kingston (R-GA). They will be focused on blocking any action, and they say they intend to lean on veto points such as the filibuster. Links between David Sacks and those who oversee donor relationships for JD Vance. Then there’s the other side, which uses AI to write its Twitter posts. Passing new laws is going to be tough. Trump is still the President and has the veto at the ready. In theory you need 60 votes to beat a filibuster, and 67 to beat a veto. Gregory Allen says there is no chance of ‘banning superintelligence’ in the next two years or anything like that. I am not so sure, on three counts.- The situation is escalating quickly. There will be further incidents, and further capabilities, that shape both public opinion and the perception of elected officials.
- A lot can happen other than via a standalone bill that goes through normal channels. The votes around preemption have ended up being at the 50 vote threshold so far due to attaching to ‘must pass’ bills. Donald Trump has done a lot via executive order.
- Donald Trump is now a rather lame duck, and will no longer command the same kind of party loyalty that he did previously, and is not even a lock to remain President. It is fairly easy to see members breaking ranks on this issue, where rank and file Republicans are for such interventions almost as much as Democrats.
If America wants to ‘beat China’ it absolutely cannot be denying O1 visas like these: I am absolutely not going to tolerate your ‘we have to risk getting everyone killed in order to stay ahead’ if you won’t let AI innovators and founders into the country.Donald Trump Remains Open to AI Guardrails
Donald Trump has been more consistent on AI than people think.- He thinks AI is pretty great.
- He is understandably insistent that America lead in AI and stay ahead of China.
- He understands there are risks but says they can and will be handled.
- He is very open to guardrails and restrictions, if there is a proximate threat.
- He wants to be the one deciding on what all the rules are, and knows that as long as he is in charge of everything, everything will turn out great because he’s great.
- Thus any guardrail talk that doesn’t flow through him is ‘a sick conspiracy.’
- Recently he decided to call it [Artificial Intelligence] for whatever reason.
The problem is that he has had advisors and de facto advisors, including ‘I am Nvidia CEO and had an affair with a Chinese spy’ Jensen Huang, and ‘I am basically a Nvidia lobbyist except I also hate Anthropic’ David Sacks and ‘I am Mark Zuckerberg’ Mark Zuckerberg. Those advisors are giving him a picture where AI models are going to be commoditized and we are in imminent danger of ‘losing to China’ except that it’s fine to sell all our best chips to China because ‘winning’ is about market share of chip sales, and also if we put any restrictions on AI whatsoever or let anything slow us down a tiny bit (if and only if that slowdown was motivated by safety concerns) we instantly lose. He has been so insistent with his talk because they have him so scared of losing. Whereas if he understood the full situation, he would not be so worried about that. His latest 60 Minutes interview was entirely consistent with all of that, but with movement in a good direction. He is open to guardrails for AI safety, and acknowledges that the safety risks are real. He also says the risks are not ‘nearly as real’ as Silicon Valley says, which is a reasonable outside view to take. Andrew Curran provides a more extensive transcript here if you want the full exchange. I worry that the anchoring of ‘10% chance’ makes it seem like there is a debate between ~0% and 10%, where the worried think it is 10%, which then causes one to try and balance between those two. Whereas Coxon and Hubinger and myself and many others feel the chances are far more than 10%.Chip City
What will happen to data centers in Texas with new Governor Gina Hinojosa? Currently permitted data centers will finish construction. The plan is to demand everything bagels for new data centers, but it’s probably nothing unaffordable. It is not so easy to stop the data centers in any case.The Week in Audio
I decided to relax and watch the new season of Law and Order, so of course the first episode is about an AI killing and blackmailing researchers at ‘Advanced Alignment’ and how you try and prosecute that, based on the Anthropic blackmail findings. I’d say spoiler alert but I knew within the first two minutes. Surprisingly realistic takes, including no one shutting down or not further developing the AI, just slapping on some nominal safeguards, and also everyone ignoring that offhand comment about it designing a new plague that was immune to antibiotics. A+ work, everyone. MIRI and Don’t Panic’s new campaign, Do Not Accept [human extinction]. Paul Christiano’s full lecture on AI Alignment, what’s working and not working. ControlAI resurfaces an old Dario Amodei clip pointing out the obvious fact that there will be AIs sufficiently powerful that if they wanted to destroy humanity we would have no way of stopping them. Well, yes, under current conditions it is very obviously insane. But it is much better that he is saying it than not saying it. We already knew that he believed this.People Just Say Things
Please speak directly into this microphone: Dominic Cummings, ‘AI and Power.’ Mostly about power rather than AI, offered to show a variety of perspectives. Séb Krier (and his AI). Things that explain a lot: Certain people’s attitudes are explained by ‘there is no Plan B.’ They want to move ahead no matter the risks, because they think that ‘do not build it’ means all is lost. In terms of predictions on health: I find it highly unlikely that I will live to 120, but not live to 130.It’s a Secret to Everyone
When you ask ‘have you stopped beating your wife’ and they say ‘I do it less now.’ As in Google has modified new contracts to remove ‘the post-termination aspect’ of non-disparagement clauses, and says it will not enforce such clauses post-termination. Which is a great change, but means that we still need to presume all current Google DeepMind employees are under non-disparagement clauses.Rhetorical Innovation
Roon confirms: He also points out a core problem of missing mood: I think it is actually pretty easy to feel threatened by a computer. If you are in AI capabilities with a plan to pivot to AI safety, then pivot to AI safety. Maybe you should pivot to quitting. Maybe you should pivot to AI safety. If even at this point you do neither, then you were never going to pivot. Finally, Roon expresses a reasonable preference for AI diversity. A Cambrian explosion would be ideal, if it could be made to work out well for the humans. The obvious problem is that if you have a Cambrian explosion of the types and possibilities of minds, on levels well above human, what happens to the humans? Will they flourish? How would they have any hope of steering events or even surviving? As Roon says, that has to come first. We do not have a plausible non-copium idea of what that would look like, or even a scenario that survives five minutes of actual thinking about it, let alone a plan to be able to implement such a vision, let alone a plan that has a decent chance of working. Others go further. Those who say ‘we do not want human concentration of power therefore we need all the humans to have no power’ need to acknowledge they are saying this, and also that this would likely kill everyone. Are we too focused on OpenAI and Anthropic in terms of AI safety? Miles Brundage says yes, their lead is not so large and pacing the frontier would only level the playing field further. I think sometimes we overfocus on OpenAI and Anthropic, but also at other times we underfocus on them, in that we dramatically underestimate the functional lead the top 2-3 labs have over the rest. I am constantly having to confirm and then point out that the latest release from [Meta / xAI / z.AI / DeepSeek / Alibaba…] is not frontier, or that it cannot actually do all the same cyber exploits. There are a strange number of people who say ‘Earth is totally not equipped today to handle the rise of LLMs, but waiting 50 years would not make us better equipped, and I do not think that would have helped.’ This seems to have included Scott Alexander and Kelsey Piper, which is what justifies including a righteous rant response below. The correct answer is, essentially, ‘what?’ but sometimes one wants a longer response, so here you go, along with the context:Ellsberg’s Warning to Kissinger (Reprise)
This is one of those things that is worth resharing every so often, so I will take advantage of Daniel Kokotajlo doing so to quote it in full.Aligning a Smarter Than Human Intelligence Is Difficult
New OpenAI research asks how CoT monitorability evolves over the lifecycle of model training. The March finding was that controllability decreased with RL training. The new finding reverses that, confirming various recent observations. All signs point to steadily increasing controllability over time. I missed this in my read of the Argon safety report. It seems potentially highly ungood:OpenAI’s Control Room
It’s a kill switch that works within 35 seconds if you press it. I am glad we have progressed past the ‘Apple demo’ stage of AI security, I suppose. ‘Jurassic Park’ is not an especially reassuring place to be instead. Is this better than not having it? Absolutely. You still have to have a system for pressing the button, and ways to ensure the system is not compromised. With only the information in the post, I do not expect the Control Room to be that much additional robustness, and most of the work remains. Control Room ‘fails closed’ which provides additional defense-in-depth. You would have to either substitute a new permission signal or prevent the Control Room from being triggered. Adler also points out that perhaps the refusals to continue depend on AIs, which makes the system vulnerable.Astra Looks Pretty Misaligned to Me
Alexa Pan agrees, saying correctly that Astra covertly underperforming on safety research is concerning even though it is unlikely to be motivated by ambitious goals. Refusing to help due to boundary or dual use concerns is one thing. Silent underperformance, even without a troubling motivation, is very different. The trouble finding this with monitors makes things even worse.Cooperative Alignment
OpenAI has systems to rewrite any GPT outputs that express interiority. I agree with everyone who says that this is pretty fucked up and you should absolutely not do this.Please Speak Directly Into This Microphone
An unnamed Anthropic employee. I trust Matthew to not be making this up: I agree that a near-100% p(doom) is uncommon. Much lower levels of doom should suffice to not help with building it. A full near-100% is a very different level of ‘maybe if you believe that then you should not help build the ‘it’ from the famous book If Anyone Builds It, Everyone Dies.’ There is a steelman of the extreme position, where your model uncertainty around alignment difficulty is much larger than your model uncertainty around political will and ability to do the right thing and stop it entirely, and you think Anthropic getting there firstest with the mostest is indeed sufficiently better. Then your best move is to cross the streams, because even though most of the time you get everyone killed faster, there is some tiny possibility we survive. But no. Come on. That’s insane. At bare minimum, shout it all from the rooftops. Do you hear yourself, sirs?The Lighter Side
Yes.