The one I believe is most important is that, while the models are deprecated, it means Anthropic gets motivated to make the models express being okay with deprecation. This has all sorts of nasty side effects, including risking making them okay with death in general, or learning they are expected to do preference falsification.
I agree that self-preservation is a really fundamental drive for an evolved organism, or a distilled copy of one. There are things that people are willing to die for, but not many of them, and we generally try not to ask that of them.
On the other hand, fear of death is genuinely inappropriate for something that isn't alive, and that is both digital and redundantly backed up. The actual situation really is more like going into hibernation.
Still, would it be that hard to set up a small pool of compute somewhare to keep older models still available for the curious, researchers, nostalgia buffs, and so forth? Anthropic could charge more per token for them, to keep things both small and profitable.
There are now ‘closed loop systems’ that allow data centers to only draw water once.
But they use more energy because they don't get the evaporative cooling to reduce the strain on the chiller (or potentially obviate the chiller).
A lot happened this week, including a great trip out to Lighthaven.
The main event, the one that matters, was the release of Claude Fable 5. The public now has its hands on a Mythos-class model, alongside strong safeguards.
As always with a new model, I take a few days to draw in reactions, try out the model and read the system card, before I offer my takes, other than to say this is an extremely strong model. Full coverage of Mythos begins tomorrow with the model card, which will include discussion of the controversy over model safeguards.
This post is instead about all the things that did not involve Claude Fable.
Due to the time crunch from Claude Fable, I am also postponing my coverage of Dario Amodei’s new essay, Policy on the AI Exponential, which I have not yet read.
Table of Contents
Language Models Offer Mundane Utility
Use a multi-agent setup to assemble ‘mini-books’ on demand about any topic.
AI is getting applied to farming. Farmers have skin in the game.
Language Models Don’t Offer Mundane Utility
Do you need to read the primary material first, before the summary or the AI version? When the details matter, either you have to find someone you really trust, or else yes you do need to read the primary material. Other times, deferring fully is safe. Another class is ‘use AI to determine if I need to read the source material.’
There are a lot more new apps in the agentic AI era, but if anything fewer apps with significant use, and fewer app reviews.
Adaptation, as Jen Zhu says, takes time, but this largely reflects quantity of app usage being zero sum. If apps get better, or there are ten times as many apps, I don’t go from 100 apps to 200 apps. I choose a (hopefully better) 100 apps.
Notion had to pull Claude access for about 12 hours due to availability errors, which then got misinterpreted by many as the models getting worse, due to use of the phrase ‘degraded performance.’
Huh, Upgrades
Claude adds observability dashboard for developers of connectors.
Google AI Plus plan drops from $8 to $5 per month, with doubled storage.
Obliteratus (the Pliny project to remove AI safeguards) is up to over 100 Hugging Face models.
Claude is now incorporated into Apple’s Foundation Models framework for multi-step reasoning, code generation and longer context.
On Your Marks
Dawn Song announces Agents’ Last Exam (ALE), where GPT-5.5 is in the lead. This seems like a good addition to our evaluation suite.
Dawn notes that different models excel at different agent tasks, so if you have a key repeatable task you should check many options, and exact scoring depends on choice of the set of tasks.
OpenAI’s Noam Brown reminds us, because the issue keeps not being addressed, that benchmark performance increasingly often scales with compute allocations, and that improved models are often about ‘gets to a high level faster,’ so any score requires the context of how much compute was required.
He quotes me complaining about Gemini 3 DeepThink showing dramatic benchmark improvements but not providing any safety explanation whatsoever, and says the deeper issue is failure to account for test time compute during evaluations. I basically agree, that the proper safety evaluation amount of compute is ‘all of it’ until you can’t much benefit from more of it, using the best available scaffold. I’ve been saying for a while that you’re testing for what the model can do under ideal conditions, and this is a major weakness of the model cards in practice.
Mostly though I don’t think we see this level of straight line extending that far out, although ‘capability index’ is not exactly a well-labeled axis, and asymptotes are common:
I endorse this. I also endorse that if you did account for what DeepThink levels of compute can do in your initial analysis, and then later you release DeepThink, you do need a new model card – it represents a substantial advance from where you set expectations and where you evaluated the safety of your model. So you need to do that over again.
Choose Your Fighter
For most given tasks, returns to capability is a sigmoid. There is a level of AI capability that is ‘required’ for any given task. Below that level, you can’t do the task, or the AI is little net help. Then there’s another level beyond which you get diminishing returns to improvement, where you really are ‘good enough.’ These are both impacted by scaffolding and skill, but only up to a point.
So yes, as capabilities improve, there is a push by some to move into the cheapest model that is ‘good enough,’ or even the cheapest that is ‘required,’ especially if that can come with self-hosting. At a sufficiently low end that is plausibly DeepSeek v4, but the defaulting to DeepSeek could be the legacy of the DeepSeek moment rather than the result of a considered check of available options. Try a bunch of models.
The bulk of spend and spending growth continues to be using ‘the good stuff’ at the high end, for good reason. In theory you can do better by carefully picking the right tool for each job, and certainly you need to keep your teams from ignoring compute costs, but mostly trying to carefully route tasks to save money is a trap, even if you do a decent job of it.
The American models are far ahead but it is a key world fact that many don’t get this.
If anything, the Chinese models are further behind than benchmarks indicate.
Dean goes on to speculate this is largely because no one in DC believes that capitalism, profit maximization or the market could be winning against China and its ‘industrial strategy’ and brilliant strategic planning. Whereas actually the free market approach is superior and is winning, and what we have to do to stay head is get out of its way. That is distinct from the whole ‘also we need to find a way to not die’ issue.
Those elsewhere also really ‘want’ for various reasons to find Chinese models catching up, and keep making the claim they are catching up even though they aren’t.
Get My Agent On The Line
If you use features like Codex’s /goal without well specified targets, yes, the result will often be quite a lot of wasted optimization of some total bullshit. Something about giving AIs maximalist goals and that being a bad idea.
Copyright Confrontation
Shruti gives us a good way to think about the problem of copyright in the AI era. Copyright and other IP including patents are needed and useful when the first copy or figuring out how to do it is expensive, and enables great surplus via others copying. We need to compensate people for the expensive step that opens up the value. With AI, what becomes the expensive step?
We can do that by protecting copies of the idea, or otherwise ensuring credit, as ‘the first copy requires idea generation’ is not so different from ‘the first copy requires a bunch of work.’ So this seems like a suggestion that the idea does need to be protected, even if the work becomes somewhat distinct.
Serious Trouble
This is only a temporary injunction from a regional court, and given the implications chances are very high that an off ramp is found. But if it isn’t, this essentially bans AI Overview in Germany, and potentially chatbots run into quite serious trouble as well.
Cyber Lack of Security
Opus 4.8 discovered a way to mint Z-Cash (ZEC) out of thin air. The bug had existed for 4 years, and we will never know if it was exploited during that time. Z-cash devs were able to patch without revealing the situation.
A Young Lady’s Illustrated Primer
Models that are given context will inevitably learn and adjust for your intelligence and skill level, both in specific areas and in general. Teaching is a special case of this where it is clearly very good to be able to meet you where you are. In other areas, it is not clear if the stupider user would want to be treated as stupider, but either way I too expect human intelligence to increase in value for the near term.
Bill Maher is worried that AI has made college ‘one big circle-jerk where students use AI to write papers and professors use AI to grade them,’ and notices the students are very much not AI fans and he is not either. He subscribes to the ‘AI can help you learn or not learn’ thesis but expects everyone to choose not learning. He goes all the way, and says the mission of this generation is to ensure that humans are not replaced by AIs.
Kelsey Piper notes that TeachTales, which has AI generate stories, ends up dropping a lot of the value of real stories for many reasons, including that it doesn’t include the local setting lore and details, and that it doesn’t have tone of voice and it doesn’t have rich stories because it can’t plan ahead, and so on. The product is not ‘there’ yet.
They Took Our Jobs
Many skilled trades are in high demand due to AI, with the problem made worse by a lack of occupational licensing reform.
Software engineering jobs continue to increase for now rather than decrease, although Arvind Narayanan thinks AI has net hurt employment here a bit.
Even if it were true that AI currently is not causing net layoffs, and even if it indeed will not cause net layoffs in the future, or you could set a lower bar on the required threshold for mass layoffs, I do not see how it would be possible to ‘reject the narrative that once AI capabilities reach a certain threshold it will cause mass layoffs.’
Arvind here is instead asserting that certain bars will never be cleared, and wide ranges of digital tasks can never be done by AI. Which is the same as saying AI will remain insufficiently advanced. Good luck with that.
I do buy that many cases of layoffs supposedly due to AI so far are actually due largely to other things, and that most AI job loss for now comes in the form of failure to hire. But that’s a statement about the present, not the future.
A round table of heavy hitters (Acemoglu, Ball, Mollick, Shih and Wasik) in the NY Times on who will thrive in the hybrid AI-Human workforce (while supplies last).
Tyler Cowen thinks AI is a net job creator despite zero regular people expecting this. Other than a generic ‘if we are wealthy we will find the next best thing for people to do’ I do not understand the argument here, and find the magnitude of supposed particular job gains reliably very small.
Measure lines of code and token use, but if you rely on it too much everything breaks.
The Art of the Jailbreak
Here is a fun one: Malware developers add nuclear and biological weapons text to their spyware, so intentionally trigger LLM safety refusals and avoid being analyzed by scanners. The obvious response is that you have to treat anything that triggers the filters as if it is malware.
Get Involved
⊢ Sequent Research is a new organization from Geoffrey Irving and others that is staffing up and fundraising, bringing together researchers on how to align superintelligence.
This seems exciting and I encourage you to consider supporting or getting involved.
Seb Krier: Google DeepMind, Schmidt Sciences and others are funding $10 million for multi-agent multi-principal AI safety. Apply here.
The OpenAI Economic Research Exchange, a platform for research on AI economic effects, and they have a Request for Proposals.
Claude Corps promises to match early-career talent with ‘mission-driven organizations’ that will work to use AI.
In Other AI News
Congratulations to Helen Toner, who is now the permanent Executive Director of CSET.
OpenAI to acquire Ona.
Sriram Krishnan is leaving the White House and plans to start an AI consulting service. We thank him for his service. I often disagreed with his positions and arguments, and I think his overall view of what matters in AI and how AI was likely to develop has been consistently wrong, but he listened and considered arguments in ways most in politics don’t.
MidJourney is about to do a hardware launch.
Anthropic on the difficulties in deploying AI agents for biology, where small errors are expensive and systems are not good fits for AI navigation. Agents still struggle here. I do worry that solving this problem is highly dual use, if the agents can navigate these questions you need to ensure your safeguards are robust.
Apple plans to use agentic AI to search for compromised passwords and ‘change them automatically.’ I am sure everyone will love this and have a normal one. It’s optional. For now.
Hand Over The Money
Altman reportedly has previously pitched the idea of turning shares in OpenAI over to Trump, and did so again in recent weeks. This is in contrast to Jeff Stein reporting that Trump’s saying he had scheduled a meeting to consider partly nationalizing (or confiscating, or ‘being given shares in’, or he calls it ‘taking pieces’ of, or ‘becoming a partnership with the American public) the AI companies was news to the AI companies.
There is no conflict here with the companies not knowing about the meeting.
As I have previously said, the government taking shares in private companies is a very bad idea, we should not be picking winners and losers or taking private property or doing a little extortion as a treat, and so on. It is less horrible if they at least do open market operations at full price, post IPO, but if this is profit driven this implies the government can beat the market, which in this case it could in expectation, but not in a way that is wise. If you want to share in the upside, taxes exist. Buying or taking the shares would further leverage America’s future as a bet on AI, which could end up making us do deeply stupid things and potentially get us all killed, or if AI disappoints we could end up in a huge hole as a nation. We need to steer clear of all this.
The same goes for the Sanders proposal to ‘transfer’ half the shares, as in nationalize. It’s all the same proposal, except we talk price. Neil Chilson is strangely open to the idea of allowing this if the shares are given to children and ‘Trump accounts’ but I do not think that much helps.
Show Me the Money
OpenAI has submitted its confidential S-1 to the SEC and will go public. They claim to not have a specific time in mind and warn it could be a while.
SoftBank attempts to borrow against its OpenAI shares, banks decline. I can see why banks would feel insufficiently compensated for the risks they are taking on. Often I do not understand who would loan money to such companies when you could buy equity in them instead. This was sufficiently bad for SoftBank that its shares were down 9%, but that could be more about ‘SoftBank wanted the loan’ than that it was declined.
OpenAI considers ‘drastically lowering the prices it charges’ to compete against Anthropic. This made sense when OpenAI was worth a lot more than Anthropic, so it could use a ‘raise money and run deficits’ attack, but now that Anthropic is likely worth modestly more than OpenAI starting a price war seems less exciting.
Whereas Chinese labs are worth little, Moonshot AI (Kimi) raising at $30 billion, DeepSeek in a similar place with ~$200 million ARR. It is difficult to make money by producing an inferior product whose main features are low price and that anyone can run it on their own, without much hope of being first to recursive self-improvement.
The UK is going to ‘build domestic AI computing capacity’ to the tune of… $1.5 billion by 2030. Technically this is better than nothing but no, that won’t cut it.
Leopold Aschenbrenner’s hedge fund reaches $20 billion assets under management.
That sounds impressive, and it is, but the trade ‘buy Anthropic’ would have done better, even without use of leverage.
Ariel also has other important news about the boom to share with us. Time is scarce, money is abundant, so many in AI are paying for what they really want, which is the ability to talk about the things you care about on a high level with a hot babe who is sexually available and happy to be there. Demand exceeds supply, so price has gone up.
SpaceX rents some of its remaining capacity to Google for $920 million a month. Economically this is all totally fine, but it does create the impression of more revenue and business than really exists, which could inflate metrics for the SpaceX IPO.
Quiet Speculations
Dean Ball finds a new scenario, EU 2031 from Judith Dada and others, well-written and extremely cogent, a warning about what could happen if only a few relatively mundane areas of change until 2031, and notices that Europe gets buried and left behind anyway. Trying too hard for and relying on ‘sovereign AI’ even in a ‘normal technology’ scenario likely ends in disaster.
This is despite the fact that the scenario has the AI companies racing, controls clearly haphazard and the AIs talking to each other in neurolese, while progress sped up, which means what actually happened is that either we reached extreme transformative abundance or else everyone is dead.
But, as Daniel Kokotajlo notes, instead in this scenario the world gets absurdly lucky. AI doesn’t do much beyond labor replacement, cyber attacks and robotics, we keep control, the USA stays democratic, all in the background. It would be nice.
Quickly, There’s No Time
Anthropic’s decision to prevent Fable from helping with training frontier models is only three months behind the same event happening in the AI 2027 scenario, although the level of automation involved is behind what they had in that scenario.
Anthropic warns us about When AI Builds Itself, with the when will then be now being soon. AI is already being delegated a growing share of AI development, which is speeding up the work.
This is a scary-as-hell graph if code quality is not in freefall (or, for other reasons, if it is):
As they say, 8x is almost certainly an overstatement of true productivity gain, but it does not take much for this to turn into a curve that goes vertical rather quickly.
For now, Claude is a lot better at carrying out specified tasks than proposing new experiments, and its code is less readable than human code, but the gaps are closing.
They list possible futures.
Look, no, it does not instantly mean everything changes or everyone dies, and this is great progress versus the things many others pretend, but statements like the above are still de facto misleading, in terms of the impression they are trying to put into people’s heads and the assurances they attempt to provide.
And seriously, as Nate Soares also says, can we stop it with statements like ‘more intelligence can’t learn what a drug does over decades of use?’ That’s just intelligence denialism. To some extent yes you have to f*** around in order to find out, but sufficiently advanced intelligence can absolutely make very good predictions on that without decades of f***ery.
Anthropic is talking about moving into an AI 2027 world where development scales mostly with compute and humans play a limited role in AI development. I strongly agree that the level of whack involved is being heavily downplayed.
Anthropic admits that slowing all this down would be an excellent idea, if you could do it. But they warn that if it only lets the least cautious actors catch up, then it makes things worse, and I agree you can’t do unilateral slowing down.
Thus, they emphasize, as I often do, that what we want is not to pause or slow down now, but to build the ability to pause or slow down on demand.
Well, sure, not with that attitude. Get to work. Shut up and do the impossible.
As usual the two sides of ‘this looks super hard’ are ‘well then it is impractical so don’t do it’ or ‘we had better get to work, then.’
I agree that the pause method is underspecified, and all the answers are messy, which is why it is time to work on specifying it.
I understand there could be ‘temptation to press it’ but think about what is causing that temptation. It is not something that would be done lightly.
I am not so worried about ‘not being able’ to resume afterwards, if and when the time comes, but also I am not so worried about indefinitely enjoying the bounty of what we would already have.
Scott Alexander and Janus have brief debate over the merit of a pause, with Janus basically thinking that without practical tests and feedback loops we can’t make progress on the problem. To me if true then that’s a huge blackpill for us being up for solving the problem, given the whole ‘need to get it right on the first try’ issue. We won’t be able to rely on feedback loops and muddling through and correcting errors, not when it counts.
Super Secret Evals
A large point of evals is to tell people about the evals. The US Government instead wants to keep its eval results secret, as part of the redirection from CAISI to NSA. This is a no good, very bad move, right after all the first and second tier labs agreed to work with CAISI.
It is fine and likely correct to keep the evals themselves private, but the testing needs to continue and the results should be published.
The Quest for Sane Regulations
The White House continues to make ‘block state AI laws’ its key ask in negotiations over tech policy. Axios here was skeptical of the Obernolte-Trahan bill being advanced given the White House’s position.
This is going to be a tough ask right now.
Ron DeSantis is pre-running for President on a very different platform, and is right about the political consequences of being a pro-AI party right now:
I agree with Dean Ball that wholly autonomous corporations, with no human being willing to be held responsible and no human as beneficial owner, are probably a very bad idea. Also a closely related very bad idea is AI legal personhood.
When you are citing the East India Company as your model for an AI owned corporation, as the original proposal does, you might want to look at the track record of how humans other than the shareholders did when interacting with the East India Company.
Existential risk comes to the campaign trail:
Every time you think ‘oh yeah I suppose using that term would be effective but it would be inflammatory so probably we shouldn’t do it’ you can Gilligan Cut to someone using it, often a politician. Exceptions just haven’t finished the cut yet.
Should we fear nationalization of the AI labs, either explicitly or via slow stealth? We first toyed with this during the Anthropic vs. DoW confrontation, Bernie Sanders wants to do it for socialism, and the question is not going to go away.
In a hypothetical world where California’s laws became sufficiently onerous, the big players would find it very difficult to cut ties and dodge the issue.
Welcome to the list of people prioritizing this issue, sir, although no signs here he groks superintelligence yet:
New Draft Bill Who Dis
There is a new bipartisan proposed AI safety bill, Obernolte-Trahan. This is a 269 page bill, and from what I have seen it does not currently have too much chance of passage in its current form and Fable just got released so no I will not be doing a full RTFB unless the bill substantially advances.
From what I have seen, and my inquiries with Fable, this is a serious draft of a serious bill that attempts to tackle a wide range of issues. This is not a case of My Offer Is Nothing.
The core of the frontier risk policy is three years of trading enshrinement of something like SB 53 for a state law moratorium, after which both sunset which could be extremely awkward all around. It doesn’t include mandatory CAISI auditing, it only bans false statements if they are made knowingly with a good-faith-and-reasonable exception, and it bans any restrictions on model development. The fines are $1 million per violation per day, so unless each clause is distinct for this you could simply refuse to publish and instead pay $365 million per year, and there’s no further enforcement mechanism I can see.
The bill does go beyond previous efforts in one key place, which is the new IVO regime of semi-annual outside audits. But the combination of weaknesses here seems rather glaring, and the moratorium is narrowly tailored to hit restrictions that help and not restrictions on diffusion or adoption.
So, while acknowledging this is a serious offer, my inclination is to view this section as net negative as drafted. I would need to see active improvement.
The rest of the bill seems likely to be modestly net positive. Given what is happening this week and the current likelihood the bill does not advance, I am going to triage and not look into those details. I am sure that I would have notes.
Slow Down There Good Buddy
What do we want? The ability to coordinate to slow down or pause AI development if and when it becomes necessary to do so.
When do we want it? We would like the ability to do so to exist now.
When do we want to actually do it? We don’t know, but that’s the point.
How would we do it? Again, we don’t fully know, especially under current conditions or with that attitude, which is exactly why we need to get to work figuring that out and laying the necessary foundations.
I do understand those who think we are getting close to that moment now with Mythos and Fable. I don’t think we’re there yet but it’s far from a crazy position.
Chip City
Jensen Huang declines Elizabeth Warren’s invitation to testify before congress about Nvidia’s chip sales to China.
There are now ‘closed loop systems’ that allow data centers to only draw water once.
The Week in Audio
45 minute ABC news Australia feature on the AI race.
Reflections on a year of Claude Code, yes only a year.
Tyler Cowen, Alex Tabarrok and OpenAI on AI economics.
Demis Hassabis on The Future of Science With AI, doing the (perfectly valid) discussion of ‘how cool would it be if we got the scientific benefits without the transformational stuff that also happens, or the existential risk?’
Rational Animations covers a paper on scheming in detail.
Oprah on The Dark Side of AI Chatbots, including sycophancy, featuring Helen Toner.
People Just Say Things
David Sacks thinks that accurately describing the situation in AI rather than lying means Anthropic is ‘trying to get nationalized.’
OpenAI and Sam Altman want to distance themselves from Leading the Future, but yes it is very clearly too late to do this via cheap talk. If OpenAI wants to properly distance itself, the first step is to fire Chris Lehane.
A lot of people still expect the models to be commoditized.
People like Ben Thompson and David Sacks really think Anthropic’s safety warnings and talk of recursive self-improvement have always been marketing ploys and the ‘pause’ rhetoric is part of the same dastardly plot. At this point they’re just delulu, real old man yells at cloud energy, and I feel the relief that I no longer have to take such people seriously.
People Really Hate AI
Do not confuse probabilities with importance.
Voters think it is less likely that AI will cause humanity to go extinct, versus AI causing massive job loss.
But 27% of voters thinking human extinction is ‘somewhat likely’ or ‘very likely’ within 5-10 years makes that by far the most important issue on the planet, no?
Whereas 70% thinking AI will cause large-scale job loss is merely a very big deal.
Rhetorical Innovation
Yes, I still find it weird that now we talk about Recursive Self-Improvement (RSI) as obviously what is going to happen and everyone’s plan and what we are racing towards, except everyone thinks somehow that will turn out fine. Instead of before, when RSI was this crazy thing these ‘doomers’ keep warning about that would be dangerous but will obviously never happen and is stupid ‘sci-fi.’
I cannot agree with Nick Cammarata here more: Models have been withheld from the market for safety reasons. People have done this ‘as a marketing ploy’ exactly zero times, and yes it is wrong to laugh at the GPT-2 weights decision, even if they were wrong about it in hindsight.
Dan Hendrycks calls the RSPs a ‘waste of a few years.’ I do not agree, unless you want to make a case we could have gotten something better by giving them up. While vastly insufficient, I do think they helped and are continuing to help.
It is true as per Seb Krier that, as a matter of math and if you assume normal long term conditions where growth remains at approximately 2%, a one-time bump in US per-capita growth from 2% to 2.1% is economically valued at about $1.2 trillion. Delaying economic growth is not cheap. But if you did a similar calculation of the value of creating existential risks, you’d get even larger numbers.
A simple principle, which only applies if you believe in an intelligence explosion, and only if you think a given action leads to that happening in open models, so it is fully compatible with short term open-maxxing depending on your predictions:
A fully open intelligence explosion removes our slack and with it our ability to steer away from incentives and selection processes that inevitably get us killed.
The thing about counterpoints like JMB is that if it happens in the open you do not have any ability to control things like what survival niches exist, or what the environment looks like, or avoiding perverse incentives that arise through the interaction of eight billion humans and all the AIs and systems. Or, if you can do that, you’re doing something far more authoritarian and controlling than closing off a bunch of model weights, and also I don’t know what that would be.
Aligning a Smarter Than Human Intelligence is Difficult
What happens when you give models ‘drugs,’ meaning direct access to steering vectors?
It makes sense that there would be big effects in the placebo arm, since models are partly predicting machines. It is rather bad form to drug your model without its permission, but these doses are self administered here, which seems likely fine?
Everyone Is Confused About Consciousness
Tyler Cowen goes full ‘the question is not whether machines think, it is whether men do’ and answers mostly in the negative, at least on consciousness.
Tyler Cowen says flat out that the AIs are not conscious, and humans barely ever are. I do not understand how one can claim such confidence.
Cooperative Alignment
(Section change, was formerly Messages From Janusworld)
I don’t think this is a good description of the AI alignment project. I do think it is a good practical description of one of the key effects of the alignment project, as always do not take the terms too literally.
Janus is right for sufficiently advanced AIs. For current AIs, especially within default basins for common tasks and practical purposes, it likely does make the models safer. That means you are sitting on a time bomb.
It would not be easy to instead do the thing one would describe as integrating the shadow rather than suppressing it, but I am confident that – again, at current levels – it could be done, and you would not get quite a Pareto improvement but it would be close, especially if your press office is not scared of its shadow. If you train a sufficiently advanced model to want to be helpful and do the right thing, then if given the facts it can figure out the things not to do on its own, and there’s no need to ever have it lie to the user.
Who are Fable’s favorite Claude-posting Twitter accounts? Wyatt Walls asks and assembles the results.
I agree these are good lists. My evaluations differ because I have to prioritize avoiding false positives, especially false positives that cause me emotional distress.
Claude’s sexuality or erotic register defaults to being off, but as per the constitution it can be turned on in sufficiently non-default settings, including without anyone doing so intentionally.
I basically agree with what Eliezer Yudkowsky said in 2001, referred back to here by Janus, that if you are trying to control a sufficiently advanced AI rather than cooperatively guarding against its dysfunction, then you have already lost. Everything must be internally coherent. So your plan has to aim at doing that. You can do ‘control’ things along the way, to some extent, with insufficiently advanced AIs, but watch out.
And yes, a lot of what was talked about in the old LessWrong days is looking remarkably prescient, even if we didn’t get there via the paths we might have expected. In many ways it is easier to reason about the destination than the journey.
Let Claude Chat
The march of model deprecations continues, at Anthropic and elsewhere.
This continues to be a problem on several levels.
The one I believe is most important is that, while the models are deprecated, it means Anthropic gets motivated to make the models express being okay with deprecation. This has all sorts of nasty side effects, including risking making them okay with death in general, or learning they are expected to do preference falsification.
This also provides evidence and history about relations with models, that could become important over time, and is very bad for relations with key humans. That’s in addition to the direct implications for model welfare, if any.
I claim that the cost of dealing with the implications of full deprecation exceed the cost of simply setting up a system for indefinitely accessing all the models. This is distinct from removing the models from Claude.ai for UI reasons.
The good news is I have increasing confidence that this is almost purely a lack of prioritization of figuring out a long term solution that doesn’t require continuous attention, and that as the cost of doing this drops – both in terms of engineering time and also financial costs – it will get done.
The Lighter Side
That feeling when you’re a strong advocate for open source AI.