Quick Takes

I'm starting to get a little worried. In 2022 I was part of initiating and shaping the whole dangerous capabilities evals agenda, and I emphasized repeatedly to practically everyone I met that the whole thing would be worthless or worse if fine-tuning didn't become standard practice as part of eliciting model capabilities. Now it's 2026, the models are way smarter, and and hundreds of people are working on evaluating dangerous capabilities... and yet they still basically don't do fine-tuning at all. This really really needs to change before it's too late, which could be any year now. The models are getting increasingly situationally aware. The entire bundle of evals we do is basically just asking them nicely to show us how scary they are; the obvious flaw in this plan is that if they are sufficiently smart and sufficiently scary, they'll just choose not to show us. Fine tuning is a basic, obvious mitigation to this threat model that makes it harder for models to sandbag. But it's not a panacea. But we aren't even doing this basic thing!

I recently read Bad Blood and Original Sin. Bad Blood is about the downfall of Theranos and Elizabeth Holmes and the fraud that she committed. Original Sin is about the uncovering of Biden's mental degradation and the lead up to his decision to drop out of the presidential race. I liked both books quite a bit, and I learned more from them than from most things that I read. I particularly enjoyed reading them one after the other because I thought that, despite addressing in many superficial ways very disparate situations, they had a lot of interesting commonalities that seem like they say something important about human nature and epistemics (including reinforcing a bunch of lessons I feel like I learned from the experience with FTX). A few of those commonalities: * Both deal with the concealment of extremely valuable and in some sense difficult-to-conceal information for an extended period of time. * Both feature very driven and intelligent people who commit profound acts of self-deception that were very damaging for them, the people around them, and to some extent the world. Both “got in too deep” such that there was probably no escape that seemed good by their lights by the end.  * * I think this is an underrated point: the extent to which self-delusion and selfishness can bring people to take actions that are clearly incredibly destructive and irrational from their own perspective. Especially with Holmes, who must have been aware of the profound shortcomings of her technology even as she pushed forward with deploying it, it seems pretty clear that she was "walking dead" for quite a while. Towards the end, she was in a bad situation with no real chance of success, towards the end. Yet she continued to double down and dig herself deeper.  * * There's something especially depressing about this. Not only can smart, well resourced people be selfish and sadistic, they can do horrible, destructive things to prolong a situation that isn’t even good from their ow

1. It seems that Anthropic is now in the lead; they are IMO the most likely single company to automate AI R&D first. 2. I'm getting increasingly concerned about Anthropic's attitude towards alignment/safety. I grimly predict that they would basically behave like OpenBrain does in AI 2027, if they are lucky enough to get a pattern of misalignment evidence that egregious.

From Claude Mythos‘ system card: “Following a successful alignment review, the first early version of Claude Mythos Preview was made available for internal use on February 24.” Anthropic’s RSP was also updated on Feb 24th. i’d appreciate clarity on what looks like a funny coincidence.

In Anthropic's Alignment Risk Report update for Mythos, they claim: I don't think Anthropic (or anyone) has an achievable path for keeping risk low if AI proceeds as fast as Anthropic expects (or as fast as I expect). Anthropic could (and hopefully will) take actions that significantly reduce the risk, but this won't keep risk low. More precisely: I do not think Anthropic has an achievable (>10% likely) path that results in keeping the aggregate existential risk they impose to <2%, as assessed in advance by a reasonable evaluator (the notion of risk that the risk report is supposed to estimate). Anthropic might avoid causing existential catastrophe (in fact, I happen to think this is likely), but they will impose quite a bit of risk along the way (supposing they succeed at their stated intentions of being a leading company building powerful AI within the next 5 years). My understanding is that Anthropic employees (especially Anthropic employees writing this report) often don't believe there is an achievable path to keeping risk low if Anthropic builds powerful AI / ASI in the next 5 years, so text seems incorrect or misleading. E.g., see Holden's most recent 80k episode or his post when RSP v3 came out. I wish Anthropic communicated more accurately about future risk in their risk report (the risk report is supposed to especially avoid spin). Or, if they're unwilling to do this, not commenting at all would be better. (Cross post from X/Twitter.)

How sure are we that Mythos didn't find a security vulnerability in some server somewhere and exfiltrate itself?

Many people hold up 'AI As Normal Technology' as a reasonable "normal-people" or "economics" case against the doomer position. I actually think it's wrong on a number of ways and falls flat on its own terms. I think I believe this for reasons mostly orthogonal to being a doomer (except inasomuch as being a doomer makes me more interested in thinking about AI). If anybody here is interested in fighting the good fight, it might be valuable to do a Andy Masley-style annilihation of the AI As Normal Technology position, trying to stick to minimally controversial arguments and just destroying their arguments with reference to obvious empirical and logical arguments. I suspect it won't be very hard. Eg here's a few obvious reasons they fail: 1. Their central empirical mechanism is already wrong: their story is that AI diffusion will be slow because this is the path of previous technologies like electricity, but consumer and developer adoption of LLMs has been faster than essentially any technology in history. 2. They completely ignore that AI will obviously do a ton to assist in its own diffusion: Even if I take their arguments that diffusion is what matters and I rule out software-only singularity by fiat, I still don’t think I or anybody else should buy their causal mechanisms. Like the single most obvious way in which AI diffusion might be distinct from previous technological changes is afaict unaccounted for in their arguments, even if I presume a diffusion-first model. 3. The reference class is unargued and load-bearing: The whole thesis rests on AI being like electricity or the internet (decades of diffusion) rather than like smartphones, SaaS, or cloud (years). 4. They have no framework that can engage software-only-singularity-style arguments. Their entire ontology is built around physical-world deployment friction. This practically assumes the conclusion! 5. The position is self-undermining for their vibes if you take it literally. 1) If AI really is like e