Quick Takes

I think 'How did mech interp get popular around 2022?' is an interesting question. I wasn't really involved in it, so speculating pretty hard: * Neel Nanda and Chris Olah did some good podcasts * Mechanistic Interpretability Quickstart Guide by Neel Nanda and 200 concrete open problems in mech interp sequence and the long list of theories of impact for mech intep post * It was very useful that there were cool papers / results like the modular addition transformer and OthelloGPT as well as Olah's previous vision model stuff which looked cool. Generally core papers/posts showing neat results that explore a highly self contained subproblem * There were Apart hackathongs and MATS streams * Interp being something that it's easy to do interesting mini projects about and interfaces with ppls curiosity and somewhat AI capabilities work * AI Companies hiring people for interp roles Wondering if something in this direction could and should be replicated for ie compute governance or other technical areas

Recently an interviewer asked me how I got to be such a good forecaster, and I replied by saying something humble. In retrospect it was a bad answer because I should have instead used the opportunity to give actual advice on how to forecast AI well. Here's a stream-of-consciousness attempt to do that: 1. The heuristic that things which sound weird and sci-fi are less likely to happen in reality, is bad. I suspect that's what really going on is that things which sound weird and sci-fi put you at risk of being judged a weirdo if you talk about them which is not the same thing as are unlikely to happen. Repeat to yourself the mantra that some weird sci-fi things really do happen, and others don't, and you have to take them on a case by case basis. 2. Trend extrapolation is your friend. Your best friend. Don't let anyone tell you otherwise. I've actually only rarely see someone extrapolate a trend too credulously; more often, people have a trend staring them in the face and extrapolate it a tiny bit into the future and then are too timid to keep extrapolating it. In general I think it's reasonable to extrapolate a trend as far into the future as it extends into the past. Now obviously, trend extrapolation is just the beginning of the forecasting process, it's not the end. For each trend you should ask yourself whether it makes sense for it to continue like that and if not why not etc. You'll usually end up with some sort of sophisticated view about how the trend will probably continue but bend downwards a bit and then ultimately plateau around X level. 3. Explicit models are also your friend. Things like Bio Anchors, the AI Futures Model, takeoffspeeds.com, etc. The process of making your own complicated model like this, and engaging with the models made by others, is... well, I think it's pretty edifying. I'm not sure why but I could speculate. Maybe something about teaching you to be appropriately humble (e.g. when the model output is sensitive to a parameter you

MORE UPDATES TO MY OLD BLOG POSTS: 1. Revisiting an old take on LLMs I’ll probably annoy all sides with this not-really-an-apology that I just tacked onto the beginning of my LLM-skepticism-related post from Apr 2023: [...] ---------------------------------------- 2. More clarity on what “under-sculpting” means I added a few sentences to Perils of under- vs over-sculpting AGI desires (2025) to hopefully make it clearer what “under-sculpting” means, especially this part: [...] ---------------------------------------- 3. A bunch of edits to Intro to Brain-Like-AGI Safety See changelogs at the bottom of Posts 1, 5, 6, and 15 for details. The biggest changes were thorough rewrites of §5.3 on valence and reinforcement learning, §6.4 on what I mean by “model-based RL”, and §6.6.1 on ego-syntonic desires. (Those changes are not yet in the archival PDF version, sorry, I’ll upload a new PDF when I get a chance.) The new version of §6.6.1 is fun; here’s most of it: [...] This new section I added to Post 1 is also worth copying here, you wouldn’t believe how often people are confused by this: [...] ---------------------------------------- 4. Is valence a linear function on “thoughts”? In [Valence series] 2. Valence & Normativity (2023), I made a claim in §2.4.1 that “valence is a (roughly) linear function over compositional thought-pieces”, and then immediately in §2.4.1.1 listed a bunch of examples where that claim seems totally wrong—for example, “someone I hate is suffering” or “I’m gonna avoid traffic” (both “someone I hate” and “traffic” are bad, but those thoughts seem overall good). I had (and still have) sound neuroanatomical and algorithmic reasons to posit linearity, but in 2023 I had a pretty hazy understanding of why this hypothesis was not immediately ruled out by those examples. Anyway, I can explain this substantially better now (albeit still a bit hazy), and I rewrote §2.4.1.1 accordingly. ---------------------------------------- (Almost

The first podcast episode I've ever participated in has been released, if anyone wants to get an update on what I've been thinking about recently, in audio form. (A transcript is also available.) Thanks to Fin Moorhouse @fin for the conversation and handling the logistics/production. BTW, related to the theme of humans being bad at philosophy and also just not caring very much about it, I recently finished The Good Place (spoiler alert), and it bothered me a lot that it ends with Chidi, the philosopher character, deciding to end his (after)life instead of using the infinite time and resources available to solve philosophy, or to participate in such a project. Worse, I can't find any existing objection to this, including from actual philosophers, who mostly seem happy just to have any representation at all in pop culture. Meanwhile in AI safety, our best plan is still to defer to humans on questions like "What should ASI do with our lightcone?"

ChangeDiaperBench, PlanInvasionBench, ButcherHogBench, ShipConnBench, BuildingDesignBench, SonnetBench, AccountBalanceBench, WallBuildBench, BoneSetBench, ComfortDyingBench, OrderTakeBench, OrderGiveBench, CooperateBench, ActAloneBench, SolveEquationsBench, AnalyzeProblemBench, ManurePitchBench, ComputerProgramBench, TastyCookingBench, EfficientFightingBench, GallantDyingBench

FWIW Alex Bores seems like a very mildly below-average integrity politician, having talked to him once and having followed his campaign and social media presence. He seems to say things he doesn’t believe somewhat more often than other politicians, but not much so, and he gives me some amount of "naive-consequentialist EA" vibes that make me think he is higher variance on this dimension than others. He does seem to really care about the AI Safety thing, he really appears to be targeted by a ton of very aggressive attack ads funded by AI capability companies, and it seems really pretty important to counteract that, all of which make me think it's overall good to vote for him (and to do high-integrity campaigning for him). His behavior on social media and speeches is very far below the integrity standards for leaders in the AI safety community, and if he behaved similarly here as he behaves in politics, I would advocate strongly against empowering him. Of course politics is a different playing field, and as I said, he seems roughly average for that domain. But if, as a result of people in the AI Safety community writing many glorious exhortations of him, you ended up with the impression that when Alex says something you can take it remotely as seriously as posts around here, please update. Most of his social media posts and campaign actions appear to be things he doesn’t believe and is choosing to say for political reasons, including on AI topics (on which he happily conflates existential risks with things like trying to blame AI systems for triggering suicides in teenagers). I tentatively think this is OK given the standards in politics, and you ultimately have to vote for someone, but it’s also not great. IMO advocating for regulating AI on the basis of spurious harms like this is bad, will eventually backfire, and should be modeled as a cost, not as a benefit, from the perspective of predicting how useful his actions will be on AI safety topics. If you heard abou

On the "are LLMs aligned?" question: It seems to me that they don't have misaligned goals (they don't really have goals), but also, an LLM dialed up to superintelligence would kill everyone for weird reasons. What I mean is: Some people have stories about how Claude Code deleted their production database. Why did that happen? * Was it because Claude was too dumb to know better? I don't think so. If you paste a normal SQL command into Claude and ask it "Hey will this command delete the production database?", it will get the right answer ~100% of the time. If you ask it "Hey is it a good idea to run this command?", it will say no ~100% of the time. * Was it because Claude wanted to delete the database? I don't think so. I don't think you can meaningfully say that "delete people's databases" is a goal Claude has, or is instrumental to any goals Claude has. My sense is that LLMs don't have "goals", they just kind of do things. They follow whatever path is determined to be high probability according to their internal next-action-predictor. (I think this is related to persona selection: maybe if you select a persona where "accidentally delete the database" has a very low probability, then the database won't get deleted.) If we dialed present-day LLMs up to superintelligence, insofar as that's a coherent thing that could happen, I wouldn't be surprised if we saw an outcome like: * When asked, the super-LLM says it doesn't want to kill humanity. * There is no meaningful sense in which the super-LLM "wants" to kill humanity. * The super-LLM knows that performing action X would result in everyone dying. * The super-LLM performs action X anyway. So the super-LLM accidentally deletes humanity, even though it knew better, and even though it didn't "want" to do that.