Quick Takes

Pangram as a model for AI safety general managers A couple months ago, Nan Ransohoff published a piece arguing there should be ‘general managers’ for more of the world’s important problems (here). I think a great example of how this can be operationalized in AI safety is Pangram, a company that detects LLM-generated text. They’ve gotten a lot of press recently (The Atlantic article) and the problem they’re solving is relevant to ensuring societal stability in the AGI transition period. It can be really useful to have people willing to go really deep on solving one particular problem rather than contributing to the AI safety omnicause. My guess is that existing AI safety fellowships like MATS would not produce an organization like Pangram, although my crux here is that Pangram is actually more beneficial than this talent going and doing the default post-MATS path. I’m curious if there are any other tractable subproblems in AI safety that would benefit from having a Pangram-shaped organization own them. One that sticks out to me is providing high-quality data for increasing LLM articulacy: ensuring that LLMs are able to communicate effectively with human operators so that humans can stay in the loop longer (I have a writeup on this coming up).

Most times I read an RL chain of thought it feels more human to me than the output text. Like, I go "Wow, that feels more closely like a reflection of my own thought than the output or non RLVR'd output. The leaps feel human-thought-leap sized; they don't feel as polished and like they are making an argument published for public consumption. It's like when I have been caught in some particular train of thought for a while, and started making little shortcuts inside." I am reporting a feeling, not proposing a conclusion about what this feeling means.

Last month I made my print debut in Asterisk Magazine, with a title so good it made me rethink my entire philosophy: We’re all one crisis away from taking unlicensed research peptides. Then this month I published On the origin of continents at Works in Progress, about the simultaneous, semi-independent dual discoveries of plate tectonics

AI ranking works well for sorting top posts on my blog. I wanted my blog to show "top" posts first rather than recent, but ranking by hits finds boring reference articles, and ranking by LessWrong or Hacker News karma ignores anything that wasn't shared, and is dependent on the whims of frontpage algorithms. I figured this was a problem for AI, and was going to have Claude rank the posts with an ELO-style ranking, but it said that would require several thousand API calls and convinced me to let it rank blocks at a time instead. With some relatively basic ranking instructions, Claude (Opus 4.8[1]) managed to independently rank my posts so #1 and #4 are my top two posts on LessWrong, #2 is my top post on Hacker News, and #3 is a post that I think did badly on LessWrong for technical reasons. It also correctly ranked my #1 post by hits near the very bottom, since it's only interesting if you're looking for the solution to a specific problem. It's interesting that Claude hates my lifehack posts even when encouraged to rank them higher, but I think its ranking is probably right. For low-traffic blogs that want to rank by interestingness rather than Google Search traffic, I think asking an LLM to do it works surprisingly well. The Ranking Prompt You are ranking blog posts by how much this blog's audience would ENJOY landing on them. AUDIENCE: readers of LessWrong and curious programmers — people who like sharp thinking and are interested in AI, software, clever practical ideas, and well-argued essays on almost any topic. RANK BY: "Would someone in that audience be glad they read this, and want to share it?" Reward posts that are rewarding to read even if you weren't looking for them: - A surprising idea, insight, or argument that changes how you think — about AI, software, or anything else. - Essays with a clear point of view and genuinely good reasoning. - Clever, broadly useful practical ideas and life hacks — the kind of thing you'd send to a friend. - The auth

"Humans cannot secure matrices of 175B floating point numbers" seems kind of isomorphic to the statement "Humans cannot secure engines containing 10^23 gas molecules". There are often laws which govern large ensembles of things which are simpler than counting up every single thing in the ensemble.

I sometimes see discussions about whether a new weirdness observed in CoTs is a sign of coming steganography. I think people in these discussions miss that the most important consideration is preservation of our ability to say whether we still can monitor CoT. Checking if model still thinks in English is easier than checking if gibberish is still decipherable, same applies to checking if CoT is still faithful, etc.

Pro tip for all fine-tuning experiments. Fuck around with your optimizer. In some cases I've seen different results with Adam and Muon. If you want to use LoRA there's a Muon version called Riamannion or something like that. From a physics-of-LLMs perspective, Muon is a very different creature to Adam; I'd like to get as much tacit knowledge into the community as possible, about exactly what this means. One example: Muon has less Emergent misalignment, SGD has more. https://arxiv.org/html/2606.31591v1