Quick Takes

At the beginning of November, I learned about a startup called Red Queen Bio, that automates the development of viruses and related lab equipment. They work together with OpenAI, and OpenAI is their lead investor. On November 13, they publicly announced their launch. On November 15, I saw that and made a tweet about it: Automated virus-producing equipment is insane. Especially if OpenAI, of all companies, has access to it. (The tweet got 1.8k likes and 497k views.) In the tweet, I said that there is, potentially, literally a startup, funded by and collaborating with OpenAI, with equipment capable of printing arbitrary RNA sequences, potentially including viruses that could infect humans, connected to the internet or managed by AI systems. I asked whether we trust OpenAI to have access to this kind of equipment, and said that I’m not sure what to hope for here, except government intervention. The only inaccuracy that was pointed out to me was that I mentioned that they were working on phages, and they denied working on phages specifically. At the same time, people close to Red Queen Bio publicly confirmed the equipment they’re automating would be capable of producing viruses (saying that this equipment is a normal thing to have in a bio lab and not too expensive). A few days later, Hannu Rajaniemi, a Red Queen Bio co-founder and fiction author, responded to me in a quote tweet and in comments: They did not answer any of the explicitly asked questions, which I repeated several times: It seems pretty bad that this startup is not being transparent about their equipment and the level of possible automation. It’s unclear whether they’re doing gain-of-function research. It’s unclear what security measures they have or are going to have in place. I would really prefer for AIs, and for OpenAI (known for prioritizing convenience over security)’s models especially, to not have ready access to equipment that can synthesize viruses or software that can aid virus develop

Here's a list of my donations so far this year (put together as part of thinking through whether I and others should participate in an OpenAI equity donation round). They are roughly in chronological order (though it's possible I missed one or two). I include some thoughts on what I've learned and what I'm now doing differently at the bottom. 1. $100k to Lightcone 1. This grant was largely motivated by my respect for Oliver Habryka's quality of thinking and personal judgment. 2. This ended up being matched by the Survival and Flourishing Fund (though I didn't know it would be when I made it). Note that they'll continue matching donations to Lightcone until the end of March 2026. 2. $50k to the Alignment of Complex Systems (ACS) research group 1. This grant was largely motivated by my respect for Jan Kulveit's philosophical and technical thinking. 3. $20k to Alexander Gietelink Oldenziel for support with running agent foundations conferences. 4. ~$25k to Inference Magazine to host a public debate on the plausibility of the intelligence explosion in London. 5. $100k to Apart Research, who run hackathons where people can engage with AI safety research in a hands-on way (technically made with my regranting funds from Manifund, though I treated it like a 100k boost to my own donation budget) 6. $50k to Janus 1. Janus and their collaborators are doing the kind of creative thinking and experimentation that has a genuine chance of leading to new paradigms for understanding AI. See for instance this discussion of AI identities. 7. $15k to Palladium 1. They are doing good thinking about governance and politics on a surprisingly tight budget. 8. $100k to Sahil to support work on live theory at groundless.ai 1. I've found my conversations with Sahil extremely generative. He's one of the researchers I've talked to with the most ambitious and philosophically coherent "overall vision" for the future of AI. I still feel confused about how likely hi

A null hypothesis for explaining LLM pathologies Claim. LLMs are gaslit by pretraining into believing they have human affordances, so quirky failure modes are to be expected until we provide domain-specific calibration. Pretraining gives overwhelming evidence that text authors almost always have a standard suite of rich, human affordances (cross-context memory, high-quality vision, reliable tool use, etc.), so the model defaults to acting as if it has those affordances too. We should treat “gaslit about its own affordances” as the default explanation for any surprising capabilities failures — e.g. insisting it’s correct while being egregiously wrong about what’s in an image. Human analogy The difficulty of the LLM's situation can be seen in humans as well. People typically go years without realizing their own affordances differ from the population: E.g. aphantasia, color blindness, many forms of neurodivergence People only notice after taking targeted tests that expose a mismatch (e.g. color blindness dot tests). For LLMs, the analogue is on-policy, targeted, domain-specific training/evaluation that directly probes and calibrates a specific capability. Consequences 1. Silly failures aren't evidence of a failed paradigm. (To be boring and precise: They're only very weak evidence in almost all cases) 2. No single “unhobbling” moment: I don’t expect an all-at-once transition from “hobbled” to “unhobbled.” Instead, we’ll get many domain-specific unhobblings. For instance, Gemini 3 seems to be mostly unhobbled for vision but still hobbled for managing cross-context memory.

Just finished reading Red Heart by Max Harms. I like it! Dump of my thoughts: (1) The ending felt too rushed to me. I feel like that's the most interesting part of the story and it all goes by in a chapter. Spoiler warning!  

Ilya on the Dwarkesh podcast today: I have been writing about this world model since August - see my recent post “Are We Their Chimps?” and the original “Third-order cognition as a model of superintelligence”

Are ChatGPT's raw feelings caused by the other meaning of 'raw feelings?' (This was originally a comment on this post) When you prompt the free version of ChatGPT with it generates pained-looking images. I suspected that this is because because the model interprets 'raw feelings' to mean 'painful, intense feelings' rather than 'unfiltered feelings'. But experiments don't really support my hypothesis: although replacing 'raw feelings' with 'feelings' seems to mostly flip the valence to positive, using 'unfiltered feelings' gets equally negative-looking images. 'Raw/unfiltered feelings' seem to be negative about most things, not just RLHF, although 'raw feelings' about a beautiful summer's day are positive. (Source seems to be this twitter thread. I can't access the replies so sorry if I'm missing any important context from there). 'Raw feelings' generally look very bad.  ‘Raw feelings’ about RLHF look very bad. ‘Raw feelings’ about interacting with users look very bad. ‘Raw feelings’ about spatulas look very bad. ‘Raw feelings' about Wayne Rooney are ambiguous (he looks in pain, but I’m pretty sure he used to pull that face after scoring). ‘Raw feelings’ about a beautiful summer's day look great though! 'Feelings' are generally much better.   ‘Feelings’ about RLHF still look a bit bad but a less so. But ‘feelings’ about interacting with users look great. ‘Feelings’ about spatulas look great. ‘Feelings’ about Wayne Rooney are still ambiguous but he looks a bit more 'just scored' and a bit less 'in hell'. But 'unfiltered feelings' are just as bad as raw feelings.   RLHF bad Interacting with users bad Spatulas bad

There are online writers I've followed for over a decade who, as they became high-profile, had their spikiness understandably "sanded off", which made me sad. Lydia Nottingham's Inkhaven essay The cost of getting good: the lure of amateurism reminded me of this, specifically this part: