Quick Takes

I notice that there is this idea among AI safety people that conditional on AIs not being misaligned, building superintelligence is a public good and is a pretty exciting prospect. This is not how many average people in the US feel. I was describing to an older family member that Anthropic focuses on code because they are trying to build a claude that can build a smarter claude which can build a smarter claude which can … The reaction to this prospect was disgust, not because he intuitively felt AIs would likely be misaligned. It was more like on a gut level this amount of “playing god” felt totally antisocial and demonic and in general not respectful of an intuitive taboo against divine transgression (see Jurassic Park, Frankenstein, the recent popularization of Oppenheimer, Tower of Babel, etc). Seems important to consider that people can feel this way when communicating with the public or policy makers.

Is there a good report on "what's the likely economic outcome if we get to keep using AIs, say, 1 year from now, but ban training new AI, and things otherwise play out approximately normally? (If not, I think there probably should be) Maybe this might be an ordinary economist kinda report, since they broadly don't believe in superintelligence and are probably not imagining anything crazier than AI a year from now anyway?

Thoughts on: When is it even possible (or likely) for single neurons to encode single concepts, based on model architecture As part of my mech-interp research, I'm thinking a lot about the question "how would I do this if I was a neural network?". Specifically, given some toy model architecture, how can I set the weights so that it actives some task. Todays conclusion is that single ReLU neurons are almost useless. To do identify any non-linear pattern (e.g. XOR), you need at least two ReLUs. We should not expect any feature to be represented by a single ReLU neuron, since anything that can be picked out by a single ReLU, was already linearly accessible in the first place, so why use the ReLU at all. Therefore, anything meaningful in a ReLU-MLP will use multiple neurons. GELUs are a bit more expressive, i.e. not completely monotonic, but I don't think they are differens enough from ReLUs is enough to matter. I have not though about SwiGLUs, and all the other ones MLP variants. I might get back to that, or share your thoughts in the comments. Residual stream neurons will not be interpretable, for the simple reason that the residual stream has no privileged basis. Convolutional neurons seems unusually well suited for single concepts, I think, which is why we see interpretable neuons in AlexNet. Although caveat that this is a post-diction, that I haven't though super hard about.

People keep proposing that I should do a public debate with @Neel Nanda on "is mech interp valuable". But I think me and Neel don't really disagree on very much on the topic anymore. I largely agree with his position on interpretability articulated in "A Pragmatic Vision for Interpretability" and elsewhere, and both of us are pretty sympathetic to each other's research strategy and prioritization. I think Neel and I genuinely disagreed from 2023 to mid 2025, but by the end of that time period we had pretty similar positions, mostly due to Neel coming around to my position on some broad questions about interp and me feeling happier with the quality of interp research (thanks substantially to Neel).

Often when I give advice I do it as an anecdote. This especially applies to: * Giving advice to anyone who isn't a close friend (i.e. someone I know well). * Giving unsolicited advice. Instead of suggesting what someone else should do, I tell a story of a similar problem I had and solved, and then let the other person pull out what ever part of the lesson applies to them. I find this works better for a number of reasons. * I don't know their context enough to know if what worked for me will work for them. E.g. I don't know if the direction I needed adjustment is the same as the direction they need adjustment. (https://slatestarcodex.com/2014/03/24/should-you-reverse-any-advice-you-hear/) * It avoids the sazen problem. (https://www.lesswrong.com/s/CkphjEuLfGnsYEzan) This style works great in text forums (anything from FB to LW) where there is low feedback and more space. I.e. you're more likely to want to give unsolicited advice, since asking first is a long delay. And you're less likely to bother anyone by going on a long monolog, because they can read it in their own time, or not. But it also works well in 1-on-1 or very small group conversations. Although if it's a long story, you should ask first if it's welcome. I think I developed this style of advice because I hate when people who don't know me super well try to tell me what to do, but I also have the normal human urge to offer advice. So I tried to find a way to share my wisdom that does not rout though the thing that I don't like people doing to me. And then I just got positive feedback from doing it this way, so now I do it more. Looking back at this short form, I notice that I never wrote "you should", or even "you" (in previous paragraphs). I'm not telling you what to do, because I don't know what constraints you have. But I would like if more advice directed at me, were in this form.

There's a lot of economics people around here. Does anyone know of someone who'd be great for writing a report on public health implications of genetically heritable diseases? There's some interest in having a report on plausible / likely economic impacts of government policy around PGT-P (parents selecting between their embryos using polygenic scores for disease risks, intelligence, longevity, etc.). E.g., what would be the likely implications for public health, healthcare costs, government healthcare costs, or general economic productivity, of policies such as: * Legalizing PGT-P * Subsidizing PGT-P for couples already doing IVF * Subsidizing IVF for couples to enable them to use PGT-P The person should be: 1. Credible / authoritative as an academic source. E.g. ideally a professor of economics, population health, epidemiology, public health, or similar; or a grad student or postdoc with such a PI who could supervise & sign off on the work. 2. Technically able to do this evaluation work well, and write it in a way that is clear and useful for a government official.

Reading some of the critiques of Plan A online, I'm increasingly convinced that many people start with a premise that we live in a "normal" world, where only "normal" things happen. With that premise, anything that predicts that wild and crazy things could happen must be wrong. I think that premise is clearly empirically false, but it's hard to see because all the previous wild and crazy reality-shifting things that have already happened are now accepted as "normal".