In fact, the "plan" seems to be the exact reverse or so of my preference ordering on what it should be.
Is this due to it basically being a Pareto frontier of political capital needed vs probability of causing doom, or some other reason?
This is well-written, enjoyable to read, and not too long, but I wish the author called it something more intuitive like "Bootstrap Problems". Outside (and even maybe inside) the tiny Dwarf Fortress community no one will know what an Anvil Shortage is, and it's not really a Sazen because people can understand the concept without having read this post. Overall I give it +1.
1I'm giving this +1 review point despite not having originally been excited about this in 2024. Last year, I and many others were in a frame where alignment plausibly needed a brilliant idea. But since then, I've realized that execution and iteration on ideas we already have is highly valuable. Just look at how much has been done with probes and steering!
Ideas like this didn't match my mental picture of the "solution to alignment", and I still don't think it's in my top 5 directions, but with how fast AI safety has been growing, we can assign 10 researchers to each of 20 "neglected approach"es like this, so it deserves +1 point.
The post has an empirical result that's sufficient to concretize the idea and show it has some level of validity, which is necessary. Adam Jones has a critique. However, the only paper on this so far didn't make it to a main conference and only has 3 cites, so the impact isn't large (yet).
The thesis has been basically right in the last 18 months, and still holds. I think the only way one could have done better than this investing would be taking on concentrated positions on AI stocks. Now, the case for options might be even stronger given the possibility of being in an AI bubble, as you're protected to the downside and options are still fairly cheap (VIX is 17 as I write this).
With recent events like Nvidia's political influence and the AI super PAC, it's also looking more likely that we're heading to a capitalistic future where post-singularity wealth matters for influence. I take this seriously enough that I've been curtailing consumption now to hopefully buy a small galaxy later, or whatever it ends up buying.
This seems to hold up a year later, and I've referenced it several times, including citing it in Measuring AI Ability to Complete Long Tasks. This report's note on power availability being limiting also preceded the 2025 boom in AI-relevant energy stocks. Overall it deserves +1 point.
Every time I think of doing research in a field that's too crowded, this post puts a faint image in my head of an obnoxious guy banging a drum at the front of a marching band. This is a real issue we need to keep in mind, both in AI safety research and elsewhere, and the number of LW posts that I remember at all two years later is pretty small, so this obviously deserves at least +1 review point.
The section on status made me pay more attention to my desire for status itself, but that's probably just me.
I didn't believe the theory of change at the time and still don't. The post doesn't really make a full case for it, and I doubt it really convinced anyone to work on this for the right reasons.
- No BOTEC
- Models of social dynamics are handwavy
- In alignment and other safety work, we have force multipliers like neglectedness and influencing government. What's the multiplier here? Or is there no intervention that has multiplier effects in the long-term multipolar risk space?
- Why wouldn't the effect just get swamped by the dozens of medical AI startups that are raising billions of dollars as we speak?
It may do a good job of giving the author's perspective, but given all these gaps it's not very memorable today in explaining the risks OP cares about-- even if we do end up worrying about them in 5-10 years.
Too serious for an april fool's day post and just not very funny compared to something like "On the Impossibility of Supersized Machines" or "Open Asteroid Impact". Maybe I'm biased because I now work at METR and most people around me believe in RSPs, but I didn't get much value from either the humor or commentary then either.
I think this post is great, it's a super uncommon perspective, brave and hits all the top questions. It could be more detailed but since people rarely talk about detransitioning it was interesting just to read roughly how it went for OP. I'd say its timeless value is probably more like 160 karma than 122.
I was also there, and my take is there was actually fairly little specific, technical discussion about the economics and politics of what happens post-AGI. This is mostly due to it not being anyone's job to think about these questions, and only somewhat because they're inherently hard questions. Not really sure what I would change.