Review
Yes. Intuition is valuable when used in concert with rational analysis, and perhaps that's under-appreciated particularly in x-risk.
And: We should all be doing this for x-risk, in all directions. Just short of wasting time. If you've got nagging doubts about whether you're spending your time properly, you should investigate them with your rational mind.
I work on x-risk full time. I have my model of what the risks are, and where the effort should go. My model has lots of uncertainty. I have spent serious time understanding Bayesian reasoning and statistics and other reasoning and statistics , logic, science, and even philosophy in case that helps (it does).
And I'd still guess that I'm not just wrong, but importantly wrong.
The situation surrounding x-risk and particularly AGI is highly complex. It's so complex that I don't think anyone with a merely human mind can claim to have finished their work on figuring out where their effort should go (and therefore where their emotional motivation should lie).
So what I think we can say is that strategic thinking is important (that is, figuring out what's more likely to happen in the future, and therefore where your efforts should go in the present). This is an expansion of the point made here: I think you should do as the post suggests, and take seriously your intuitions surrounding x-risk.
Where I guess I disagree is the implication that some rational analysis can settle those nagging doubts. It can't. The situation is too complex, and we're all trying to think with monkey brains (with their nontrivial biases and severe limited resources).
But this type of thinking can help settle your doubts. Combining intuition with analysis gets us closer to the truth.
But there's clearly such a thing as paying too much attention to nagging doubts. Weighing strategic analysis against getting object-level work done is a judgment call. We mustn't work on strategy to the exclusion of doing actual work (much less simply fret or worry, that is, pay attention to nagging doubts without making real progress in resolving them).
I'd guess that even most people working on alignment x-risk are spending too little time on strategic analysis. But those people are self-selected for getting things done. There are other people who are excessively paralyzed with strategic analysis, who are paying too much attention to their nagging doubts (which probably pull in multiple directions).
One unique situation here is that most professions have already had most of the useful strategic analysis done. Sometimes that's done by your predecessors; if you're a plumber or electrician, people have tried a bunch of business models and techniques. In faster-developing fields, a hierarchical structure often designates who's meant to do strategic analysis (management) and who's supposed to sit down and get shit done.
Alignment and other x-risk fields have neither of those sources of strategic analyisis. We're all mostly doing our own work, based on others' public writings. This all takes time.
It's time well spent, until it isn't. Intuition and analysis are both needed, and when enough is enough is always a judgment call that itself requires intuition and analysis, once again emplyed in artful combination.
The LessWrong Review runs every year to select the posts that have most stood the test of time. This post is not yet eligible for review, but will be at the end of 2025. The top fifty or so posts are featured prominently on the site throughout the year.
Hopefully, the review is better than karma at judging enduring value. If we have accurate prediction markets on the review results, maybe we can have better incentives on LessWrong today. Will this post make the top fifty?
I was chatting recently to someone who had difficulty knowing how to orient to x-risk work (despite being a respected professional working in that field). They expressed that they didn't find it motivating at a gut level in the same way they did with poverty or animal stuff; and relatedly that they felt some disconnect between the arguments that they intellectually believed, and what they felt to be important.
I think that existential security (or something like it) should be one of the top priorities of our time. But I usually feel good about people in this situation paying attention to their gut scepticism or nagging doubts about x-risk (or about parts of particular narratives about x-risk, or whatever). I’d encourage them to spend time trying to name their fears, and see what they think then. And I’d encourage them to talk about these things with other people, or write about the complexities of their thinking.
Partly this is because I don't expect people who are using intellectual arguments to override their gut to do a good job of consistently tracking what the most important things to do are on a micro-scale. So it would be good to get the different parts of them to sync more.
And partly because it seems like it would be a public good to explore and write about these things. Either their gut is onto something with parts of its scepticism, in which case it would be great to have that articulated; or their gut is wrong, but if other people have similar gut reactions then playing out that internal dialogue in public could be pretty helpful.
It's a bit funny to make this point about x-risk in particular because of course the above all applies to whatever topic. But I think people normally grasp it intuitively, and somehow that's less universal around x-risk. I guess maybe this is because people don't have any first-hand experience with x-risk, so their introductions to it are all via explicit arguments … and it's true that it's a domain where we should be unusually unwilling to trust our gut takes without hearing the arguments, but it seems to me like people are unusually likely to forget that they can know anything which has bearing on the questions without already being explicit (and also that perhaps the social environment, in encouraging people to take explicit arguments seriously, can accidentally overstep and end up discouraging people from taking anything else seriously). These dynamics seem especially strong in the case of AI risk — which I regard as the most serious source of x-risk, but also the one where I most wish people spent more time exploring their nagging doubts.