A bugbear of mine is being told to be surprised or astonished. Especially when I wouldn't be, but even when I am. It's not up to you, author! This post is riddled with that particular dark art, which I perceived to be on the rise in LW.

The real danger is less to “humanity” and more to specific humans. Because when multicellularity arose, some cells made it, some didn’t, and those that didn’t probably suffered greatly.

I'd quibble, where this depends strongly on the interpretation of 'suffered', and 'made it'!

Another important question is whether (or under what conditions) it's nice to be part of a greater whole. The kinds of primitive (?) greater wholes that humans currently get themselves into seem to vary in that regard, and one could imagine far greater variety there. Would it be nice to be a mitochondria-ish or gene-ish component of a larger being?

And one should note that, in the examples you're giving, and perhaps in your described (if not conceived) vision, these are exclusive, hierarchical aggregations. (After all, my cells are part, exclusively, of me, and my mitochondria are part, exclusively, of a given cell, etc.)

In sharp contrast, contemporary human super-entities are very overlapping, as well as being porous, in ways which I think are very important and valuable (we often consider it a malaise if an aggregate is totalising and exclusionary, perhaps one of the bad markers of a cult - that fuzzy term).

I appreciate and partly agree with this perspective, but I think you're expressing it quite a bit too strongly.

As salient counterpoints, consider that nature repeatedly (in unrelated phyla) invented ganglia and brains, and that human-invented systems sometimes benefit from distribution and parallelism while other times benefiting from centralisation and vertical scaling.

So I certainly don't think you can say that nature empirically strenuously asserts in either direction.

I do think you're on the money saying

There is little reason to suppose an silicon-based and carbon-based life cannot likewise form greater wholes.

though of course a key question is whether it's competitive (and under what conditions), and if not, what can be done about that (or what can be done to set conditions so that it is).

I think the benchmark is intended to measure performance on an even narrower proxy than this—roughly, the sort of tasks involved in ordinary, everyday software engineering.

Note that METR has also published a subsequent attempt to broaden the class of activities, and has some suggestive results that the qualitative exponentially increasing time horizon phenomenon is somewhat robust, but the growth rate varies between domains.

On climate change, I just think it's a point worth making: people are getting exercised about very minor contributions to resource consumption by current AI firms, which is a bit silly, but it is continuous with the kinds of radical and extinctive activities which might stamp out humans forever!

On persuasion, it looks like you might have gone way too hard, if you've been arguing for arbitrary puppeteering out of the gate! (Though perhaps the quote in your mom's voice is hypothetical?)

I'm also offering an example of a way in to discussion (again most workable one-to-one), pointing out the many and varied ways that humans persuade and coerce each other.

On bio, I'm disagreeing that the mom test criteria override the importance of emphasising important interlinked issues. It's not necessary to mention bio in all conversations, but you shouldn't shy away from it, and it's a very available and reasonable example to turn to of the kinds of vulnerabilities that could wipe out huge populations or even all humans.

On 'totally outclassed', I was just offering some ways that in conversation you can make that point relatable. It's generally far more workable one-to-one, since you're having a conversation. Less likely to work in writing, though maybe.

My comment was actually (perhaps unhelpfully) a series of somewhat independent comments. I'll fork under here. In sum, you could say I'm arguing that the identified heuristics for 'mom test' aren't necessarily well fit, in part by giving reasons and angles-on-reasons which are (in my experience) more effective than those implied by the discussion you give to justify the heuristics. I'm also offering a few angles which I've found to be useful conversation openers.