Persuasion plays games with thinking of its targets, some other modes of explanation offer food for thought that respects autonomy and doesn't attempt to defeat anyone. Perhaps you should be exactly as skeptical of any form of communication, but in some cases you genuinely aren't under attack, which is distinct from when you actually are.

And so it's also worth making sure you are not yourself attacking everyone around you by seeing all communication as indistinguishable from persuasion, all boundaries of autonomy defined exactly by failure of your intellect to pierce them.

Maybe there are modes of engagement that should be avoided, and many ideas/worldviews themselves are not worth engaging with (though neglectedness in your own personal understanding is a reason to seek them out). But as long as you have allocated time to something, even largely as a result of external circumstances, doing a superficial and half-hearted job of it is a waste. It certainly shouldn't be the intent from the outset, as in the quote I was replying to.

If AGI is human-equivalent for the purposes of developing a civilization, a collective of AGIs is at least as capable as humanity, plus it has AI advantages, so it's much more capable than a single AGI instance, or any single human. This leads to ASI being often used synonymously with AGI lately (via individual vs. collective conflation). Such use of "ASI" might free up "AGI" for something closer to its original meaning, which didn't carry the implication of human-equivalence. But this setup leaves the qualitatively-more-capable-than-humanity bucket without a label, that's important for gesturing at AI danger.

I think the other extreme for meaning of "ASI", being qualitatively much stronger than humanity, can be made more specific by having "ASI" refer to the level of capabilities that follows software-only singularity (under the assumption that it does advance capabilities a lot). This way, it's neither literal technological maturity of hitting the limits of physical law, nor merely a collective of jagged-human-level AGI instances wielding their AI advantages. Maybe "RSI" is a more stable label for this, as in Superintelligence Strategy framing where "intelligence recursion" is the central destabilization bogeyman, rather than any given level of capabilities on its own.

you sympathize with them while not taking their worldview seriously

There is no reason at all to take any idea/worldview less than seriously. For the duration of engagement, be it 30 seconds as a topic comes up, or 30 minutes of a conversation, you can study anything in earnest. Better understanding, especially of the framing (which concerns are salient, how literal words translate into the issues they implicitly gesture at), doesn't imply your beliefs or attitudes must shift as well.

if you aren’t willing to change your beliefs, why should they

This is not just an inadvisable or invalid principle, but with the epistemic sense of "belief" it's essentially impossible to act this way. Beliefs explain and reflect reality, anything else is not a belief, so if you are changing your beliefs for any reason at all that is not about explaining and reflecting reality, they cease being beliefs in the epistemic sense, and become mental phenomena of some other nature.

I will repeatedly interject something along the lines of “you keep talking about this as a problem that it falls upon me to solve, while in reality we are all sitting in the same boat with respect to existential AI risk, so that you in fact have as much reason as me to try to work towards a solution where we are not all murdered by superintelligent AIs a few years down the road”.

This demands that others agree with you, for reasons that shouldn't compel them to agree with you (in this sentence, rhetoric alone). They don't agree, that's the current situation. Appealing to "in reality we are all sitting in the same boat" and "you in fact have as much reason as me to try to work towards a solution" should inform them that you are ignoring their point of view on what facts hold in reality, which breaks the conversation.

It would be productive to take claims like this as premises and discuss the consequences (to distinguish x-risk-in-the-mind from x-risk-in-reality). But taking disbelieved premises seriously and running with them (for non-technical topics) is not a widespread skill you can expect to often encounter in the wild, without perhaps cultivating it in your acquaintances.

Nesov notes that making use of bigger models (i.e. 4T active parameters) is heavily bottlenecked on the HBM on inference chips, as is doing RL on bigger models. He expects it won't be possible to do the next huge pretraining jump (to ~30T active) until ~2029.

HBM per chip doesn't matter, it's HBM per scale-up world that does. A scale-up world is a collection of chips with sufficiently good networking between them that can be used to setup inference for large models with good utilization of the chips. For H100/H200/B200, a scale-up world is 8 chips (1 server; there are typically 4 servers per rack), for GB200/GB300 NVL72, a scale-up world is 72 chips (1 rack, 140 kW), and for Rubin Ultra NVL576, a scale-up world is 144 chips (also 1 rack, but 600 kW).

use of bigger models (i.e. 4T active parameters) is heavily bottlenecked on the HBM

Models don't need to fit into a single scale-up world (using a few should be fine), also KV cache wants at least as much memory as the model. So you are only in trouble once the model is much larger than a scale-up world, in which case you'll need so many scale-up worlds that you'll be effectively using the scale-out network for scaling up, which will likely degrade performance and make inference more expensive (compared to the magical hypothetical with larger scale-up worlds, which aren't necessarily available, so this might still be the way to go). And this is about total params, not active params. Though active params indirectly determine the size of KV cache per user.

He expects it won't be possible to do the next huge pretraining jump (to ~30T active) until ~2029.

Nvidia's GPUs probably won't be able to efficiently inference models with 30T total params (rather than active) until about 2029 (maybe late 2028), when enough of Rubin Ultra NVL576 is built. But gigawatts of Ironwood TPUs are being built in 2026, including for Anthropic, and these TPUs will be able to serve inference for such models (for large user bases) in late 2026 to early 2027.

The general principle is that sufficiently smart people by default win most competitions among 100 randos (given sufficient training, when that's at all relevant) that they care to enter.

To be "not-insane", you don't need rationality in this narrow sense, in most circumstances. You don't need to seek out better methods for getting things right, you just need some good-enough methods. A bit of epistemic luck could easily get you there, no need for rationality.

So the issue of behaving/thinking in an "insane" way is not centrally about lack of rationality, rationality or irrationality are not particularly relevant to the issue. Rationality would help, but there are many more things that would also help, some of them much more practical for any given object level issue. And once it's resolved, it's not at all necessary that the attitude of aspiring to rationality was attained, that any further seeking out of better methods/processes will be taking place.

Rationality is not correctness, not truth or effectiveness, it's more narrow, disposition towards better methods/processes that help with attaining truth or effectiveness. Keeping intended meaning narrow when manipulating a vague concept helps with developing it further; inflation of meaning to cover ever more possibilities makes a word somewhat useless, and accessing the concept becomes less convenient.

If Omega tells you what you'll do, you can still do whatever. If you do something different, this by construction refutes the existence of the current situation where Omega made a correct prediction and communicated it correctly (your decision can determine whether the current situation is actual or counterfactual). You are in no way constrained by existence of a prediction, or by having observed what this prediction is. Instead, it's Omega that is constrained by what your behavior is, it must obey your actions in its predictions about them. See also Transparent Newcomb's Problem.

This is clearer when you think of yourself (or of an agent) as an abstract computation rather than a physical thing, a process formally specified by a program rather than a physical computer running it. You can't change what an abstract computation does by damaging physical computers, so in any confrontation between unbounded authority and an abstract computation, the abstract computation is having the final word. You can only convince an abstract computation to behave in some way according to its own nature and algorithm, and external constructions aren't going to be universally compelling to abstract algorithms (such as Omega being omniscient, or the thought experiment being set up in a certain way).