There are at least 3 levers of social influence, and I suspect that we undervalue the 3rd one for getting anything done, especially when it comes to AI safety. They are 1) government policy 2) the actions of firms 3) low-coordination behaviors of individuals influenced by norms. There is a subclass of #3 that is having its day in the sun, the behaviors of employees of the US federal government, a.k.a. the "Deep State." If their behaviors didn't matter there wouldn't be a perceived need by the Trump administration to purge them en masse (and replace with loyalists, or not). But if government employees' low-coordination individual choices matter, then so can the choices of members of the general population.
States and firms are modern instruments, whereas (at least if you trust some of the accounts from Dawn of Everything) for about 100,000 years the more organic form of coordination was all humans had, and it worked surprisingly well (for example, people could and did travel long distances and count on receiving shelter from strangers).
As already stated, we rely on norm-observance in government employees performing their duties, and in everyone else to more or less comply with laws in functioning welfare states, but traditional norm enforcement is weakened by liberal laissez fair values. But if one believes (a la Suleyman's The Coming Wave) that AI undermines the liberal welfare state, which is likely to be captured by powerful AI firms, then one shouldn't discount norm-enforced resistance emerging to fill the void for an increasingly disenfranchised population.
It is therefore a mistake to treat the race dynamics of AI development between firms and nation-states as an inevitable force pointing in only one direction. Given a critical mass of people recognizing that AI is bad for them, low-coordination resistance is possible, despite the absence of democratic policy-making.
On the flip side, this also suggests a tipping point where AI economic disruption becomes extremely violent, between powerful government-capturing firms wishing to maintain control and general populations resisting. Thus we should consider the existence of a hidden race between would-be powerful government-capturing firms and a would-be resistant population.
That summary doesn't sound to me to be in the neighborhood of the intended argument. I would be grateful if you pointed to passages that suggest that reading so that I can correct them (DM me if that's preferable).
Where I see a big disconnect is your conclusion that "AI will have an incentive to do X." The incentives that the essay discusses are human incentives, not those of a hypothetical artificial agent.
The subject of your post is the recurring patterns of avoidance you're observing, without mentioning the impact on those more receptive and willing to engage. Nevertheless, I figure you'd still appreciate examples of the latter:
A link to the GD website was sent to me by a relatively distance acquaintance. This is a person of not-insignificant seniority at one of the FAANGs, whose current job is now hitched to AI. They have no specific idea about my own thoughts about AI risk, so my inference is that they send it to anyone they deem sufficiently wonky. The tone of the message in which they sent the link was "holy smokes, we are totally screwed," suggesting strong buy-in to the paper's argument and that it had an emotional impact.
There are two kinds of beliefs, those that can be affirmed individually (true independently of what others do) and those that depend on others acting as if they believe the same thing. They are, in other words, agreements. One should be careful not to conflate the two.
What you describe as "neutrality" to me seems to be a particular way of framing institutional forbearance and similar terms of cooperation in the face of the possibility of unrestrained competition and mutual destruction. When agreements collapse, it is not because these terms were unworkable (except for in the trivial sense that, well, they weren't invulnerable to gaming and do on) but because cooperation between humans can always break down.
@AnthonyC I may be mistaken, but I took @M. Y. Zuo to be offering a reductio ad absurdum response to your comment about not being indifferent between the two ways of dying. The 'which is a worse way to die' debate doesn't respond to what I wrote. I said
With respect to the survival prospects for the average human, this [whether or not the dying occurs by AGI] seems to me to be a minor detail.
I did not say that no one should care about the difference.
But the two risks are not in competition, they are complementary. If your concern about misalignment is based on caring about the continuation of the human species, and you don't actually care how many humans other humans would kill in a successful alignment(-as-defined-here) scenario, a credible humans-kill-most-humans risk is still really helpful to your cause, because you can ally yourself with the many rational humans who don't want to be killed either way to prevent both outcomes by killing AI in its cradle.
You have a later response to some clarifying comments from me, so this may be moot, but I want to call out that my emphasis is on the behavior of human agents who are empowered by automation that may fall well short of AGI. A "pivotal act" is a very germane idea, but rather than the pivotal act of the first AGI eliminating would-be AGI competitors, this act is carried out by humans taking out their human rivals.
It is pivotal because once the target population size has been achieved, competition ends, and further development of the AI technology can be halted as unnecessarily risky.
If an unaligned AI by itself can do near-world-ending damage, an identically powerful AI that is instead alignable to a specific person can do the same damage.
If you mean that as the simplified version of my claim, I don't agree that it is equivalent.
Your starting point, with a powerful AI that can do damage by itself, is wrong. My starting point is groups of people whom we would not currently consider to be sources of risk, who become very dangerous as novel weaponry, along with changes in relations of economic production, unlock the means and the motive to kill very large numbers of people.
And (as I've tried to clarify in my other responses) the comparison of this scenario to misaligned AI cases is not the point, it's the threat from both sides of the alignment question.
Thanks, title changed.
I agree, and I attempted to emphasize the winner-take-all aspect of AI in my original post.
The intended emphasis isn't on which of the two outcomes is preferable, or how to comparatively allocate resources to prevent them. It's on the fact that there is no difference between alignment and misalignment with respect to the survival expectations of the average person.
There's a gap in the Three Mile Island/Chernobyl/Fukushima analogy, because those disasters were all in the peaceful uses of nuclear power. I'm not saying that they didn't also impact the nuclear arms race, only that, for completeness, the arms race dynamics have to be considered as well.