What I find lacking is any depth to the retrospection. Hanania's willingness to update his position clears a rather low bar. To go further one has to search for systematic causes for the error. For example, being wrong with the markets seems like a good opportunity to examine the hidden incentives that cause the market to get it wrong, not shrug and say "who could have known?"
I feel the question misstates the natsec framing by jumping to the later stages of AGI and ASI. This is important because it leads to a misunderstanding of the rhetoric that convinces normal non-futurists, who aren't spending their days thinking about superintelligence.
The American natsec framing is about an effort to preserve the status quo in which the US is the hegemon. It is a conservative appeal with global reach, which works because Pax Americana has been relatively peaceful and prosperous. Anything that threatens American dominance, including giving...
Richard, reading this piece with consideration of other pieces you've written/delivered about Effective Altruism, such as your Lessons from my time in Effective Altruism and your recent provocative talk at EA Global Boston lead me to wonder what it is (if anything) that leads you to self-identify as an Effective Altruist? There may not be any explicit EA shibboleth, but it seems to me to nevertheless entail a set of particular methods, and once you have moved beyond enough of them it may not make any sense to call oneself an Effective Altruist.
My mental mo...
A relevant, very recent opinion piece that has been syndicated around the country, explaining the universal value of faith:
https://www.latimes.com/opinion/story/2025-03-29/it-is-not-faith-that-divides-us
There's a gap in the Three Mile Island/Chernobyl/Fukushima analogy, because those disasters were all in the peaceful uses of nuclear power. I'm not saying that they didn't also impact the nuclear arms race, only that, for completeness, the arms race dynamics have to be considered as well.
There are at least 3 levers of social influence, and I suspect that we undervalue the 3rd one for getting anything done, especially when it comes to AI safety. They are 1) government policy 2) the actions of firms 3) low-coordination behaviors of individuals influenced by norms. There is a subclass of #3 that is having its day in the sun, the behaviors of employees of the US federal government, a.k.a. the "Deep State." If their behaviors didn't matter there wouldn't be a perceived need by the Trump administration to purge them en masse (and replace with lo...
That summary doesn't sound to me to be in the neighborhood of the intended argument. I would be grateful if you pointed to passages that suggest that reading so that I can correct them (DM me if that's preferable).
Where I see a big disconnect is your conclusion that "AI will have an incentive to do X." The incentives that the essay discusses are human incentives, not those of a hypothetical artificial agent.
The subject of your post is the recurring patterns of avoidance you're observing, without mentioning the impact on those more receptive and willing to engage. Nevertheless, I figure you'd still appreciate examples of the latter:
A link to the GD website was sent to me by a relatively distance acquaintance. This is a person of not-insignificant seniority at one of the FAANGs, whose current job is now hitched to AI. They have no specific idea about my own thoughts about AI risk, so my inference is that they send it to anyone they deem sufficiently wonky. The ...
There are two kinds of beliefs, those that can be affirmed individually (true independently of what others do) and those that depend on others acting as if they believe the same thing. They are, in other words, agreements. One should be careful not to conflate the two.
What you describe as "neutrality" to me seems to be a particular way of framing institutional forbearance and similar terms of cooperation in the face of the possibility of unrestrained competition and mutual destruction. When agreements collapse, it is not because these terms were unworkable (except for in the trivial sense that, well, they weren't invulnerable to gaming and do on) but because cooperation between humans can always break down.
@AnthonyC I may be mistaken, but I took @M. Y. Zuo to be offering a reductio ad absurdum response to your comment about not being indifferent between the two ways of dying. The 'which is a worse way to die' debate doesn't respond to what I wrote. I said
With respect to the survival prospects for the average human, this [whether or not the dying occurs by AGI] seems to me to be a minor detail.
I did not say that no one should care about the difference.
But the two risks are not in competition, they are complementary. If your concern about misalignment is...
You have a later response to some clarifying comments from me, so this may be moot, but I want to call out that my emphasis is on the behavior of human agents who are empowered by automation that may fall well short of AGI. A "pivotal act" is a very germane idea, but rather than the pivotal act of the first AGI eliminating would-be AGI competitors, this act is carried out by humans taking out their human rivals.
It is pivotal because once the target population size has been achieved, competition ends, and further development of the AI technology can be halted as unnecessarily risky.
If an unaligned AI by itself can do near-world-ending damage, an identically powerful AI that is instead alignable to a specific person can do the same damage.
If you mean that as the simplified version of my claim, I don't agree that it is equivalent.
Your starting point, with a powerful AI that can do damage by itself, is wrong. My starting point is groups of people whom we would not currently consider to be sources of risk, who become very dangerous as novel weaponry, along with changes in relations of economic production, unlock the means and the motive ...
Thanks, title changed.
I agree, and I attempted to emphasize the winner-take-all aspect of AI in my original post.
The intended emphasis isn't on which of the two outcomes is preferable, or how to comparatively allocate resources to prevent them. It's on the fact that there is no difference between alignment and misalignment with respect to the survival expectations of the average person.
The title was intended as an ironic allusion to a slogan from the National Rifle Association in the U.S., to dismiss calls for tighter restrictions on gun ownership. I expected this allusion to be easily recognizable, but see now that it was probably a mistake.
An argument for danger of human-directed misuse doesn't work as an argument against dangers of AI-directed agentic activity.
I agree. But I was not trying to argue against dangers of AI-directed agentic activity. The thesis is not that "alignment risk" is overblown, nor is the comparison of the risks the point, it's that those risks accumulate such that the technology is guaranteed to be lethal for the average person. This is significant because the risk of misalignment is typically thought to be accepted because of rewards that will be broadly shared...
I'm surprised by the lack of follow-up to this post and the accompanying thread, which took place in the immediate aftermath of the October 7th massacre. A lot has happened since then -- new data against which the original thinking could be evaluated. Also time has provided opportunity to self-educate about the conflict, which a few people admitted to not knowing a lot about. Given the human misery that has only worsened since the OP started asking questions, I would think that a follow-up would be a worthy exercise. @Annapurna ?
Ever since first hearing the music of the Disney movie "Encanto" I've been sneering at the lyrics "stars don't shine they burn/ and constellations shift" because, no, of course constellations don't shift, without really stopping to think about it. Caught in my epistemic arrogance again!
Oops, Samuel beat me to the punch by 2 minutes.
You've already noted that it doesn't really matter, but I thought I'd help fill in the blanks.
The current global regime of sovereign nation-states that we take for granted is the product of the 20th century. It's not like an existing sovereign nation-state belonging to the Palestinians was carved up by external powers and arbitrarily handed to Jews. Rather, the disintegration of empires created opportunities for local nationalist movements to arise, creating new countries based on varying and competing unifying or dividing factors such as language, tribal ...
This response is not unreasonable, but the description of "WW2-style solution" seems ignorant of the fact that Israel did occupy Gaza for decades, and had something very similar to a "puppet government" there, in the form of the Fatah party in control of the Palestinian Authority in the West Bank. Israel unilaterally withdrew in 2005, and Hamas violently took over in 2007.
The rest of it operates under the hypothesis that Hamas is opposed to the objective interests of the Palestinians of Gaza. This ends up being tautological if objective self-interest is de...
A helpful counterpoint.
Why would the human beings have to be suicidal, if they can also have the AI provide them with a vaccine?
Thank you. If I understand your explanation correctly, you are saying that there are alignment solutions that are rooted in more general avoidance of harm to currently living humans. If these turn out to be the only feasible solutions to the not-killing-all-humans problem, then they will produce not-killing-most-humans as a side-effect. Nuke analogy: if we cannot build/test a bomb without igniting the whole atmosphere, we'll pass on bombs altogether and stick to peaceful nuclear energy generation.
It seems clear that such limiting approaches would be avoide...
Hi all, writing here to introduce myself and to test the waters for a longer post.
I am an experienced software developer by trade. I have an undergraduate degree in mathematics and a graduate degree in a field of applied moral and political philosophy. I am a Rawlsian by temperament but an Olsonian by observation of how the world seems to actually work. My knowledge of real-world AI is in the neighborhood of the layman's. I have "Learning Tensorflow.js" on my nightstand, which I promised my spouse not to start into until after the garage has been tidied.
No...
The way you stated this makes it seem like your conclusion for the reason why the Democrats lost (and by extension, what they need to do to avoid losing in the future) is obviously correct. But the Median Voter Theorem you invoked is a conditional statement, and I don't think it's at all obvious that its conditions held for the 2024 US presidential election.