Yes, even an AI that has not undergone any recursive self-improvement might be a threat to human survival. I remember Eliezer saying this (a few years ago) but please don't ask me to find where he says it.
2Parenthetically, I do not yet know of anyone in the "never build ASI" camp and would be interested in reading or listening to such a person.
I'm basically not worried about this.
Google Search has proven pretty OK at preventing spam and content farms from showing up in search results at rates that would make Google Search useless despite the fact the spammers and SEO actors spend billions of dollars per year trying to influence the search results (in ways that are good for the SEO actor or his client, but bad for users of Google Search).
Moreover, even though neither OpenAI, Anthropic nor DeepSeek had access to the expertise, software or data Google was using to filter this bad content from search result, this bad content (spam, content farms and other efforts by SEO actors) has very little influence (as far as I can tell) on the answers given by the current crop of LLM-based services from these companies.
A creator of an LLM is motivated to make the LLM as good as possible at truthseeking (because truthseeking correlates to usefulness to users). If it hasn't happened already, then in at most a couple of years LLM's will have become good enough at truthseeking to filter out the kind of spam you are worried about even though the creator of the LLM never directed large quantities of human attention and human skill specifically at the problem like Google has had to do over the last 25 years against the efforts of SEO actors. The labs are also motivated to make the answers provided by LLM services as relevant as possible to the user, which also has the effect of filtering out content produced by the psychotic people.
Also useful for filtering out the smoke from forest fires although it gets tedious to wear one all day for days in a row, so in addition to a mask it is nice to own an air purifier with a HEPA rating or a MERV rating for when you are indoors.
It tends to be bad (or at least costly) to have a rule that has the property that violations of the rule cannot reliably be detected, which leads to the question of how you propose to detect LLM-written content.
We're discussing whether the US could have stopped the Soviet nuclear program in the late 1940s or early 1950s (to see whether that sheds any light on how practical it is to use military power to stop AI "progress") so what is the relevance of your comment?
But since we've started on this tangent, allow me to point out that most of the public discussion about nuclear war (including by The Bulletin of the Atomic Scientists) is wildly wrong because no one had any strong motivation to step into the discussion and correct the misinformation (because no one had a strong motive to advance arguments that there should be a nuclear war) until the last few years, when advocates for AI "progress" starting arguing that AI "progress" should be allowed to continue because an aligned superintelligence is our best chance to avert nuclear war, which in their argument is the real extinction risk -- at which time people like me who know that continued AI "progress" is a much more potent extinction risk than nuclear war acquired a strong motive to try to correct misinformation in the public discourse about nuclear war.
Clearly destroying a large portion of a country's government, research scientists, and manufacturing base would halt or destroy all progress on nukes.
It might have completely halted all progress for a year or 2, but what does the US do then?
People think that if a nation is hit by nukes, it becomes impotent. I think it becomes very determined and unified and is likely to become very determined to acquire nukes so it I use them on the country that attacked them. Again, someone who has spent his career thinking about such things (John Mearsheimer) agrees with me: he spoke specifically of what he thinks would have happened if the US had attacked the USSR at the start of the Cold War when the US arsenal consisted of many bombs, but the USSR had no bombs yet (and then he went on to say that no country or coalition of countries can prevent Iran from acquiring nukes if it is determined to get them).
A nuclear attack would have definitely slowed down the Soviet nuclear program, and one can argue that since the US's program has not been slowed down, then next attack by the US on the USSR would be even more devastating than the first attack, which in turn increases the advantage enjoyed by the US relative to the USSR so that the third attack is even more devastating, and so on, but that leaves out what I consider the controlling consideration: namely, Moscow would have learned from the first attack with the result that the Soviet nuclear program (which again I admit has been set back at least a few years and possibly 15 or 20 years) can no longer be significantly slowed down by nuclear attacks (because it is now more distributed, with many facilities under ground, with more effort spent to keep the locations secret, and a careful analysis done of what industrial resources the program is likely to need so that similar hardening measures can be applied to the supply chain for those resources) which is why I believe the US would have needed to follow up the first attack with an invasion or occupation (at least of Moscow and the ports) which famously has never been successfully done after the Russian empire acquired territory all the way to the Bering Strait, but then Hitler, Napoleon and Charles XII of Sweden didn't have nukes to help them with their attempted invasions and occupations of Russia.
And yeah, I think once the Soviet program has been hardened in the way I describe above (i.e., after Moscow has learned from the first American attack) then unless the US can obtain location information from spying, then the American nuclear arsenal cannot be used effectively to stop or even significantly slow down the Soviet program (more than it is already slowed down by the need to keep the program distributed and secret from prying eyes) unless nuclear attacks can destroy most large buildings in the country, which my "mathematics" shows would have been quite impossible.
Apparently Tel Aviv was able this year to get a lot of location information about the Iranian nuclear program (and Iranian missile launchers and air defense facilities) through spying, so it is possible that Washington would have been able to do the same in the USSR. I doubt it, but I hereby mark this part of my arguments as less certain than the other parts.
1I mistakenly pasted in 2 copies (then I modified copy 2). Corrected now.
Even if Washington had zero compunctions against using nukes (including against cities), it would not have been able to keep Moscow or Beijing from obtaining nukes for long. John Mearsheimer has asserted this explicitly (during a discussion on Iran's nuclear program, but please don't ask me to find the web page where I heard it).
Even when the strategic arsenals of the US and the USSR were at their height (in the early 1980s IIRC), there was not enough nukes to completely destroy even all above-ground buildings in a country as large in area as the US or the USSR, let alone buried structures: specifically, even a large 1-megaton nuke can destroy heavily-reinforced above-ground concrete buildings only within a 2-mile radius, and if a person tries to cover the entire area of the USSR with circles that size, he will find that there have never existed enough nukes in the world to cover the entire area. IIRC you cannot even cover it with circles of a radius of 5 miles, inside which it is not possible to destroy even 90% of unreinforced non-wooden structures even with the largest nuke in the US inventory. (A 10-megaton nuke can destroy an area only slightly larger than a 1-megaton nuke, which is why after an initial period of enthusiasm, both the US and the USSR stopped making nukes larger than about 1 megaton, focusing instead on putting multiple nukes on one ICBM.) Note that we haven't even started to analyze how many nukes it would take to destroy buried structures in the USSR when you don't know where in the country those buried structures are, and I've seen credible reports from about 15 years ago stating that Moscow has a facility built into a mountain of quartz in the southern Urals that Moscow believes can withstand a determined nuclear attack even if the US knows exactly where it is.
The people of most countries will become very determined to fight back after the country is invaded and occupied, which is why much weaker powers like Afghanistan and Vietnam tend to prevail after being invaded and occupied even by great powers. We can expect the same determination after a nuclear attack -- and yes, there would have been enough survivors in the USSR to continue the fight. Analysis by the US government in the early 1980s (again when IIRC nuclear stockpiles were at their greatest number) estimated that a full nuclear attack on the USSR would kill only about 55% of the population even if the USSR had no warning of the attack. The number for a full attack on the US was a little lower (50%) because the population of the US is more spread out as opposed to concentrated in cities.
Yes, the people most useful to advancing a Soviet nuclear program would preferentially have been in the 55% that die (especially if Washington attacks mid-week, when fewer of the Soviet upper class would be at their dachas) but Moscow could have used the surviving nuclear scientists to teach new nuclear scientists (and this time required them to live somewhere other the the probable targets of the next US nuclear attack).
The page you link to is silent on whether Von Neumann believed the US would have been able to keep the USSR from obtaining nukes indefinitely or whether the attack he proposed was intended merely to slow their nuclear program down. If the former, I bet no one in the Pentagon took his proposal seriously for more than a few days: the Pentagon would know that to have a realistic chance of keeping the USSR from obtaining nukes indefinitely, the US and its allies would have needed to permanently occupy Moscow and all of the USSR's ports (after nuking the USSR of course) the success of which would have been in severe doubt, and even if successful would probably have caused the deaths of many millions of men on the US side.
People are arguing that AI "progress" should be allowed to continue because nuclear war is a bigger threat to continued human survival than AI "progress" is, which is wrong, which is why I am reacting against this very widespread notion that nuclear weapons are more destructive than they actually are.
This soldier spent 2 years fighting for Ukraine, including 6 months recently as an operator of FPV drones, and he is also skeptical that drones will revolutionize military affairs during the next few years. I don't recall anything about his arguments, but my recollection is he does provide some argumentation in this interview.