Watch AI materials-science & bioscience abilities closely

Yair Halberstadt

Watch AI materials-science & bioscience abilities closely — LessWrong

16 Watch AI materials-science & bioscience abilities closely

by Yair Halberstadt

14th Sep 2026

1 min read

16

One of the most frequently suggested ways AI could wipe out humanity is via an engineered virus.

I have no doubt that a sufficiently intelligent AI given sufficient time could socially-or-otherwise engineer his way to have a whole team of people working on developing super-viruses to wipe out humanity, and could also kill us in any one of a dozen other ways.

However much more important for very short term risks is whether a slightly more intelligent LLM, with spiky abilities, could do such a thing.

The argument against is that designing complex viruses requires iteration. It's insufficient just to trick a single lab into making a virus for you once, you need to have a lab where you can repeatedly iterate till you get it right. That's a far higher bar to clear, and requires a correspondingly more advanced AI, giving us more time to get alignment right.

Our earliest evidence for this will be AIs abilities in materials and biological sciences. Achievements here are in some ways similar to the Navier Stokes counterexample, in that they just require finding a single example that fits a bunch of complex criteria to achieve specific desired results. However, unlike NS, testing them requires significant interaction with the physical world and complex equipment.

If frontier LLMs are able to one-shot room temperature superconductors, or cures for cancer, we should be extremely worried about what they do with a virus. If they cannot^[1], that suggests that we can focus less on "AI emails a DNA synthesis company the plans for a supervirus" as a threat model, and focus instead on other threat models, and ensuring they never get long term access to a lab^[2].

^{^}
Either because they're simply not capable of it, or it requires significant iteration.
^{^}
For example - make it illegal to have an autonomously operated wet lab.

Frontpage

16

Watch AI materials-science & bioscience abilities closely

5Philipp Risius

2Yair Halberstadt

0Alexander Gietelink Oldenziel

16Philipp Risius

6Thomas Kwa

6Alexander Gietelink Oldenziel

10Philipp Risius

6RogerDearnaley

New Comment

8 comments, sorted by

top scoring

Click to highlight new comments since: Today at 10:38 AM

[-]Philipp Risius2d50

I am a materials scientist by training and while I haven’t spent much time specifically on superconductivity, my impression is that the existing materials such as hydrides under extreme pressures are already pushing what is physically possible. While a discovery of such a material would undoubtedly be massive, my current best estimate is that room-temperature superconductors at ambient pressure probably (70%) don’t exist in our universe. In any case, this seems like a significantly harder problem to me than designing viruses; ab initio approaches struggle to accurately predict high-temperature superconductivity, so AI would have to first make significant theoretical progress.

Cancer cures, on the other hand, I don’t see any reason for not being feasible, but I’m out of my depth there.

[-]Yair Halberstadt2d20

Take room temperature superconductors as an example, not the target? What other problems in materials science are millennium prize level?

[-]Alexander Gietelink Oldenziel2d00

Seems overtly confident to say room-tempreature superconductors don't exist.

We see continued progress on the benchmarks over the past decades and there isn't a physical law that says they don't exist.

[I'm not a material scientist by training]

[-]Philipp Risius2d162

Lots of nuances that make me assign 70% probability (as opposed to, say, 99% if a physical law prevented it):

Conventional superconductivity is right out; the mechanism is well-enough understood to claim that's most likely impossible (here I actually would assign ~99% probability);
Unconventional superconductivity is not that well understood, but generally speaking you need a very specific electronic structure. The cuprates, for example, have been tuned extensively and only get roughly half-way to room temperature;
High-pressure hydrides might well reach superconductivity at or above room temperature, but require immense pressures which can only be reached under extreme conditions such as diamond anvil cells.

More importantly, theoretically modeling these ab initio is difficult, because density functional theory (tool of choice for electronic structure calculations) on its own can't do it. Beyond that, you quickly run into problems. A guy in my lab spent an entire PhD developing a method for accurately predicting the ordinary electrical resistance (Ohm's law) of a few simple materials from first principles.

[-]Thomas Kwa2d60

Quantum chemistry simulation is a hill climbable task (since we have more accurate, slower rungs of DFT like double hybrid that can be tested against less accurate, faster rungs, plus methods outside DFT altogether). So it is reasonable that AI will be strongly superhuman at it next year.

[-]Alexander Gietelink Oldenziel2d60

Sir I believe you may not be familiar with my friend Claude 7 the Superintelligence

[-]Philipp Risius2d100

If these don't exist in our universe, then it doesn't matter how intelligent your friend is. If they do exist, my claim is that finding them via ab initio simulation is significantly harder than designing quite complex viruses. Therefore "AI can design room-temperature ambient-pressure superconductors" does not provide a good warning signal for "AI can design complex viruses".

For instance, I can guarantee you that your friend the superintelligent AI will not be able to calculate the full many-body wave function of some system of, say, 1000 electrons.

[-]RogerDearnaley2d61

There is a physical law that says the hotter a system gets, the harder it is to make electrons pair up so they can superconduct. Thus there is some maximum temperature above which there are no superconducting materials. We just don't know what it is.

Moderation Log