In the section "For those who think that open-source AGI code and weights are the solution"
If we had the DNA sequence of an extremely dangerous virus, would it be best to share it publicly or not? If the answer is obvious to you in this case, think twice about the case for AGI algorithms and parameters.
The National Institute of Health's answer is "yes". Here's variola major (smallpox) for example. So those arguing that it's a bad idea to share ML algorithms and artifacts should either make the case that the NIH is wrong to share the smallpox genome or make the case that sharing some subset of ML algorithms and artifacts is more dangerous than sharing the smallpox genome.
In fairness, some people have in fact made decent cracks at the argument that sharing some types of ML-related information is more dangerous than sharing the smallpox genome. Still, I think the people arguing that the spread of knowledge is the thing we want to target, rather than the spread of materials, could do a better job of making that argument. But the common-sense "you wouldn't share the genome of a dangerous virus" argument doesn't work because we would, in fact, share the genome of a dangerous virus (and I personally think that it's actively good that we share the genomes of dangerous viruses, because it allows for stuff like this).
1
1Personally, my take is that this is a dangerous mistake on the part of NIH and the worldwide academic community generally, that so much dangerous information has been released in regards to virology and bioweapons. Unfortunately, I'm clearly currently in the minority among scientists who think this. I wish I had a good idea of how to change people's minds on this issue, and the connected one of publishing AI capabilities implementation details.
I initially thought there must be some simple reason that publishing the DNA sequence is not a dangerous thing to do, like "ok, but given that you would need a world class lab and maybe even some techniques which haven't even been invented yet to get it to work, it's not a dangerous thing to publish".
According to this article from 2002, synthesising smallpox would be tricky, but within the reach of a terrorist organisation. Other viruses may be easier.
“Scientifically, the results are not surprising or astounding in any way,” says virologist Vincent Racaniello of Columbia University. “The point here, of course, is that the DNA can be synthesized from the [genetic] sequence, and this could be done by any third-rate terrorist.”
Apparently, large organisations like the NIH are foolhardy enough to publish dangerous data like this. I wonder if there's some other justification, like "the data was already public, in such a way that it could not be removed"
11
He starts by emphasizing
[...]
And goes on to do a pretty great job addressing "those who think