LESSWRONG
LW

Comment Permalink
whestler32

I initially thought there must be some simple reason that publishing the DNA sequence is not a dangerous thing to do, like "ok, but given that you would need a world class lab and maybe even some techniques which haven't even been invented yet to get it to work, it's not a dangerous thing to publish".

According to this article from 2002, synthesising smallpox would be tricky, but within the reach of a terrorist organisation. Other viruses may be easier. 

“Scientifically, the results are not surprising or astounding in any way,” says virologist Vincent Racaniello of Columbia University. “The point here, of course, is that the DNA can be synthesized from the [genetic] sequence, and this could be done by any third-rate terrorist.”

Apparently, large organisations like the NIH are foolhardy enough to publish dangerous data like this. I wonder if there's some other justification, like "the data was already public, in such a way that it could not be removed"

Reply11
See in context
by Judd Rosenblatt
1 min read
3

70

This is a linkpost for https://yoshuabengio.org/2024/07/09/reasoning-through-arguments-against-taking-ai-safety-seriously/

He starts by emphasizing

The issue is so hotly debated because the stakes are major: According to some estimates, quadrillions of dollars of net present value are up for grabs, not to mention political power great enough to significantly disrupt the current world order.

[...]

The most important thing to realize, through all the noise of discussions and debates, is a very simple and indisputable fact: while we are racing towards AGI or even ASI, nobody currently knows how such an AGI or ASI could be made to behave morally, or at least behave as intended by its developers and not turn against humans.

And goes on to do a pretty great job addressing "those who think

  • AGI and ASI are impossible or are centuries in the future
  • AGI is possible but only in many decades
  • that we may reach AGI but not ASI, those who think that AGI and ASI will be kind to us
  • that corporations will only design well-behaving AIs and existing laws are sufficient
  • who think that we should accelerate AI capabilities research and not delay benefits of AGI
  • talking about catastrophic risks will hurt efforts to mitigate short-term human-rights issues with AI
  • concerned with the US-China cold war
  • that international treaties will not work
  • the genie is out of the bottle and we should just let go and avoid regulation
  • that open-source AGI code and weights are the solution
  • worrying about AGI is falling for Pascal’s wager"
AI Risk1AI1
Frontpage

70

Yoshua Bengio: Reasoning through arguments against taking AI safety seriously
New Comment


3 comments, sorted by
top scoring
Click to highlight new comments since:
[-]faul_sname42

In the section "For those who think that open-source AGI code and weights are the solution"

If we had the DNA sequence of an extremely dangerous virus, would it be best to share it publicly or not? If the answer is obvious to you in this case, think twice about the case for AGI algorithms and parameters.

The National Institute of Health's answer is "yes". Here's variola major (smallpox) for example. So those arguing that it's a bad idea to share ML algorithms and artifacts should either make the case that the NIH is wrong to share the smallpox genome or make the case that sharing some subset of ML algorithms and artifacts is more dangerous than sharing the smallpox genome.

In fairness, some people have in fact made decent cracks at the argument that sharing some types of ML-related information is more dangerous than sharing the smallpox genome. Still, I think the people arguing that the spread of knowledge is the thing we want to target, rather than the spread of materials, could do a better job of making that argument. But the common-sense "you wouldn't share the genome of a dangerous virus" argument doesn't work because we would, in fact, share the genome of a dangerous virus (and I personally think that it's actively good that we share the genomes of dangerous viruses, because it allows for stuff like this).

Reply11
[-]Nathan Helm-Burger41

Personally, my take is that this is a dangerous mistake on the part of NIH and the worldwide academic community generally, that so much dangerous information has been released in regards to virology and bioweapons. Unfortunately, I'm clearly currently in the minority among scientists who think this. I wish I had a good idea of how to change people's minds on this issue, and the connected one of publishing AI capabilities implementation details.

Reply
[-]whestler32

I initially thought there must be some simple reason that publishing the DNA sequence is not a dangerous thing to do, like "ok, but given that you would need a world class lab and maybe even some techniques which haven't even been invented yet to get it to work, it's not a dangerous thing to publish".

According to this article from 2002, synthesising smallpox would be tricky, but within the reach of a terrorist organisation. Other viruses may be easier. 

“Scientifically, the results are not surprising or astounding in any way,” says virologist Vincent Racaniello of Columbia University. “The point here, of course, is that the DNA can be synthesized from the [genetic] sequence, and this could be done by any third-rate terrorist.”

Apparently, large organisations like the NIH are foolhardy enough to publish dangerous data like this. I wonder if there's some other justification, like "the data was already public, in such a way that it could not be removed"

Reply11
Moderation Log
3Comments