How about existing examples of specific technologies where the risks outweigh the benefits? And perhaps a statement or two as to why this is.
Let me start with the obvious one - nuclear weapons. Nuclear weapons can be released in a matter of minutes, in theory, and at one time there was enough of them ready to fire to wipe out all the urban areas for Western civilization. Also, no real defense.
With this said, it is unclear to me as to the real probabilities of this risk. If an accidental release had occurred, and a whole city in the Eastern or Western bloc had been lost, would this have likely lead to an escalating series of exchanges resulting in every available weapon being fired? Also, was there any point with the policymakers with legitimate power considered an all out attack to destroy the other side, in essense hoping to extermine all military and infrastructure while "only" losing some cities.
And, despite this apparent existential risk, relatively few people were killed in global wars between nuclear armed nations after their development.
The reason why the nuclear weapons are such a risk is that human civilization is concentrated in nice neat target clusters, limits on control systems makes interception of incoming reentry vehicles difficult and unreliable, and due to tech limitations, anti-ICBM defenses are much more complex and resource intensive than the missiles themselves.
Molecular nanotechnology, if used to build "killer nanobots" that combust organic molecules with oxygen and so essentially incinerate all life in the biosphere a few molecules at a time, presumably have a similar problem. Destructive, weaponized bots would not need to be self replicating, and could be orders of magnitude simpler than a device that could somehow hunt down and disable the weaponized robots.
And of course, malicious AI. Pretty much unless the ecological niches that a malicious AI would occupy (aka all the major computing systems and automated factories) are already occupied by friendly AIs, we're all screwed.
What mass ratio of mass consumed to bot mass per unit time are you supposing? If a bot can consume its own mass ten times per second, and the lab produces a kiloton of bots before releasing them, it still takes nearly two years to consume all biomass on earth, if there is no replenishment. (560 GT biomass counting only carbon.) Presumably we're all in trouble well before that, but it still seems slow compared to what I read into your comments. I also think the bots would likely be slower and less numerous than that. (Heat concerns, rate of movement, and complexity requirements of self-repair would likely make these bots slow, but I don't really know how slow.)
Today's post, Raised in Technophilia was originally published on 17 September 2008. A summary (taken from the LW wiki):
Discuss the post here (rather than in the comments to the original post).
This post is part of the Rerunning the Sequences series, where we'll be going through Eliezer Yudkowsky's old posts in order so that people who are interested can (re-)read and discuss them. The previous post was My Best and Worst Mistake, and you can use the sequence_reruns tag or rss feed to follow the rest of the series.
Sequence reruns are a community-driven effort. You can participate by re-reading the sequence post, discussing it here, posting the next day's sequence reruns post, or summarizing forthcoming articles on the wiki. Go here for more details, or to have meta discussions about the Rerunning the Sequences series.