Also from Life at the Speed of Light:
...the Presidential Commission for the Study of Bioethical Issues released a report in December 2010 entitled New Directions: The Ethics of Synthetic Biology and Emerging Technologies...
Among its recommendations to the president, the commission said that the government should undertake a coordinated evaluation of public funding for synthetic-biology research, including studies on techniques for risk assessment and risk reduction and on ethical and social issues, so as to reveal noticeable gaps, if one considered that “public good” should be the main aim. The recommendations were, fortunately, pragmatic: given the embryonic state of the field, innovation should be encouraged, and, rather than creating a traditional system of bureaucracy and red tape, the patchwork quilt of regulation and guidance of the field by existing bodies should be coordinated.
Concerns were, of course, expressed about “low-probability, potentially high-impact events,” such as the creation of a doomsday virus. These rare but catastrophic possibilities should not be ignored, given that we are still reeling from the horrors of September 11. Nor should they be overstated: though one can gain access to “dangerous” viral DNA sequences, obtaining them is a long way from growing them successfully in a laboratory. Still, the report stated that safeguards should be instituted for monitoring, containment, and control of synthetic organisms — for instance, by the incorporation of “suicide genes,” molecular “brakes,” “kill switches,” or “seatbelts” that restrain growth rates or require special diets, such as novel amino acids, to limit their ability to thrive outside the laboratory. As was the case with our “branded” bacterium, we need to find new ways to label and tag synthetic organisms. More broadly, the report called for international dialogue about this emerging technology, as well as adequate training to remind all those engaged in this work of their responsibilities and obligations, not least to biosafety and stewardship of biodiversity, ecosystems, and food supplies. Though it encouraged the government to back a culture of self-regulation, it also urged it to be vigilant about the possibilities of do-it-yourself synthetic biology being carried out in what it called “noninstitutional settings.” One problem facing anyone who casts a critical eye over synthetic biology is that the field is evolving so quickly. For that reason, assessments of the technology should be under rolling review, and we should be ready to introduce new safety and control measures as necessary.
One open question in AI risk strategy is: Can we trust the world's elite decision-makers (hereafter "elites") to navigate the creation of human-level AI (and beyond) just fine, without the kinds of special efforts that e.g. Bostrom and Yudkowsky think are needed?
Some reasons for concern include:
But if you were trying to argue for hope, you might argue along these lines (presented for the sake of argument; I don't actually endorse this argument):
The basic structure of this 'argument for hope' is due to Carl Shulman, though he doesn't necessarily endorse the details. (Also, it's just a rough argument, and as stated is not deductively valid.)
Personally, I am not very comforted by this argument because:
Obviously, there's a lot more for me to spell out here, and some of it may be unclear. The reason I'm posting these thoughts in such a rough state is so that MIRI can get some help on our research into this question.
In particular, I'd like to know: