James: See The Hidden Complexity of Wishes. You can't think of everything you'd need to ban.

sophiesdad: I don't see convincing anyone who matters in the government of the potential of AGI, let alone the need for Friendliness, as very likely. Tim Kyger (DOD employee) says "I don't know a *soul* in DoD or any of the services off the top of my head that has any *inkling* of the very existence of trans-H or of the various technical/scientific lanes of approach that are leading to a trans/post-human future of some sort. Zip. Zero. Nada." If you could get the government involved, it would most likely make things much worse. Paraphrasing Eliezer(2002), you'd be a lot less likely to get a positive outcome than an outright ban on AGI, an utterly unworkable set of regulations designed and enforced by people who don't understand the problem, or a Manhattan Project run by people who don't understand the problem and virtually guaranteed to destroy the world.