Link from March that apparently hasn't been discussed here: Y-Combinator's Sam Altman thinks AI needs regulation:

“The U.S. government, and all other governments, should regulate the development of SMI [Superhuman Machine Intelligence],”

“The companies shouldn’t have to disclose how they’re doing what they’re doing (though when governments gets serious about SMI they are likely to out-resource any private company), but periodically showing regulators their current capabilities seems like a smart idea,”

“For example, beyond a certain checkpoint, we could require development [to] happen only on airgapped computers, require that self-improving software require human intervention to move forward on each iteration, require that certain parts of the software be subject to third-party code reviews, etc.,”

The regulations should mandate that the first SMI system can’t harm people, but it should be able to sense other systems becoming operational,

Further, he’d like to see funding for research and development flowing to organizations groups that agree to these rules.

Sounds sensible.