I've been using something like "A self-optimizing AI would be so powerful that it will just roll over the human race unless it's programmed to not do that."
That's good, but the "unless it's programmed to not do that" bit should probably be expanded a little bit to make it clear that programming an AI "to not do that" is a lot harder than we intuitively expect.
I've been using something like "A self-optimizing AI would be so powerful that it will just roll over the human race unless it's programmed to not do that."
Any others?