Your argument is modeling AI as a universal optimizer.
I agree that AI that is an universal optimizer will be more likely to be in this camp (especially the 'take control') bit, but I think that isn't necessary. Like, if you put an AI in charge of driving all humans around the country, and the way it's incentivized doesn't accurately reflect what you want, then there's risk of AI misbehavior. The faulty reward functions post above is about an actual AI trained used modern techniques on a simple task that isn't anywhere near a universal optimizer.
The argument that I don't think you buy (but please correct me if I'm wrong) is something like "errors in small narrow settings, like an RL agent maximizing the score instead of maximizing winning the race, suggest that errors are possible in large general settings." There's a further elaboration that goes like "the more computationally powerful the agent, and the larger the possible action space, the harder it is to verify that the agent will not misbehave."
I'm not familiar enough with reports of OpenCog and others in the wild to point at problems that have already manifested; there are a handful of old famous ones with Eurisko. But it should at least be clear that those are vulnerable to adversarial training, right? (That is, if you trained LIDA to minimize some score, or to mimic some harmful behavior, it would do so.) Then the question becomes if you'll ever do that on accident while doing something else deliberately. (Obviously this isn't the only way for things to go wrong, but it seems like a decent path for an existence proof.)
All AIs will "misbehave" -- AI is software, and software is buggy. Human beings are buggy too (see: religious extremism). I see nothing wrong with your first paragraph, and in fact I quite agree with it. But I hope you also agree that AI misbehavior and AI existential risk are qualitatively different. We basically know how to deal with buggy software in safety-critical systems and have tools in place for doing so: insurance, liability laws, regulation, testing regimes, etc. These need to be modified as AI extends its reach into more safety-criti...
There have been a few attempts to reach out to broader audiences in the past, but mostly in very politically/ideologically loaded topics.
After seeing several examples of how little understanding people have about the difficulties in creating a friendly AI, I'm horrified. And I'm not even talking about a farmer on some hidden ranch, but about people who should know about these things, researchers, software developers meddling with AI research, and so on.
What made me write this post, was a highly voted answer on stackexchange.com, which claims that the danger of superhuman AI is a non-issue, and that the only way for an AI to wipe out humanity is if "some insane human wanted that, and told the AI to find a way to do it". And the poster claims to be working in the AI field.
I've also seen a TEDx talk about AIs. The talker didn't even hear about the paperclip maximizer, and the talk was about the dangers presented by the AIs as depicted in the movies, like the Terminator, where an AI "rebels", but we can hope that AIs would not rebel as they cannot feel emotion, so we should hope the events depicted in such movies will not happen, and all we have to do is for ourselves to be ethical and not deliberately write malicious AI, and then everything will be OK.
The sheer and mind-boggling stupidity of this makes me want to scream.
We should find a way to increase public awareness of the difficulty of the problem. The paperclip maximizer should become part of public consciousness, a part of pop culture. Whenever there is a relevant discussion about the topic, we should mention it. We should increase awareness of old fairy tales with a jinn who misinterprets wishes. Whatever it takes to ingrain the importance of these problems into public consciousness.
There are many people graduating every year who've never heard about these problems. Or if they did, they dismiss it as a non-issue, a contradictory thought experiment which can be dismissed without a second though:
We don't want our future AI researches to start working with such a mentality.
What can we do to raise awareness? We don't have the funding to make a movie which becomes a cult classic. We might start downvoting and commenting on the aforementioned stackexchange post, but that would not solve much if anything.