In episode 158 of Cortext podcast, CGP Grey gives their high-level reason why they are worried about AI.

My one line summary: AI should not be compared to nuclear weapons but instead to biological weapons or memes, which evolve under the implicit evolutionary pressures that exist, leading to AI's that are good at surviving and replicating. 

The perspective is likely known by many in the community already, but I had not heard it before. Interestingly, there have actually been experiments where they just put random strings of code in an environment where they interact, and self-replicating code appeared. See Cognitive Revolution podcast on 'Computational Life: How Self-Replicators Arise from Randomness', with Google researchers Ettore Randazzo and Luca Versari.

I quote the relevant part of the podcast below, but I recommend listening because the emotion and delivery is impactful. It is from 1:22:00 onwards.

To be explicit and not beat around the bush, when I try to think, “Oh, what is beyond this barrier, beyond which it might be impossible to predict?” it's like, well, if I’m just in Vegas and placing odds on this roulette wheel, almost all of those outcomes are extraordinarily bad for the human species. There are potentially paths where it goes well, but most are extremely bad for a whole bunch of reasons.

I think of it like this: people who are concerned like me often analogize AI to something like building nuclear weapons. It’s like, “Ah, we’re building a thing that could be really dangerous.” But I just don’t think that’s the correct comparison, because a nuclear weapon is a tool. It's a tool like a hammer. It’s a very bad hammer, but it is fundamentally mechanical in a particular way.

But the real difference, where do I disagree with people, where do other people disagree with me, is that I think a much more correct way to think about AI is to compare it to biological weaponry. You’re building a thing able to act in the world differently than how you constructed it. That’s what biological weapons are—they’re alive. A nuclear bomb doesn’t accidentally leave the factory on its own, whereas biological weapons do, can, and have. And once a biological weapon is out in the world, it can develop in ways that you never anticipated ahead of time.

That’s the way I think about these AI systems.

[...]

The reason I like to talk about it this way, particularly with biological weapons, is because I want to shortcut discussions that can be distracting, like, “Are these things alive? Are they thinking thoughts? Blah blah blah” That’s an interesting conversation, but when you’re thinking about what to do, that whole conversation is nothing but a pure distraction. This is why I prefer the biological weapon analogy—no one is debating the intent of a lab-created smallpox strain. No one wonders if the smallpox virus is “thinking” or "does it have any thoughts of its own?". Instead, people understand that it doesn’t matter. Smallpox germs, in some sense, “want” something: they want to spread, they want to reproduce, they want to be successful in the world, and are competing with other germs for space in human bodies. They're competing for resources. The fact that they’re not conscious doesn’t change any of that.

So I feel like these AI systems act as though they are thinking, and fundamentally it doesn’t really matter whether they are actually thinking or not because externally the effect on the world is the same either way. That’s my main concern here: I think these systems are real dangerous because it is truly autonomous in ways that other tools we have ever built are not.

It's like, look, we can take this back to another video of mine, This video will make you angry, which is about thought germs and I have this line about thought germs which - I mean memes but I just don't want to say the word because I think that that's like distracting in the modern context - but it's like memes are ideas and they compete for space in your brain and their competition is not based on how true they are. Their competition is not based on how good for you they are. Their competition is based on how effectively they spread, how easily they stay in your brain, and how effective they are at repeating that process.

And so it's the same thing [as biological weapons] again. You have an environment in which there are evolutionary pressures that slowly change things. I really do think one of the reasons it feels like people have gotten harder to deal with in the modern world is precisely because we have turned up the evolutionary pressure on the kinds of ideas that people are exposed to. Ideas have in some sense become more virulent. They have become more sticky. They have become better at spreading because those are the only ideas that can survive once you start connecting every single person on Earth, and you create one gigantic jungle in which all of these memes are competing with each other.

What I look at with AI and with the kind of thing that we're making here is we are doing the same thing right now for autonomous and semi-autonomous computer code. We are creating an environment under which, not on purpose but just because that's the way the world works, there will be evolutionary pressure on these kinds of systems to spread and to reproduce themselves and to stay around and to like "accomplish whatever goals they have". In the same way that small pox is trying to accomplish its goals. In the same way that mold is trying to accomplish its goals. In the same way that anything which consumes and uses resources is under evolutionary pressure to stick around so that it can continue to do so.

That is my broadest highest level most abstract reason why I am concerned. I feel like getting dragged down sometimes into the specific of that always ends up missing that point. It's not about anything that's happening now. It's that we are setting up another evolutionary environment in which things will happen - which will not be happening because we directed them as such - they will be happening because this is the way the universe works.