"Whereas humans aren't benevolent, despite what would seem to be significant selection pressure for benevolence operating over the same time-period."
Evolution doesn't act for the good of the species, so this looks wrong.
That's an interesting distinction. When I said:
And suppose that by "benevolent" we mean capable of consistently and reliably acting in the long-term best interests of humans.
...I in fact meant humans as individuals. And traits that act in the long-term best interests of individuals do in fact exert selection pressure on the genome.
But perhaps you're suggesting that by "benevolent" I ought to have meant capable of consistently and reliably acting in the long-term best interests of humanity as a species, and not necessarily the individual?
It's way late in my time zone and I suspect this question isn't technically coherent on the grounds that the right answer to "why care about AI risk?" is going to be complicated and have a bunch of parts that can't be separated from each other. But I'm going to share a thought I had anyway.
It seems to me like probably, the answer to the question of how to make AIs benevolent isn't vastly more complicated than the answer of how to make them smart. What's worrisome about our current situation, however, is that we're currently putting way more effort into making AIs smart than we are into making them benevolent.
Agree? Disagree? Have an orthogonal answers to the title question?