That's an interesting distinction. When I said:
And suppose that by "benevolent" we mean capable of consistently and reliably acting in the long-term best interests of humans.
...I in fact meant humans as individuals. And traits that act in the long-term best interests of individuals do in fact exert selection pressure on the genome.
But perhaps you're suggesting that by "benevolent" I ought to have meant capable of consistently and reliably acting in the long-term best interests of humanity as a species, and not necessarily the individual?
Ah, I was thrown by the plural at the end of your definition.
But by saying humans aren't benevolent, you mean that there are no/few humans for which it's true that "this person consistently acts in hir own best interests?"
It's way late in my time zone and I suspect this question isn't technically coherent on the grounds that the right answer to "why care about AI risk?" is going to be complicated and have a bunch of parts that can't be separated from each other. But I'm going to share a thought I had anyway.
It seems to me like probably, the answer to the question of how to make AIs benevolent isn't vastly more complicated than the answer of how to make them smart. What's worrisome about our current situation, however, is that we're currently putting way more effort into making AIs smart than we are into making them benevolent.
Agree? Disagree? Have an orthogonal answers to the title question?