However, humans and human societies are currently near some evolutionary equilibrium.
I think there's plenty of evidence that human societies are not near some evolutionary equilibrium. Can you name a human society that has lasted longer than a few hundred years? A few thousand years?
On the biological side, is there any evidence that we have reached an equilibrium? (I'm asking genuinely)
It's very possible that individual intelligence has not evolved past its current levels because it is at an equilibrium, beyond which higher individual intelligence results in lower social utility.
The consensus among biologists seems to be that social utility has zero to very little impact on evolution. See http://en.wikipedia.org/wiki/Group_selection
In fact, if you believe SIAI's narrative about the danger of artificial intelligence and the difficulty of friendly AI, I think you would have to conclude that higher individual intelligence results in lower expected social utility, for human measures of utility.
Higher levels of human intelligence result in a lower expected social utility for some other species (we are better at hunting them). It does not result in lower expected social utility for humans as we are generally good to other humans. Higher levels of individual intelligence have brought us the great achievements of human kind with very few downsides. The concern with AGI is that it might treat humans as humans treat some other species.
If anything, the reason we don't see a rapid rise of intelligence among human beings is that it does not provide much evolutionary benefit. In modern societies, people don't die for being dumb (usually) and sexual selection doesn't have much impact since most people only have child with a single partner.
On the biological side, is there any evidence that we have reached an equilibrium? (I'm asking genuinely)
On one hand, evolution appears to work in a punctuated manner, meaning that individual components of evolutionary systems are usually at equilibrium.
On the other hand, brain volume in our ancestors rose smoothly from 3 million years ago to the present.
On the other other hand, some Neanderthals had larger brains than modern humans.
...Higher levels of human intelligence result in a lower expected social utility for some other species (we are better at h
An interesting new article, "Cooperation and the evolution of intelligence", uses a simple one-hidden-layer neural network to study the selection for intelligence in iterated prisoners' dilemma and iterated snowdrift dilemma games.
The article claims that increased intelligence decreased cooperation in IPD, and increased cooperation in ISD. However, if you look at figure 4 which graphs that data, you'll see that on average it decreased cooperation in both cases. They state that it increased cooperation in ISD based on a Spearman rank test. This test is deceptive in this case, because it ignores the magnitude of differences between datapoints, and so the datapoints on the right with a tiny but consistent increase in cooperation outweigh the datapoints on the left with large decreases in cooperation.
This suggests that intelligence is an externality, like pollution. Something that benefits the individual at a cost to society. They posit the evolution of intelligence as an arms race between members of the species.
ADDED: The things we consider good generally require intelligence, if we suppose (as I expect) that consciousness requires intelligence. So it wouldn't even make sense to conclude that intelligence is bad. Plus, intelligence itself might count as a good.
However, humans and human societies are currently near some evolutionary equilibrium. It's very possible that individual intelligence has not evolved past its current levels because it is at an equilibrium, beyond which higher individual intelligence results in lower social utility. In fact, if you believe SIAI's narrative about the danger of artificial intelligence and the difficulty of friendly AI, I think you would have to conclude that higher individual intelligence results in lower expected social utility, for human measures of utility.