We don't know how to measure many of the things humans value, so we use proxy measures. These proxies are leaky allowing exploitation.
I'm tempted to call Bakkot's idea King Robert's Rule after the character in A Game of Thrones. He was very good at becoming King, but bad at ruling.
It's not that there's a failure of measurement, it's that sometimes there are high rewards for winning, and (while you want your King to be competent at war), competence at war isn't how people are measuring their ideal King, it's just that the route to becoming King might include being a war leader.
or: Why Everything Is Terrible, An Overview.1
It sounds like a theory which explains too much. But it's not a theory, hardly even an explanation, more a pattern that manifests itself once you start trying to seriously answer rhetorical questions about the state of the world. From many perspectives, it's obvious to the point of being mundane, practically tautological, but sometimes such obvious facts are worth pointing out regardless.
The idea is this: The subset of participants which rises to prominence in any area does so because its members have traits helpful to becoming prominent, not necessarily because they have traits which are desirable. Thus, without ongoing and concerted effort, a great many arenas end up dominated by players employing strategies which are bad for everyone.
This comes up again and again: