LESSWRONG
LW

Comment Permalink
by Bakkot
4 min read
43

80

or: Why Everything Is Terrible, An Overview.1

 

It sounds like a theory which explains too much. But it's not a theory, hardly even an explanation, more a pattern that manifests itself once you start trying to seriously answer rhetorical questions about the state of the world. From many perspectives, it's obvious to the point of being mundane, practically tautological, but sometimes such obvious facts are worth pointing out regardless.

The idea is this: The subset of participants which rises to prominence in any area does so because its members have traits helpful to becoming prominent, not necessarily because they have traits which are desirable. Thus, without ongoing and concerted effort, a great many arenas end up dominated by players employing strategies which are bad for everyone.

 

This comes up again and again:

  • Why does science (or rather, the publisher-based model thereof) so frequently produce results which are laughably wrong? Because those journals which don't publish retractions or reproductions will more frequently be the first to publish revolutionary results, and so become more widely read and widely cited. Journals don't attract authors by being as accurate as possible; they win by looking important.
  • Why do cigarette companies target kids and teens whenever they think they can get away with it, and breed tobacco for maximized nicotine? Because those companies which do will turn more profit and thus last longer and grow faster than those that don't, and so have more resources to devote to proliferating. Companies don't expand by playing fair; they win by making and keeping customers.
  • Why is the Make-A-Wish Foundation sitting on more donations than it knows what to do with when the Against Malaria Foundation could have used that money to save literally tens or hundreds of thousands of lives per year? Because knowing how to elicit donations is a skill almost completely unrelated to knowing how to spend donations, and because American children with cancer make for better advertising than African children with malaria. Charities don't get donations by making the best possible use of their money; they win by advertising effectively towards potential donors. (cf. Efficient Charity)
  • Why do governments inevitably end up run by career lawyers and politicians instead of scientists and economists2? Because polarizing rhetoric and political connections look better than a nuanced, accurate understanding of the issues. There is only finite time for training and practice, and eventually a choice must be made between training in looking good and training in being good. People don't get elected or appointed by being good Bayesians; they win by being popular.
  • Why do the big media channels seem to be more concerned with celebrities than science, and spend more time talking about individual murders than they do entire genocides? Because those channels talking about Laci Peterson seem more personal and are thus more watched than those talking about some religious sect in China. Television programming isn't determined by what's important; what wins is what's watched.
  • Why is the sex ratio in animals almost always nearly 1:1, when a population with one male for every five females could grow faster and adapt to problems more readily? Because in such a population, or in any population with a sufficiently large gender imbalance, a gene causing a woman to only have male children will be vastly overrepresented in the grandchild generation relative to the rest of the population, and so shift the balance closer to 50/50. Genes don't proliferate by being good for the species; they win by being good for themselves. (cf. Evolutionarily stable strategy, evolutionary game theory.)
  • Why do most big businesses make use of sweatshop conditions and shady tax dodges? Because the businesses which do so will outperform the businesses which don't. Corporations don't grow by being nice; they win by being profitable.
  • Why do so many apparently intelligent people spend hours per day idly browsing the likes of Reddit, Hacker News, or TVTropes (or indeed LW), when a similar dedication to active self-improvement could have made them a master of a field inside of a decade? (Using for back-of-the-envelope's sake the supposition that 10,000 hours of practice are required for mastery of some specific art, we find that three hours per day for ten years is approximately 1.1 masteries.) Because which activities become habitual is determined by their immediate dopamine release, and for intelligent people the act of (say) reading about strategies for becoming an effective entrepreneur makes for more instant dopamine than does the painful daily grind involved in actually becoming an entrepreneur. Activities don't become part of daily life by being useful; they win by tricking your brain into making them feel good.

It's extremely important to remember that none of this requires active malice, not even foresight or awareness of the strategy utilized. If someone or something happens upon a strategy like those described above, it will outperform its peers and become more widespread. This requires no conspiracy, no evil forces at work in the world, not even any individual shifting in their personal stance; these are just the stable strategies  towards which the set of surviving players eventually converges. 
The next question: What can we do about it?

1I have distinct recollections of having read an article much like the one I've written here at some point in the past. However, I can't find said article, so at the least we can let this article serve as a refresher or another viewpoint on the matter. (ETA: evgenit and gwern both point out that the article I'm thinking of is Scott Aaronson's Malthusianisms. [Aaronson refers to these states as Nash equilibria, which is not strictly correct; there's no underlying assumption about the rationality of the players here. You don't need intelligent participants for selection to operate. This is more a quibble over terminology than anything.])
2Until recently, China stood as a notable exception; now it appears that the next generations of leaders will have built their entire careers on shilling the party line.
3Tangentially related reading material: Bruce Schneier's new book.
4No, neither footnote 3 nor this footnote actually have corresponding backreferences.
5I wasn't entire sure which section of the site this would be best suited for. Hopefully this is appropriate. ETA: Also, as this is my first submission outside of comment threads, any feedback is highly appreciated.
Moloch
Frontpage

80

The Stable State is Broken
New Comment
43 comments, sorted by
top scoring
Click to highlight new comments since:
[-]Alicorn250

Your list of examples is lovely and illustrative. There are enough of them to show the common thread through all of them easily, and you write them concisely and readably.

Reply
[-]evgenit160

Reminds me of Scott Aaronson's Malthusianisms. Is this the article you couldn't find?

Also, I am not sure your example of science is correct: After all, plenty of very famous journals do publish retractions, and some that do not are (rightly) laughed at (parapsychology journals for example).

Reply
[-]Antisuji90

Your article and Aaronson's are similar in tone and superficially similar in content, but yours embodies a more specific and more interesting (to me) central idea. To wit, Malthusianisms is basically about Umeshisms, which is really a restatement of the Pareto Principle. Whereas this article is about the Hansonesque idea that success in a domain is not about proficiency in the domain.

Reply
[-][anonymous]60

The idea seems to be in essence a restatement or reapplication of Campbell's Law: A system's metrics and its goals are not equivalent, so successful behavior tends to become deranged. Though they are not quite the same idea.

Not to say that expounding upon the idea is a bad thing. Your examples are quite elegant.

Reply
[-]fubarobfusco20

Campbell's Law posits that a metric is used because it has historically correlated with a difficult-to-measure desirable property; but that it becomes deranged only when it is used to make decisions that the measured people care about.

Historically, better-educated students do well on standardized tests, when those tests don't matter. But once you enact a test to discriminate amongst students for purposes those students care about (like getting into prestigious colleges), your measurement of academic achievement will be confounded by your measurement of test-taking skills.

Reply
[-][anonymous]00

Certainly. I think the same principle applies in many of the listed cases, though. Scientific publications in particular likely developed their current standards at least partially because in the past they filtered for genuinely revolutionary results.

One might as easily say that Campbell's Law is a sub-principle of the observed phenomenon.

Reply
[-]evgenit00

Replications make much more sense as an example. You could also add the file-drawer problem in research. Why do we not see studies that do not find anything? Because there is no prestige in publishing them. (Some journals do try to correct for this, but they have to explicitly do that)

Reply
[-]fiddlemath140

I agree, this class of problems is enormous, and has a hand in practically all mismanagement and misdirection of human effort. The problem is so aggravating, though, because humans seem not to expect it to happen. Why do these feel like "problems", when the underlying behavior is exactly what we'd expect given our knowledge of stable strategy?

In the case of human endeavor, I suspect this is a problem because we do not try hard enough to defend against it. We seem surprised and indignant when systems that purport to do good are "gamed." If more people were more aware that this was the consequence of strategic agents, they might watch harder for signs of destructive strategy, and more carefully design the systems that they build and manage against such strategy.

(Hmm. I notice that I'm posing a purportedly-fully-general strategy to a fully-general problem, without evidence or examples. And I'm claiming that better global understanding is a solution, when it is probably just applause lights to my imagined audience. And I still think that what I'm saying is right! Wow. You should probably ignore this comment's content, but I'll still leave it here, as a counterexample.)

Reply
[-]orthonormal40

Also because our ancestral environment didn't have large enough selection pools for the social and economic examples to arise- we could blame any failures in the tribe on particular people, and so we think of such things as vice rather than obvious selection bias.

Reply
[-]fburnaby20

To add some credence to your recommendations: since actually understanding the logic of stable strategies, I feel much less frustrated by the examples cited by Bakkot than I used to when I assumed they were the result of evil. I also view them as problems to be solved, not enemies to scorn. This really truly seems like an improved disposition being caused by understanding.

Thought to be fair, my actions have been changed much less than my dispositions have. Such understanding has, at most, impacted my behaviours which I associate with far-mode: how I vote, argue and make life decisions. My leisure activities, smoking habits, purchasing habits haven't changed.

Reply
[-]Mitchell_Porter120

I find this article to be riddled with questionable statements, judgements, and explanations. The argument schema is that "X is prominent where Y would be better" - but in most cases, it's not at all clear that Y would be better, or that the explanation for X's prominence is correct. In some cases, the triumph of Y would be in the author's self-interest, in other cases I think it's just that he's not aware of alternative considerations or perspectives.

Let's start with the claim that rule by scientists and economists would be better than rule by politicians and lawyers. I see this as self-interest, or even as the vanity of believing that one's own group - here, rationalists and quantitative thinkers - is generally superior. Politicians have people skills and lawyers know how the system works, both attributes which are actually needed to run a government well. At its worst, a government by number-crunching intellectuals would be an ineffectual technocracy, first trying to reshape the world according to the wacky theories of academics, and then helpless when a mob of pitchfork-wielding villagers show up, wanting to kill the monster.

It's funny that in one paragraph, we hear that science is frequently dysfunctional, and then a few paragraphs later, that scientists should be in charge. So maybe it's only the right sort of scientists who should be in charge. Someone else remarked that parapsychology is hardly a good example of what is prominent in science; parapsychologists are shunned like creationist biologists, as pseudoscientists. So, just as the item about politics can be read as a statement that "people in my cultural group should be running everything", perhaps the item about science can be read as a statement that "stuff like parapsychology shouldn't exist at all!" Certainly, the remarks about journals which achieve prominence because they have low quality control and so get to publish the breakthroughs, seems like pure mythmaking, a "just-so story"; I can't think of a single journal which matches that description - and there ought to be a dozen such journals, if this really is the basic explanation for the existence of bad science in print. In the real world, breakthroughs tend to be achieved by quality people and published in quality journals which everyone reads because they are known to have high standards. Bad journals may provide a space in which bad science gets to see print, but they aren't "widely read and widely cited".

The argument that having five girls for every boy would be better for the species is new to me. Obviously this is sexually appealing for males, so we can count this as another case in which the supposedly all-around-better situation conveniently happens to be better for the author's specific group. Despite the evo-psych opinion that women want quality rather than quantity in a mate and that they are more bisexual than men, the odds are that women would be personally worse off in such a world. However, what is asserted here is that this 1:5 sex ratio is evolutionarily superior to the 1:1 ratio, because the population grows and adapts faster. That is a very dubious assertion. It's often said that the 1:1 regime works through a functional division in which males are a reservoir of genetic variance and female choice of mates provides a selective filter. Maybe in reality the mate selection by both sexes is equally important, but for different traits. In any case, I don't see the knockdown argument in favor of 1:5, once you take sexual selection into account.

I could question some of the other items too.

Reply
[-]Vaniver90

Politicians have people skills and lawyers know how the system works, both attributes which are actually needed to run a government well.

As an example, consider Laplace, one of the cleverest mathematicians in history, who got fired from his ministry position after six weeks.

Geometrician of the first rank, Laplace was not long in showing himself a worse than average administrator; since his first actions in office we recognized our mistake. Laplace did not consider any question from the right angle: he sought subtleties everywhere, only conceived problems, and finally carried the spirit of "infinitesimals" into the administration.

Reply
[-]Vaniver20

The argument that having five girls for every boy would be better for the species is new to me.

I've seen it in a lot of places; it works better for species where male parental investment is minimal (which isn't as true for humans). Even then, the "good of the species" math and the "good for my line" math point in different directions.

Reply
[-]Kaj_Sotala100

Another example applies to memes: the ideas which survive are not the ones that are the best for people, but which are the best at replicating and perpetuating themselves. As Dawkins notes in his original meme essay:

To take a particular example, an aspect of doctrine that has been very effective in enforcing religious observance is the threat of hell fire. Many children and even some adults believe that they will suffer ghastly torments after death if they do not obey the priestly rules. This is a peculiarly nasty technique of persuasion, causing great psychological anguish throughout the middle ages and even today. But it is highly effective. It might almost have been planned deliberately by a macchiavellian priesthood trained in deep psychological indoctrination techniques. However, I doubt if the priests were that clever. Much more probably, unconscious memes have ensured their own survival by virtue of those same qualities of pseudo-ruthlessness that successful genes display. The idea of hell fire is, quite simply, self perpetuating, because of its own deep psychological impact. It has become linked with the god meme because the two reinforce each other, and assist each other's survival in the meme pool.

Another member of the religious meme complex is called faith. It means blind trust, in the absence of evidence, even in the teeth of evidence. The story of Doubting Thomas is told, not so that we shall admire Thomas, but so that we can admire the other apostles in comparison. Thomas demanded evidence. Nothing is more lethal for certain kinds of meme than a tendency to look for evidence. The other apostles, whose faith was so strong that they did not need evidence, are held up to us as worthy of imitation. The meme for blind faith secures its own perpetuation by the simple unconscious expedient of discouraging rational inquiry.

Blind faith can justify anything. If a man believes in a different god, or even if he uses a different ritual for worshipping the same god, blind faith can decree that he should die -- on the cross, at the stake, skewered on a Crusader's sword, shot in a Beirut street, or blown up in a bar in Belfast. Memes for blind faith have their own ruthless ways of propagating themselves. This is true of patriotic and political as well as religious blind faith.

Reply
[-]EE43026F100

That's a very important and basic observation.

You missed another example : cancer. Cankerous cells are much better at replicating themselves than normal cells are. Pluricellular organisms have a multitude of systems to keep their component cells in check, yet they still fail at it from time to time. Biology has had billions of years of evolution to fine tune how it enforces cooperation within larger organisms. Can we do better, especially as the components we're considering at our scale may be as complex and clever to us as cells are to an organism? (Meaning we may not have a comparative advantage even though we're subtler than evolution).

So aside from asking what we can do next, I'd like to add : "Can we do something next?" In order to enforce a system within which you won't observe such an effect, you might need to be larger, have more resources than the sum of all you're trying to steer. Otherwise, some part of that system will eventually take over. And even then, chance events may always remain beyond your capacity to control.

Reply
[-]Mass_Driver00

Cancer (and anti-cancer immune systems) might be a very fruitful analogy. To fight the tendency of systems to fall toward a stable state of suboptimal selfishness and shallowness, it might help to explicitly punish self-promotion or explicitly reward competence.

Something like the former happened in America during the Progressive Era of the 1900s and 1910s, when racketeers and robber barons were thrown out of the offices they'd schemed their way into by a cadre of self-appointed elitist technocrats.

Something like the latter happened in the 1940s and 1950s, when IQ tests, the SAT, vast increases in education expenditure, cracks in the wall of WASP solidarity, and major construction and infrastructure programs put meritocratic engineers at the top of many corporations and agencies.

Reply
[-]Eugine_Nier10

it might help to explicitly punish self-promotion or explicitly reward competence.

The problem is that then you have to keep the "reward and punish" system itself from corrupting.

Reply
[-]Voltairina20

One very general way we might address it is by building small, self-replicating versions of systems that have the tools to be more succesful and out compete existing systems for resources, but have built in components necessary to their running which stabilize the out-of-balance part.

If I wanted to make the united states more energy independant, for example, and I had the funds, I'd create an organization that installed for-free solar panels in neighborhoods at the price of a cut of future profits/savings from selling/using the power for a certain period of time. Then use those profits to slowly afford enough solar panels to transform another neighborhood, and so on. And I'd have the contracts for receiving income from clients include a part which specifies that if the organization ever sells solar panels rather than 'giving them away', that the contract holders were free of all payments and would simply own their equipment, I guess.

I suppose this'd cause an immune reaction of sorts to this 'viral' invasion of alternative energy by major power companies. But like all viruses, the company could probably find /some/ way to adapt. I'm assuming there are big holes in this model, and a lot of logistics problems. Like, growth might be intolerably slow for this to really take off. But its meant as more an example of the kind of thing - self reproducing, tooled to outcompete, with stability built in in some way.

I guess, addressing the internet addiction problems of people who're interested in self-help, having a tangible self-help club with group study sessions and regular meetups would be somewhat helpful- but LessWrong already has something a little like that with its meet ups, and I think internet distractions are still a problem. You almost want a self-help software that does things like block website access or redirect your attention to websites you /should/ be on at that moment, provides reminders or motivational quotes, insists on break times, and so on, to become popular or even required for some aspect of participation in some self-help community. Maybe it'd also include reporting tools for your self-help projects.

Reply
[-]RomeoStevens10

We don't know how to measure many of the things humans value, so we use proxy measures. These proxies are leaky allowing exploitation.

Reply
[-]NancyLebovitz30

I'm tempted to call Bakkot's idea King Robert's Rule after the character in A Game of Thrones. He was very good at becoming King, but bad at ruling.

It's not that there's a failure of measurement, it's that sometimes there are high rewards for winning, and (while you want your King to be competent at war), competence at war isn't how people are measuring their ideal King, it's just that the route to becoming King might include being a war leader.

Reply
[-]Alicorn50

"Baratheon's Rule" sounds less generic.

Reply
[-]CronoDAS00

It's long been known that it's easier to conquer than to rule...

Reply
[-]snarles00

Seems to me all of these questions are examples of market failures.

Reply
[-]syzygy00

It seems to me that in at least some of these examples you are confusing the map with the territory. Take genetics:

Genes don't proliferate by being good for the species; they win by being good for themselves.

Failing to be "good for the species" is not a fact about evolution, or genes. Thinking that evolution was supposed to be "good for the species" was just a heuristic humans used when trying to understand evolution. The "selfish gene" does not say anything meaningful about the phenomenon of evolution, it just shows that we have refined our understanding of evolution.

Now take politics:

Why do governments inevitably end up run by career lawyers and politicians instead of scientists and economists?

What does the phenomenon of government actually look like, in reality? Well, it looks like a system of human hierarchical organization in which career lawyers and politicians have a natural propensity to be on top. Thinking that the phenomenon of government has anything to do with understanding nuanced social issues is confusing the map with the territory.

Reply
[-]ESRogs00

I don't think I understand either of your points. In the genetics case, are you disagreeing with the contention that a species could proliferate more with a skewed gender ratio?

Or are you saying that whether or not it would is just uninteresting or the wrong question to ask for some reason?

Reply
[-]timtyler00

The idea is this: The subset of participants which rises to prominence in any area does so because its members have traits helpful to becoming prominent, not necessarily because they have traits which are desirable.

That sounds like a special case of Universal Darwinism. Some videos on the topic from me:

Reply
[-]buybuydandavis00

Short summary - evolutionary processes are beyond good and evil. Maybe you get a benevolent invisible hand, maybe a boot stomping a human face forever.

Reply
[-]Will_Newsome-10

Why do governments inevitably end up run by career lawyers and politicians instead of scientists and economists2?

To a second approximation the United States government is run by scientists and economists, and to a first approximation the Chinese government is also run by scientists and economists. No?

Reply
[-][anonymous]70

"Scientists and economists advise politicians" is not quite the same as "scientists and economists indirectly control policy." In practice, in the US, most advisors who are not also politicians are pitted against an equal and opposite party and then ignored. That is not universally true and it is not the only problem with the US government by a long shot, but it is a significant effect.

As for the Chinese government, you were more or less right until relatively recently. They were fairly successful too (which is not the same thing as being nice, of course).

Reply
[-]Eugine_Nier20

I think a large part of the problem is that even the scientists who rise to or near the top are selected for their ability to play politics and not their scientific ability.

Reply
[-]buybuydandavis10

I'd say they were selected for how well their conclusions support the desired policies of the politician appointing them.

Reply
[-]Eugine_Nier00

I wasn't just talking about US scientists.

Reply
[-]Larks00

In practice, in the US, most advisors who are not also politicians are pitted against an equal and opposite party and then ignored.

You expect this in equilibrium regardless of how far you are from the counterfactual with no economists, so this is not evidence either way.

Reply
[-][anonymous]00

Not following you there. In a mostly functional government I would expect to see either politicians with additional domain knowledge and few advisors or pure government functionaries with many advisors who had significant sway in their area of expertise. The current situation seems indicative of a particular ideological influence combined with the aforementioned career politician phenomenon.

I am not asserting a lack of economists - just a lack of influence over policy-in-practice.

Reply
[-]Multiheaded-30

They were fairly successful too (which is not the same thing as being nice, of course).

Many argue that this was only because until the last decade and a half Chinese society has been fairly archaic, indeed pre-industrial in many aspects such as culture, and could be directed by accordingly simple means. It takes a lot more patience, wisdom and subtlety to steer an economy which depends upon Silicon Valley enterpreteurs (as shown by its nearly universal mishandling today, although I know nothing about what should be done) than a forcefully industrialized one with many leftovers of a collectivist agrarian one - because both formal and informal relations in it are different; in Marxist terms, both the productive forces and the relations of production have been undergoing a dramatic shift in China.

Remember, almost all the talk about "democracy" and "human rights" in China is nonsense because it looks at politics first, while these concepts (concrete enough, just mostly misconstructed) arguably depend upon an economic foundation (as described by Marx) first, a nation's culture (as mostly ignored by Marx) second, and only lastly upon the formal systems and fleeting power arrangements that are "Politics" in the narrow sense. I'm not a Marxist, I just think that the Marxist approach (see e.g. Slavoi Zizek) is uncharacteristically clear and spot-on in this particular regard.

(This is a forward payment of sorts for my long-promised post on the economic underpinnings of democracy and freedom.)

[This comment is no longer endorsed by its author]Reply
[-]Multiheaded00

I've been PM'd with an explanation of the above being downvoted. Something about how mentioning Marx in any content on LW could be inflammatory, attract some wrong kinds of attention, etc, etc. To be honest, I'm incredulous and a little pissed off, but I'll comply.

In return, would the downvoters kindly please tell me if they find any specific claims in this opinion of mine mistaken or overly vague/bold/etc?

Reply
[-]satt50

Something about how mentioning Marx in any content on LW could be inflammatory, attract some wrong kinds of attention, etc, etc.

Is that an accurate paraphrase? As it is it sounds paranoid. A quick Google search turns up loads of drama-free mentions of Marx, including a sequences post.

Reply
[-]Costanza-30

You describe a very general principle, which might be described as survival of the fittest. Darwin has explained how it applies in the context of biology. "Survival of the fittest" does not always mean survival of the biggest or strongest or fastest. Tigers may be endangered, but rats and cockroaches are not. You extend the principle from genes to what Dawkins has named as memes. Also, you describe how similar principles apply in an economic free market. This is not an entirely original observation, but reminding people of these principles is good.

Having described what, by your account, is a fundamental and universal principle, you ask, what can we do about it? If it is as fundamental as you say, we acknowledge it, and understand it, and adjust ourselves to it. What do we do about the second law of thermodynamics?

You know what has been done in the past context of economics. The Soviet Union attempted to control the entire economy from the top down. Even so, the Soviet Union could not escape competition from the outside world. Right now, North Korea is trying very hard to control the economy from the top down and shelter itself from competition from the outside world.

What do we do about disease? Lots of things, of course, including nutrition and hygiene. Also, antibiotics. But we find that some bacteria become resistant to antibiotics.

Progress is slower than we would hope. The fundamental laws of nature have not yet been repealed. Right now, that includes human nature. With that said, it would be easier for neurobiologists to change human nature than for physicists to change the second law of thermodynamics.

Reply
Moderation Log
43Comments