Comment Permalink

9eB13y63

The reason that the villagers didn't trust the boy that he didn't have a track record. One reason we don't trust people who are loudly proclaiming certain kinds of doom is that they don't have a track record of accurately predicting things (e.g. Heaven's Gate), and that's an inherently important aspect of the phenomenon this post is describing. If the child had accurately predicted wolves in the past, real world villagers would have paid attention to a 15% warning.

The post is suggesting that certain kinds of risks have low probability, and the predictors don't have a track record of success because it's impossible, but that they have other lines of evidence that can be used to justify their probability estimates. In the case of "nuclear war that hasn't happened despite the scares" the evidence is events like the Cuban missile crisis or Petrov Day. But in the parable, it isn't established that the child has good arguments to justify 5% or 15% wolf appearance rates.

See in context

140 The Parable of the Boy Who Cried 5% Chance of Wolf

by KatWoods

15th Aug 2022

2 min read

140

Epistemic status: a parable making a moderately strong claim about statistics

Once upon a time, there was a boy who cried "there's a 5% chance there's a wolf!"

The villagers came running, saw no wolf, and said "He said there was a wolf and there was not. Thus his probabilities are wrong and he's an alarmist."

On the second day, the boy heard some rustling in the bushes and cried "there's a 5% chance there's a wolf!"

Some villagers ran out and some did not.

There was no wolf.

The wolf-skeptics who stayed in bed felt smug.

"That boy is always saying there is a wolf, but there isn't."

"I didn't say there was a wolf!" cried the boy. "I was estimating the probability at low, but high enough. A false alarm is much less costly than a missed detection when it comes to dying! The expected value is good!"

The villagers didn't understand the boy and ignored him.

On the third day, the boy heard some sounds he couldn't identify but seemed wolf-y. "There's a 5% chance there's a wolf!" he cried.

No villagers came.

It was a wolf.

They were all eaten.

Because the villagers did not think probabilistically.

The moral of the story is that we should expect to have a large number of false alarms before a catastrophe hits and that is not strong evidence against impending but improbable catastrophe.

Each time somebody put a low but high enough probability on a pandemic being about to start, they weren't wrong when it didn't pan out. H1N1 and SARS and so forth didn't become global pandemics. But they could have. They had a low probability, but high enough to raise alarms.

The problem is that people then thought to themselves "Look! People freaked out about those last ones and it was fine, so people are terrible at predictions and alarmist and we shouldn't worry about pandemics"

And then COVID-19 happened.

This will happen again for other things.

People will be raising the alarm about something, and in the media, the nuanced thinking about probabilities will be washed out.

You'll hear people saying that X will definitely fuck everything up very soon.

And it doesn't.

And when the catastrophe doesn't happen, don't over-update.

Don't say, "They cried wolf before and nothing happened, thus they are no longer credible."

Say "I wonder what probability they or I should put on it? Is that high enough to set up the proper precautions?"

When somebody says that nuclear war hasn't happened yet despite all the scares, when somebody reminds you about the AI winter where nothing was happening in it despite all the hype, remember the boy who cried a 5% chance of wolf.

Originally posted on my Twitter and personal blog.

Reminder that if this reaches 35 upvotes, you can listen to this post on your podcast player using the Nonlinear Library.

Cross-posted

Parables & FablesWorld Optimization

Frontpage

140

New Comment

24 comments, sorted by

top scoring

Click to highlight new comments since: Today at 9:20 PM

[-]James_Miller3y7769

Why isn't the moral of the story "If you think statistically, take into account that most other people don't and optimize accordingly?"

[-]Shmi3y164

"Don't be a straw vulcan".

[-]DirectedEvolution3y20

I wish I could find some advice on how to do that, but it’s really hard to Google.

[-]Cédric3y98

Explaining more to align understanding.

Just translate this:

"I didn't say there was a wolf!" cried the boy. "I was estimating the probability at low, but high enough. A false alarm is much less costly than a missed detection when it comes to dying! The expected value is good!"

Into normie language and should be fine.

However it's very hard to communicate nuance at scale. I have no idea how to solve that.

[-]matto3y2515

This echos an excellent post by Dan Luu that touches on problems you face when you build larger, less legible systems that force you to deal with normalization of deviance: https://danluu.com/wat/

The action items he recommends are:

Pay attention to weak signals
Resist the urge to be unreasonably optimistic
Teach employees how to conduct emotionally uncomfortable conversations
System operators need to feel safe in speaking up
Realize that oversight and monitoring are never-ending

Most of these go against what is considered normal or comfortable though:

It's difficult to pay attention to weak signals when people build awful attention traps (eg. tiktok, youtube, etc.)
People are commonly [overconfident](https://www.lesswrong.com/posts/ybYBCK9D7MZCcdArB/how-to-measure-anything#Step_2__Determine_what_you_know).
Uncomfortable conversations are uncomfortable. In certain cultures, it's better to straight out lie rather than deliver bad news.
Few organizations have set up channels for upwards communication where front line employees can flag problems. It's better to not rock the boat.
Constant oversight and monitoring are mentally and physically draining. These are also the easiest activities to cut from a budget because they're not visible until an accident happens.

What the boy should have done is establish an organization (Wolf-Spotters) whose responsibility is monitoring for signs of wolves. This organization could be staffed by professional or volunteer observers. They need to do periodic trainings and live-action drills (perhaps using a wolf suit). To the fund this, the boy should have first created a PR campaign to make people aware of the cost of unspotted wolves (death), then use that to get support of some local politicians.

(It's basically a fire department).

If the boy was open to using the dark arts, he could have executed a false flag wolf attack. That would incentivize local politicians to support his cause.

[-]MSRayne3y40

This gives me an idea... I wonder if there might be some way to use prediction markets to make safety, maintenance, monitoring etc short-term profitable for corporations (and maybe even governments), thereby incentivizing them to do it more and not cut costs?

The reason stuff like that gets cut when nothing is going on, is that it seems to be, in the short term, nothing but a drain of money, even though in the long term it's saving money - so is there some way to make those long term gains available now so that they can be optimized for?

[-]Dagon3y142

Awesome line of thinking, and absolutely believable that most people don't think numerically very well.

However, it's also believable that there's an equilibrium effect here. BECAUSE this is so unusual, and there's no particular reason to think this boy is better at estimating the risk of wolves than the median resident, it's easy to just ignore them. This is especially true when it's a hassle to check for wolves and they've never needed to before.

If the boy were socially smart, he'd have separated the warning into two parts, each with justificatin: "There's a 5% chance a wolf is here" and "5% is risky enough that you should do something about it". And here's how to check my reasoning.

Since villagers aren't homogeneous, the boy just needs to convince a few people with status on the right dimensions, who need to convince those with status generally, who will convince everyone else. That's pretty tenuous, so a lot of villages get eaten. Humans are, indeed, dumb.

[-]Daniel V3y88

One quibble, there was a little bait and switch from someone with a well-calibrated model whose calibration just hasn't been well-evidenced, to...

You'll hear people saying that X will definitely fuck everything up very soon.
And it doesn't.
And when the catastrophe doesn't happen, don't over-update.
Don't say, "They cried wolf before and nothing happened, thus they are no longer credible."

These people ARE no longer credible as they are not estimating 5% chances but 95% chances, and the lack of an event, rather than being consistent with their model, is inconsistent with their model.

Your point is still well-taken, and I think the switch is a natural reflex given the infrequency of pundits attempting to make well-calibrated or even probabilistic judgments. For example, it has been noticeable to me to see Jamie Dimon publicly assigning probabilities to differing recession/not-recession severity bins rather than just sticking to the usual binary statements often seen in that space.

[-]Ben Amitay3y75

Hypothesis: This may be not just people being ignorant about probability. This may be people being unconsciously good at game theory. If the boy could be motivated with cross-entropy-loss worth of cookies, it would be another thing. If he can simply get the microphone and yell a number that would be enough for inciting the intended action but low enough to have practically no accountability - it may make more sense for society to actively make the "crying wolf" more binary. It doesn't penalise predication error, but yelling stuff into the microphone.

[-]Sherrinford3y62

This may be called something like the "low-probability forecaster reputation problem".

People in the village were not necessarily wrong ex-ante to dismiss the warnings. They of course should not update much about forecasting abilities on not seeing a wolf after low-probability warnings. But they cannot know the accuracy of the forecasts. And if someone is an expert in certain low-probability events and nothing else, you cannot judge ability a lot based on the few forecasts. You need to examine the models used for forecasting, but what tells you that is a good use of your time? There are crackpots everywhere.

Therefore, people try to get bets or forecasts for things that are implied by the lp forecaster's model but happen more often and earlier. If your model implies big grey wolves for tonight with 15% chance, should we already see dead sheep by noon? If your model implies war between states x and y next year, what should we see next month that would not be expected by people with different models?

If you really specialize in forecasting low-probability events only and cannot show any evidence to show your model is right, then that is a tragedy. But it is understandable that people do not update based on your forecast.

[-]Viliam3y62

The wolf-skeptics who stayed in bed felt smug.

If you perceive life as a zero-sum game, this is the correct move. If there is no wolf, the wolf-skeptics get higher status than their neighbors. If there is a wolf... everyone is eaten equally, no penalty for being a wolf-skeptic.

[-]Slider3y50

So what is the point where its correct to bin the sensor as alarmist?

Pondering this I lean that the main takeaway is that if you "cry" anything then no amount of what you cry can make it effective for low-probablity events. Crying "there is a 0.001% chance of wolf" is in practise going to be nearly equivalent to communicating about 20% chance of wolf. But if you say "there is a 0.001% chance of wolf" maybe that works for 5% and if you whisper you can get under 1%.

The second is that people can disagree what prepardness thresholds are proper. Say that the village runs out 10 times in responce to 5% reports and then can only harvest half the crop for being exhausted from running all the time killing 20% people to starvation. As a wolf-expert I am not likely to be the most well versed on food shortage events. So when I make judgement calls on where it is better to be safe than sorry I am likely to be atleast somewhat ignorant. So even an honest private trade-off might not be be socially optimal. The problem disappears if the village thinks the fortifications for nothing are worth the effort.

[-]Dagon3y50

The correct approach as a villager is to take the sensor as bayesean evidence. What is your prior that there is a wolf nearby right now? What is the probability that the boy would cry 5% wolf when there is or is not a wolf (hint: it's probably not 5%/95%, and doesn't even need to add up to 1)?

In villages where wolves are common, it probably shouldn't change your estimate at all. In villages where wolves have never been a problem, it may move your estimate even higher than 5% (because the kid can be wrong in either direction, and someone even bothering to point it out is pretty unusual at all).

[-]Shmi3y45

Small probabilities are hard to calculate accurately. How did the boy know that it was a 5% chance and not 0.001% chance?

[-]jmh3y42

While not a good direct answer to your question I think an old economics article, The Origins of Predictable Behavior, (Ron Heiner, AER 1984 I think) looks into low probability, high cost situations to suggest such setting will tent to support the emergence of social rules and norms to mitigate the risks.

[-]Alex Vermillion3y3-3

https://www.lesswrong.com/posts/neQ7eXuaXpiYw7SBy/the-least-convenient-possible-world

https://parrhesia.substack.com/p/dont-fight-the-hypothetical

[-]Shmi3y93

That misses my point, which is that trusting the judgment of someone who is proclaiming opaquely calculated but accurate estimates of low probability events without an extremely good calibration track record is a bad idea.

[-]Alex Vermillion3y20

The point of (me linking) "Don't fight the hypothetical" is that the author of the thought experiment could come here and comment "The boy has incredibly calibration about coyotes and other beasts, made using old predictions and cross validated with reserved old predictions, but the others in the village don't know this and won't listen now that he's had one false alarm" and it would be completely uninteresting to discuss.

[-]9eB13y63

[-]JBlack3y3-1

Maybe the real moral is that you shouldn't set up a village in an area where it seems even remotely plausible that there are wolves so enormous that they can eat everyone in a single night.

Or at least if you do, to set up defenses that are less flimsy than "have a boy run in yelling".

[-]mako yass3y60

Wolves sometimes kill more than they need, actually. It's quite strange. So they could be normal-sized wolves. And I'm imagining this to be a population of conservationists who aren't interested in taking them out of the local ecosystem.

I'm trying to figure out the worldbuilding logic of "they didn't come so they all got eaten". What do they do when they come? Why would they be less likely to get eaten if they don't do it? And also, how does the boy only have a 5% probability?

Okay so maybe the boy sees the wolf from a distance, on a particular bridge or in a particular clearing, and can't know whether the wolf is coming towards the village. There's a 5% chance that it will. He can't stay and make sure, because then maybe he just gets caught by the wolf, and he has to run to deliver the message as soon as possible.

Let's say it's a very tough wolf and it can only be defeated with a tank. The village has only one wolf-proof tank. and it takes a while to assemble the tank crew and get its engine started, and the fuel is expensive. Once in the village, the wolf would smell the tank from its fuel, locate it, and keep anyone from getting near it, if the wolf gets to the village before the tank is started then it's over.

And why is this task left to a child? Children are generally more concerned by and obsessed with monsters than adults. They know that wolf-crying is relatively safe, and they desperately want to witness the monster (this is the whole appeal of the genre of horror), so they take that job. They also have nothing better to do.

[-]mruwnik3y40

The strange thing is that the wolf killed all the people, as wolves very seldom attack people, and when they do, they tend to do so in starved packs, be rabid or be habituated to humans. Since this is a single wolf attacking, it seems that the most effective use of resources would be an all forest campaign to get rid of rabies, as I assume the village doesn't have a wolf breeding project going.

Leaving the task to a child can make a lot of sense precisely because of how scared they are - they desperately want to witness the monster (hence will work hard to find it), but also are terrified of it, so will be incentivized to make sure that those who can protect them will know of the danger - a grown up might simply ignore wolf signs, seeing as they could have been made by a big dog or something, and they have to finish hoeing the field today. Children are also more disposable than highly trained grown ups, so can also be used as a diversion in a pinch.

As to why they got eaten because they didn't come, it's a lot easier to pick off the villagers one by one when they aren't wary, than to fend of a whole angry band of villagers actively hunting for it.

[-]mako yass3y20

Can you explain how it picks them off one by one? I mean, how large a group do you need to pick off a wolf and wouldn't most people be close to being in a group of that size naturally as a result of uh having a town.

[-]mruwnik3y12

Seeing as it's been established that this wolf is capable of homicide, it stands to reason that killing 1 person is easier that killing more. To quote the Witcher "Every swordsman’s an arse when the enemy’s not sparse" (the original sounds much better :/). The context is a village. I'm assuming that they're not that advanced technologically, as then they could use automated anti-wolf systems. Wolves tend to prefer forests and mountains, mainly because there are few humans there. So it stands to reason that this is a rather remote village where the denizens are more of the farmer/forester types. This kind of life involves a lot of time spent outside doing various chores, like tending to gardens, or chopping wood. One could posit that each household is surrounded by fences etc., but that actually doesn't help that much, as a basic wolf enclosure needs a fence that is at least 8 feet high. While this is very much culturally dependent, I have the impression that most fences and walls tend to be lower, as they're more of a marker than an impenetrable barrier. So it seems reasonable to assume that this obviously superior canine would both easily surmount any such obstacles and potentially even use them to hem in and dispatch solitary individuals.

Once the inhabitants are alerted (and actually believe the warnings!) then yes, they will probably group up for safety. At that point they should have the advantage due to their (I'm guessing) superior intellect and ability to use tools and close doors. So it's in the wolfs interest to prevent this happening. Now I'm starting to wonder whether the initial startling of the boy wasn't a cunning ruse to lower the villagers' vigilance?

It does seem somewhat improbable for the wolf to finish off the whole population. That would require either a very small population (e.g. 2-3 families), a very stupid group of people (who ignore the conspicuous proceeding disappearances) or a very enterprising wolf. One potential solution would be to attack at night when everyone is sleeping. Though here the afore mentioned doors would probably be a hindrance. That being said, it's not unheard of. Unless the doors were locked.

Nothing like adding a whole load of extra details to make the story seem more probable :D

Moderation Log