This reminds me of the classic industrial accident involving a large, pressurised storage tank. There is a man-sized door to allow access for maintenance and a pressure gauge. The maintenance man is supposed to wait for the pressure to fall to zero before he undoes the heavy steel latches. It is a big tank and he gets bored with waiting for the pressure to vent. The gauge says one pound per square inch. One pound doesn't sound like much so the man undoes the latches. Since the force is per square inch it is several hundred times larger than expected. The heavy door flies open irresistibly and kills the man.
I'm not seeing how the parable helps one be less wrong in real life. In the parable the victim has seen a dog taken by the dragon. If the maintenance man had seen an apprentice crushed in an earlier similar accident the experience would scar him mentally and he would always be wary of pressure vessels. I'm worrying that the parable is cruder than the problems we face in real life.
I don't know more than I've already said about pressure vessel accidents. Is there an underlying problem of crying wolf; too many warning messages obscure the ones that are really matters of life and death? Is it a matter of incentives; the manager gets a bonus if he encourages the maintenance team to work quickly, but doesn't go to jail when cutting corners leads to a fatal accident? Is it a matter of education; the maintenance man just didn't get pressure? Is it a matter of labeling; why not label the gauge by the door with the force per door area? Is it matter of class; the safety officer is middle class, the maintenance man is working class, the working class distrust the middle class and don't much believe what they say?
Is there an underlying problem of crying wolf; too many warning messages obscure the ones that are really matters of life and death?
This is certainly an enormous problem for interface design in general for many systems where there is some element of danger. The classic "needle tipping into the red" is an old and brilliant solution for some kinds of gauges - an analogue meter where you can see the reading tipping toward a brightly marked "danger zone", usually with a 'safe' zone and an intermediate zone also marked, has surely preven...
Related to Belief In Belief
Suppose that a neighbor comes to you one day and tells you “There’s a dragon in my garage!” Since all of us have been through this before at some point or another, you may be inclined to save time and ask “Is the dragon by any chance invisible, inaudible, intangible, and does it convert oxygen to carbon dioxide when it breathes?”
The neighbor, however, is a scientific minded fellow and responds “Yes, yes, no, and maybe, I haven’t checked. This is an idea with testable consequences. If I try to touch the dragon it gets out of the way, but it leaves footprints in flour when I sprinkle it on the garage floor, and whenever it gets hungry, it comes out of my garage and eats a nearby animal. It always chooses something weighing over thirty pounds, and you can see the animals get snatched up and mangled to a pulp in its invisible jaws. It’s actually pretty horrible. You may have noticed that there have been fewer dogs around the neighborhood lately.”
This triggers a tremendous number of your skepticism filters, and so the only thing you can think of to say is “I think I’m going to need to see this.”
“Of course,” replies the neighbor, and he sets off across the street, opens the garage door, and is promptly eaten by the invisible dragon.
Tragic though it is, his death provides a useful lesson. He clearly believed that there was an invisible dragon in his garage, and he was willing to stick his neck out and make predictions based on it. However, he hadn’t internalized the idea that there was a dragon in his garage, otherwise he would have stayed the hell away to avoid being eaten. Humans have a fairly general weakness at internalizing beliefs when we don’t have to come face to face with their immediate consequences on a regular basis.
You might believe, for example, that starvation is the single greatest burden on humanity, and that giving money to charities that aid starving children in underdeveloped countries has higher utility than any other use of your surplus funds. You might even be able to make predictions based on that belief. But if you see a shirt you really like that’s on sale, you’re almost certainly not going to think “How many people will go hungry if I buy this who I could have fed?” It’s not a weakness of willpower that causes you to choose the shirt over the starving children, they simply don’t impinge on your consciousness at that level.
When you consider if you really, properly hold a belief, it’s worth asking not only how it controls your anticipations, but whether your actions make sense in light of a gut-level acceptance of its truth. Do you merely expect to see footprints in flour, or do you move out of the house to avoid being eaten?