The Modesty Argument states that when two or more human beings have common knowledge that they disagree about a question of simple fact, they should each adjust their probability estimates in the direction of the others'.  (For example, they might adopt the common mean of their probability distributions.  If we use the logarithmic scoring rule, then the score of the average of a set of probability distributions is better than the average of the scores of the individual distributions, by Jensen's inequality.)

Put more simply:  When you disagree with someone, even after talking over your reasons, the Modesty Argument claims that you should each adjust your probability estimates toward the other's, and keep doing this until you agree.  The Modesty Argument is inspired by Aumann's Agreement Theorem, a very famous and oft-generalized result which shows that genuine Bayesians literally cannot agree to disagree; if genuine Bayesians have common knowledge of their individual probability estimates, they must all have the same probability estimate.  ("Common knowledge" means that I know you disagree, you know I know you disagree, etc.)

I've always been suspicious of the Modesty Argument.  It's been a long-running debate between myself and Robin Hanson.

"I would sooner believe that two Yankee professors would lie, than that stones would fall from heaven."

-- Thomas Jefferson, on meteors

"How would I explain the event of my left arm being replaced by a blue tentacle?  The answer is that I wouldn't.  It isn't going to happen."

-- Eliezer Yudkowsky, "A Technical Explanation of Technical Explanation"

"If a ship landed in my yard and LGMs stepped out, I’d push past their literature and try to find the cable that dropped the saucer on my roses. Lack of a cable or any significant burning to the flowers, I’d then grab a hammer and start knocking about in the ship till I was convinced that nothing said “Intel Inside.” Then when I discovered a “Flux Capacitor” type thing I would finally stop and say, “Hey, cool gadget!” Assuming the universal benevolence of the LGMs, I’d yank it out and demand from the nearest "Grey” (they are the tall nice ones), “where the hell did this come from?” Greys don’t talk, they communicate via telepathy, so I’d ignore the voice inside my head. Then stepping outside the saucer and sitting in a lawn chair, I’d throw pebbles at the aliens till I was sure they were solid. Then I’d look down at the “Flux Capacitor” and make sure it hadn’t morphed into my bird feeder. Finally, with proof in my hand and aliens sitting on my deck (they’d be offered beers, though I’ve heard that they absorb energy like a plant) I’d grab my cell phone and tell my doctor that I’m having a serious manic episode with full-blown visual hallucinations."

-- Peter K. Bertine, on the Extropian mailing list

I've always been annoyed at the notion that the bias-variance decomposition tells us something about modesty or Philosophical Majoritarianism.  For example, Scott Page rearranges the equation to get what he calls the Diversity Prediction Theorem:

Collective Error = Average Individual Error - Prediction Diversity

I think I've finally come up with a nice, mathematical way to drive a stake through the heart of that concept and bury it beneath a crossroads at midnight, though I fully expect that it shall someday rise again and shamble forth to eat the brains of the living.

Today my coworker Marcello pointed out to me an interesting anti-majoritarian effect.  There are three major interpretations of probability: the "subjective" view of probabilities as measuring the uncertainty of agents, the "propensity" view of probabilities as chances inherent within objects, and the "frequentist" view of probabilities as the limiting value of long-run frequencies.  I was remarking on how odd it was that frequentism, the predominant view in mainstream statistics, is the worst of the three major alternatives (in my view, you have to presume either uncertainty or propensity in order to talk about the limiting frequency of events that have not yet happened).

And Marcello said something along the lines of, "Well, of course.  If anything were worse than frequentism, it wouldn't be there."  I said, "What?"  And Marcello said, "Like the saying that Mac users have, 'If Macs really were worse than Windows PCs, no one would use them.'"

At this point the light bulb went on over my head - a fluorescent light bulb - and I understood what Marcello was saying: an alternative to frequentism that was even worse than frequentism would have dropped off the radar screens long ago.  You can survive by being popular, or by being superior, but alternatives that are neither popular nor superior quickly go extinct.

Followup to:  Uncritical Supercriticality

Early studiers of cults were surprised to discover than when cults receive a major shock—a prophecy fails to come true, a moral flaw of the founder is revealed—they often come back stronger than before, with increased belief and fanaticism.  The Jehovah's Witnesses placed Armageddon in 1975, based on Biblical calculations; 1975 has come and passed.  The Unarian cult, still going strong today, survived the nonappearance of an intergalactic spacefleet on September 27, 1975.  (The Wikipedia article on Unarianism mentions a failed prophecy in 2001, but makes no mention of the earlier failure in 1975, interestingly enough.)

Why would a group belief become stronger after encountering crushing counterevidence?

Followup to:  Asch's Conformity Experiment

The scary thing about Asch's conformity experiments is that you can get many people to say black is white, if you put them in a room full of other people saying the same thing.  The hopeful thing about Asch's conformity experiments is that a single dissenter tremendously drove down the rate of conformity, even if the dissenter was only giving a different wrong answer.  And the wearisome thing is that dissent was not learned over the course of the experiment—when the single dissenter started siding with the group, rates of conformity rose back up.

Being a voice of dissent can bring real benefits to the group.  But it also (famously) has a cost.  And then you have to keep it up.  Plus you could be wrong.

Followup to:  The Modesty Argument, The "Outside the Box" Box, Asch's Conformity Experiment

Asch's conformity experiment showed that the presence of a single dissenter tremendously reduced the incidence of "conforming" wrong answers.  Individualism is easy, experiment shows, when you have company in your defiance.  Every other subject in the room, except one, says that black is white.  You become the second person to say that black is black.  And it feels glorious: the two of you, lonely and defiant rebels, against the world!  (Followup interviews showed that subjects in the one-dissenter condition expressed strong feelings of camaraderie with the dissenter—though, of course, they didn't think the presence of the dissenter had influenced their own nonconformity.)

But you can only join the rebellion, after someone, somewhere, becomes the first to rebel.  Someone has to say that black is black after hearing everyone else, one after the other, say that black is white.  And that—experiment shows—is a lot harder.

Lonely dissent doesn't feel like going to school dressed in black.  It feels like going to school wearing a clown suit.

That's the difference between joining the rebellion and leaving the pack.

Followup to:  Every Cause Wants To Be A Cult, Lonely Dissent

In the modern world, joining a cult is probably one of the worse things that can happen to you.  The best-case scenario is that you'll end up in a group of sincere but deluded people, making an honest mistake but otherwise well-behaved, and you'll spend a lot of time and money but end up with nothing to show.  Actually, that could describe any failed Silicon Valley startup.  Which is supposed to be a hell of a harrowing experience, come to think.  So yes, very scary.

Real cults are vastly worse.  "Love bombing" as a recruitment technique, targeted at people going through a personal crisis.  Sleep deprivation.  Induced fatigue from hard labor.  Distant communes to isolate the recruit from friends and family.  Daily meetings to confess impure thoughts.  It's not unusual for cults to take all the recruit's money—life savings plus weekly paycheck—forcing them to depend on the cult for food and clothing.  Starvation as a punishment for disobedience.  Serious brainwashing and serious harm.

With all that taken into account, I should probably sympathize more with people who are terribly nervous, embarking on some odd-seeming endeavor, that they might be joining a cult.  It should not grate on my nerves.  Which it does.

Followup to:  A Premature Word on AI, The Modesty Argument

Once, during the year I was working with Marcello, I passed by a math book he was reading, left open on the table.  One formula caught my eye (why?); and I thought for a moment and said, "This... doesn't look like it can be right..."

Then we had to prove it couldn't be right.

Why prove it?  It looked wrong; why take the time for proof?

Because it was in a math book.  By presumption, when someone publishes a book, they run it past some editors and double-check their own work; then all the readers get a chance to check it, too.  There might have been something we missed.

But in this case, there wasn't.  It was a misprinted standard formula, off by one.

I once found an error in Judea Pearl's Causality - not just a misprint, but an actual error invalidating a conclusion in the text.  I double and triple-checked, the best I was able, and then sent an email to Pearl describing what I thought the error was, and what I thought was the correct answer.  Pearl confirmed the error, but he said my answer wasn't right either, for reasons I didn't understand and that I'd have to have gone back and done some rereading and analysis to follow.  I had other stuff to do at the time, unfortunately, and couldn't expend the energy.  And by the time Pearl posted an expanded explanation to the website, I'd forgotten the original details of the problem...  Okay, so my improved answer was wrong.

Why take Pearl's word for it?  He'd gotten the original problem wrong, and I'd caught him on it - why trust his second thought over mine?

Followup to:  The Rhythm of Disagreement

At the age of 15, a year before I knew what a "Singularity" was, I had learned about evolutionary psychology.  Even from that beginning, it was apparent to me that people talked about "disagreement" as a matter of tribal status, processing it with the part of their brain that assessed people's standing in the tribe.  The peculiar indignation of "How dare you disagree with Einstein?" has its origins here:  Even if the disagreer is wrong, we wouldn't apply the same emotions to an ordinary math error like "How dare you write a formula that makes e equal to 1.718?"

At the age of 15, being a Traditional Rationalist, and never having heard of Aumann or Bayes, I thought the obvious answer was, "Entirely disregard people's authority and pay attention to the arguments.  Only arguments count."

Ha ha!  How naive.