"I would sooner believe that two Yankee professors would lie, than that stones would fall from heaven."

-- Thomas Jefferson, on meteors

"How would I explain the event of my left arm being replaced by a blue tentacle?  The answer is that I wouldn't.  It isn't going to happen."

-- Eliezer Yudkowsky, "A Technical Explanation of Technical Explanation"

"If a ship landed in my yard and LGMs stepped out, I’d push past their literature and try to find the cable that dropped the saucer on my roses. Lack of a cable or any significant burning to the flowers, I’d then grab a hammer and start knocking about in the ship till I was convinced that nothing said “Intel Inside.” Then when I discovered a “Flux Capacitor” type thing I would finally stop and say, “Hey, cool gadget!” Assuming the universal benevolence of the LGMs, I’d yank it out and demand from the nearest "Grey” (they are the tall nice ones), “where the hell did this come from?” Greys don’t talk, they communicate via telepathy, so I’d ignore the voice inside my head. Then stepping outside the saucer and sitting in a lawn chair, I’d throw pebbles at the aliens till I was sure they were solid. Then I’d look down at the “Flux Capacitor” and make sure it hadn’t morphed into my bird feeder. Finally, with proof in my hand and aliens sitting on my deck (they’d be offered beers, though I’ve heard that they absorb energy like a plant) I’d grab my cell phone and tell my doctor that I’m having a serious manic episode with full-blown visual hallucinations."

-- Peter K. Bertine, on the Extropian mailing list

We underestimate the power of science, and overestimate the power of personal observation.  A peer-reviewed, journal-published, replicated report is worth far more than what you see with your own eyes.  Our own eyes can deceive us.  People can fool themselves, hallucinate, and even go insane.  The controls on publication in major journals are more trustworthy than the very fabric of your brain.  If you see with your own eyes that the sky is blue, and Science says it is green, then sir, I advise that you trust in Science.

This is not what most scientists will tell you, of course; but I think it is pragmatically true.  Because in real life, what happens is that your eyes have a little malfunction and decide that the sky is green, and science will tell you that the sky is blue.

A replicated scientific report is a special kind of extraordinary claim, designed by the surrounding process to be more extraordinary evidence than simple verbal claims.  It is more extraordinary evidence because the surrounding process - and I would place a far greater premium on the replication than on the peer review, by the way - is constructed to deny entrance to claims that are in fact false.  In this way, the replicated scientific report becomes capable of overcoming greater burdens of prior improbability.

There are some burdens of prior improbability so great that simple verbal claims cannot overcome them.  I would not believe someone who claimed that their coffee was disobeying conservation of angular momentum - but I might believe the same report published in Physics Today, with at least three replications.  Who would believe in quantum mechanics if a stranger walked up to us on the street and whispered it to us?

Are there some burdens of prior improbability so great that science itself cannot overcome them?

What about the claim that 2 + 2 = 5?

What about journals that claim to publish replicated reports of ESP?

Sometimes, even claims deliberately constructed to be extraordinary evidence end up just not being extraordinary enough.