Bostrom recently noted the problem of the commons in labeling efforts "important"; each managerial player has an incentive to label their project world-shakingly important, even though this devalues the priority label as used at other times or other projects, creating positive feedback in inflated labels.

This reminds me of how my grandfather, a pioneer in quantitative genetics, regularly bemoans the need to write more and more grant proposals to maintain a constant level of funding.  It's not that the funding is drying up in his field.  But suppose there's money for 20 grants, and 21 scientists in need of grants - or one scientist who'd like to run two projects, or receive more funding for one project...  One scientist doesn't get his first grant proposal funded, so he writes another one.  His second grant proposal does get funded, which uses up a grant that could have gone to another scientist, who now also has his first grant proposal denied, and has to write and send off a second grant proposal too...

The problem here is that, while some initial level of effort is beneficial, all effort beyond that is marginally zero-sum; there's a marginal return to the individual on additional efforts, but no marginal return to the group.  If there are 20 grants, then ultimately only 20 grant proposals are going to be funded.  No matter how many grant proposals anyone writes, the total funding available remains the same.  Everyone would be better off if everyone agreed to write only one grant proposal.  But in this case, there wouldn't be much competition for any given grant, and the rewards for writing another two or three grant proposals would be huge... until everyone else started doing the same thing.

In L. Sprague de Camp's fantasy story The Incomplete Enchanter (which set the mold for the many imitations that followed), the hero, Harold Shea, is transported from our own universe into the universe of Norse mythology.  This world is based on magic rather than technology; so naturally, when Our Hero tries to light a fire with a match brought along from Earth, the match fails to strike.

I realize it was only a fantasy story, but... how do I put this...

No.

Followup to:  Universal Fire

Antoine-Laurent de Lavoisier discovered that breathing (respiration) and fire (combustion) operated on the same principle.  It was one of the most startling unifications in the history of science, for it brought together the mundane realm of matter and the sacred realm of life, which humans had divided into separate magisteria.

The first great simplification was that of Isaac Newton, who unified the course of the planets with the trajectory of a falling apple.  The shock of this discovery was greater by far than Lavoisier's.  It wasn't just that Newton had dared to unify the Earthly realm of base matter with the obviously different and sacred celestial realm, once thought to be the abode of the gods.  Newton's discovery gave rise to the notion of a universal law, one that is the same everywhere and everywhen, with literally zero exceptions.

In Think Like Reality, I put forth the astonishing and controversial proposition that when human intuitions disagree with a fact, we need to either disprove the "fact" in question, or try to reshape the intuition.  (Well, it wouldn't have been so controversial, but like a fool I picked quantum mechanics to illustrate the point.  Never use quantum mechanics as an example of anything.)  Probability theory says that a model which is consistently surprised on the data is probably not a very good model.

Matt Shulman pointed out in personal conversation that, in practice, we may want to be wary of people who don't appear surprised by surprising-seeming data.  Some people affect to be unsurprised because it is a fakeable signal of competence.  Well, a lot of things that good rationalists will do - such as appearing skeptical and appearing to take other people's opinions into account - are also fakeable signals of competence.  But, in practice, Matt's point is still well-taken.

One of the great weaknesses of Science is this mistaken idea that if an experiment contradicts the dominant theory, we should throw out the theory instead of the experiment.

Experiments can go awry.  They can contain design flaws. They can be deliberately corrupted.  They can be unconsciously corrupted.  They can be selectively reported.  Most of all, 1 time in 20 they can be "statistically significant" by sheer coincidence, and there are a lot of experiments out there.

Unfortunately, Science has this notion that you can never go against an honestly obtained experimental result.  So, when someone obtains an experimental result that contradicts the standard model, researchers are faced with a dilemma for resolving their cognitive dissonance: they either have to immediately throw away the standard model, or else attack the experiment - accuse the researchers of dishonesty, or flawed design, or conflict of interest...

Someone once presented me with a new study on the effects of intercessory prayer (that is, people praying for patients who are not told about the prayer), which showed 50% of the prayed-for patients achieving success at in-vitro fertilization, versus 25% of the control group.  I liked this claim.  It had a nice large effect size.  Claims of blatant impossible effects are much more pleasant to deal with than claims of small impossible effects that are "statistically significant".

So I cheerfully said:  "I defy the data."

This excerpt from Meyers's Exploring Social Psychology is worth reading in entirety.  Cullen Murphy, editor of The Atlantic, said that the social sciences turn up "no ideas or conclusions that can't be found in [any] encyclopedia of quotations... Day after day social scientists go out into the world.  Day after day they discover that people's behavior is pretty much what you'd expect."

Of course, the "expectation" is all hindsight.  (Hindsight bias:  Subjects who know the actual answer to a question assign much higher probabilities they "would have" guessed for that answer, compared to subjects who must guess without knowing the answer.)

The historian Arthur Schlesinger, Jr. dismissed scientific studies of WWII soldiers' experiences as "ponderous demonstrations" of common sense.  For example:

1. Better educated soldiers suffered more adjustment problems than less educated soldiers. (Intellectuals were less prepared for battle stresses than street-smart people.)  
2. Southern soldiers coped better with the hot South Sea Island climate than Northern soldiers. (Southerners are more accustomed to hot weather.)  
3. White privates were more eager to be promoted to noncommissioned officers than Black privates. (Years of oppression take a toll on achievement motivation.)  
4. Southern Blacks preferred Southern to Northern White officers (because Southern officers were more experienced and skilled in interacting with Blacks).  
5. As long as the fighting continued, soldiers were more eager to return home than after the war ended. (During the fighting, soldiers knew they were in mortal danger.)  

How many of these findings do you think you could have predicted in advance?   3 out of 5?  4 out of 5?  Are there any cases where you would have predicted the opposite - where your model takes a hit?  Take a moment to think before continuing...

Suppose that your good friend, the police commissioner, tells you in strictest confidence that the crime kingpin of your city is Wulky Wilkinsen.  As a rationalist, are you licensed to believe this statement?  Put it this way: if you go ahead and mess around with Wulky's teenage daughter, I'd call you foolhardy.  Since it is prudent to act as if Wulky has a substantially higher-than-default probability of being a crime boss, the police commissioner's statement must have been strong Bayesian evidence.

Our legal system will not imprison Wulky on the basis of the police commissioner's statement.  It is not admissible as legal evidence.  Maybe if you locked up every person accused of being a crime boss by a police commissioner, you'd initially catch a lot of crime bosses, plus some people that a police commissioner didn't like.  Power tends to corrupt: over time, you'd catch fewer and fewer real crime bosses (who would go to greater lengths to ensure anonymity) and more and more innocent victims (unrestrained power attracts corruption like honey attracts flies).

This does not mean that the police commissioner's statement is not rational evidence.  It still has a lopsided likelihood ratio, and you'd still be a fool to mess with Wulky's teenager daughter.  But on a social level, in pursuit of a social goal, we deliberately define "legal evidence" to include only particular kinds of evidence, such as the police commissioner's own observations on the night of April 4th.  All legal evidence should ideally be rational evidence, but not the other way around.  We impose special, strong, additional standards before we anoint rational evidence as "legal evidence".

As I write this sentence at 8:33pm, Pacific time, on August 18th 2007, I am wearing white socks.  As a rationalist, are you licensed to believe the previous statement?  Yes.  Could I testify to it in court?  Yes.  Is it a scientific statement?  No, because there is no experiment you can perform yourself to verify it.  Science is made up of generalizations which apply to many particular instances, so that you can run new real-world experiments which test the generalization, and thereby verify for yourself that the generalization is true, without having to trust anyone's authority.  Science is the publicly reproducible knowledge of humankind.

Like a court system, science as a social process is made up of fallible humans.  We want a protected pool of beliefs that are especially reliable.  And we want social rules that encourage the generation of such knowledge.  So we impose special, strong, additional standards before we canonize rational knowledge as "scientific knowledge", adding it to the protected belief pool.

Prerequisite / Read this first:  Scientific Evidence, Legal Evidence, Rational Evidence

Consider the statement "It is physically possible to construct diamondoid nanomachines which repair biological cells."  Some people will tell you that molecular nanotechnology is "pseudoscience" because it has not been verified by experiment - no one has ever seen a nanofactory, so how can believing in their possibility be scientific?

Drexler, I think, would reply that his extrapolations of diamondoid nanomachines are based on standard physics, which is to say, scientific generalizations. Therefore, if you say that nanomachines cannot work, you must be inventing new physics.  Or to put it more sharply:  If you say that a simulation of a molecular gear is inaccurate, if you claim that atoms thus configured would behave differently from depicted, then either you know a flaw in the simulation algorithm or you're inventing your own laws of physics.

Once upon a time, there was an instructor who taught physics students.  One day she called them into her class, and showed them a wide, square plate of metal, next to a hot radiator.  The students each put their hand on the plate, and found the side next to the radiator cool, and the distant side warm.  And the instructor said, Why do you think this happens?  Some students guessed convection of air currents, and others guessed strange metals in the plate.  They devised many creative explanations, none stooping so low as to say "I don't know" or "This seems impossible."

And the answer was that before the students entered the room, the instructor turned the plate around.

Consider the student who frantically stammers, "Eh, maybe because of the heat conduction and so?"  I ask: is this answer a proper belief?  The words are easily enough professed - said in a loud, emphatic voice.  But do the words actually control anticipation?

Ponder that innocent little phrase, "because of", which comes before "heat conduction".  Ponder some of the other things we could put after it.  We could say, for example, "Because of phlogiston", or "Because of magic."

Prerequisites:  Fake Explanations, Belief As Attire

The preview for the X-Men movie has a voice-over saying:  "In every human being... there is the genetic code... for mutation."  Apparently you can acquire all sorts of neat abilities by mutation.  The mutant Storm, for example, has the ability to throw lightning bolts. 

I beg you, dear reader, to consider the biological machinery necessary to generate electricity; the biological adaptations necessary to avoid being harmed by electricity; and the cognitive circuitry required for finely tuned control of lightning bolts.  If we actually observed any organism acquiring these abilities in one generation, as the result of mutation, it would outright falsify the neo-Darwinian model of natural selection.  It would be worse than finding rabbit fossils in the pre-Cambrian.  If evolutionary theory could actually stretch to cover Storm, it would be able to explain anything, and we all know what that would imply.

The X-Men comics use terms like "evolution", "mutation", and "genetic code", purely to place themselves in what they conceive to be the literary genre of science.  The part that scares me is wondering how many people, especially in the media, understand science only as a literary genre.