The paper "Strong Inference" by John R. Platt is a meta-analysis of scientific methodology published in Science in 1964. It starts off with a wonderfully aggressive claim:

Scientists these days tend to keep up a polite fiction that all science is equal.

The paper starts out by observing that some scientific fields progress much more rapidly than others. Why should this be?

I think the usual explanations we tend to think of - such as the tractability of the subject, or the quality or education of the men drawn into it, or the size of the research contracts - are important but inadequate... Rapidly moving fields are fields where a particular method of doing scientific research is systematically used and taught, an accumulative method of inductive inference that is so effective that I think it should be given the name "Strong Inference".

The definition of Strong Inference, according to Platt, is the formal, explicit, and regular adherence to the following procedure:

1. Devise alternative hypotheses;
2. Devise a crucial experiment (or several of them), with alternative possible outcomes, each of which will, as nearly as possible, exclude one or more of the hypotheses;
3. Carry out the experiment so as to get a clean result;
4. (Goto 1) - Recycle the procedure, making subhypotheses or sequential hypotheses to refine the problems that remain; and so on.

This seems like a simple restatement of the scientific method. Why does Platt bother to tell us something we already know?

The reason is that many of us have forgotten it. Science is now an everyday business. Equipment, calculations, lectures become ends in themselves. How many of us write down our alternatives and crucial experiments every day, focusing on the exclusion of a hypothesis?

Platt gives us some nice historical anecdotes of strong inference at work. One is from high-energy physics:

[One of the crucial experiments] was thought of one evening at suppertime: by midnight they had arranged the apparatus for it, and by 4am they had picked up the predicted pulses showing the non-conservation of parity.

The paper emphasizes the importance of systematicity and rigor over raw intellectual firepower. Roentgen, proceeding systematically, shows us the meaning of haste:

Within 8 weeks after the discovery of X-rays, Roentgen had identified 17 of their major properties.

Later, Platt argues against the overuse of mathematics:

I think that anyone who asks the question about scientific effectiveness will also conclude that much of the mathematicizing in physics and chemistry today is irrelevant if not misleading.

(Fast forward to the present, where we have people proving the existence of Nash equilibria in robotics and using Riemannian manifolds in computer vision, when robots can barely walk up stairs and the problem of face detection still has no convincing solution.)

One of the obstacles to hard science is that hypotheses must come into conflict, and one or the other must eventually win. This creates sociological trouble, but there's a solution:

The conflict and exclusion of alternatives that is necessary to sharp inductive inference has been all too often a conflict between men, each with his single Ruling Theory. But whenever each man begins to have multiple working hypotheses, it becomes purely a conflict between ideas.

Finally, Platt suggests that all scientists continually bear in mind The Question:

But, sir, what experiment could disprove your hypothesis?

----

Now, LWers, I am not being rhetorical, I put these questions to you sincerely: Is artificial intelligence, rightly considered, an empirical science? If not, what is it? Why doesn't AI make progress like the fields mentioned in Platt's paper? Why can't AI researchers formulate and test theories the way high-energy physicists do? Can a field which is not an empirical science ever make claims about the real world?

If you have time and inclination, try rereading my earlier post on the Compression Rate Method, especially the first part, in the light of Platt's paper.

Edited thanks to feedback from Cupholder.