Link: Strong Inference

DanB

The paper "Strong Inference" by John R. Platt is a meta-analysis of scientific methodology published in Science in 1964. It starts off with a wonderfully aggressive claim:

Scientists these days tend to keep up a polite fiction that all science is equal.

The paper starts out by observing that some scientific fields progress much more rapidly than others. Why should this be?

I think the usual explanations we tend to think of - such as the tractability of the subject, or the quality or education of the men drawn into it, or the size of the research contracts - are important but inadequate... Rapidly moving fields are fields where a particular method of doing scientific research is systematically used and taught, an accumulative method of inductive inference that is so effective that I think it should be given the name "Strong Inference".

The definition of Strong Inference, according to Platt, is the formal, explicit, and regular adherence to the following procedure:

Devise alternative hypotheses;
Devise a crucial experiment (or several of them), with alternative possible outcomes, each of which will, as nearly as possible, exclude one or more of the hypotheses;
Carry out the experiment so as to get a clean result;
(Goto 1) - Recycle the procedure, making subhypotheses or sequential hypotheses to refine the problems that remain; and so on.

This seems like a simple restatement of the scientific method. Why does Platt bother to tell us something we already know?

The reason is that many of us have forgotten it. Science is now an everyday business. Equipment, calculations, lectures become ends in themselves. How many of us write down our alternatives and crucial experiments every day, focusing on the exclusion of a hypothesis?

Platt gives us some nice historical anecdotes of strong inference at work. One is from high-energy physics:

[One of the crucial experiments] was thought of one evening at suppertime: by midnight they had arranged the apparatus for it, and by 4am they had picked up the predicted pulses showing the non-conservation of parity.

The paper emphasizes the importance of systematicity and rigor over raw intellectual firepower. Roentgen, proceeding systematically, shows us the meaning of haste:

Within 8 weeks after the discovery of X-rays, Roentgen had identified 17 of their major properties.

Later, Platt argues against the overuse of mathematics:

I think that anyone who asks the question about scientific effectiveness will also conclude that much of the mathematicizing in physics and chemistry today is irrelevant if not misleading.

(Fast forward to the present, where we have people proving the existence of Nash equilibria in robotics and using Riemannian manifolds in computer vision, when robots can barely walk up stairs and the problem of face detection still has no convincing solution.)

One of the obstacles to hard science is that hypotheses must come into conflict, and one or the other must eventually win. This creates sociological trouble, but there's a solution:

The conflict and exclusion of alternatives that is necessary to sharp inductive inference has been all too often a conflict between men, each with his single Ruling Theory. But whenever each man begins to have multiple working hypotheses, it becomes purely a conflict between ideas.

Finally, Platt suggests that all scientists continually bear in mind The Question:

But, sir, what experiment could disprove your hypothesis?

----

Now, LWers, I am not being rhetorical, I put these questions to you sincerely: Is artificial intelligence, rightly considered, an empirical science? If not, what is it? Why doesn't AI make progress like the fields mentioned in Platt's paper? Why can't AI researchers formulate and test theories the way high-energy physicists do? Can a field which is not an empirical science ever make claims about the real world?

If you have time and inclination, try rereading my earlier post on the Compression Rate Method, especially the first part, in the light of Platt's paper.

Edited thanks to feedback from Cupholder.

The paper "Strong Inference" by John R. Platt is a meta-analysis of scientific methodology published in Science in 1964. It starts off with a wonderfully aggressive claim:

Scientists these days tend to keep up a polite fiction that all science is equal.

The paper starts out by observing that some scientific fields progress much more rapidly than others. Why should this be?

I think the usual explanations we tend to think of - such as the tractability of the subject, or the quality or education of the men drawn into it, or the size of the research contracts - are important but inadequate... Rapidly moving fields are fields where a particular method of doing scientific research is systematically used and taught, an accumulative method of inductive inference that is so effective that I think it should be given the name "Strong Inference".

The definition of Strong Inference, according to Platt, is the formal, explicit, and regular adherence to the following procedure:

Devise alternative hypotheses;
Devise a crucial experiment (or several of them), with alternative possible outcomes, each of which will, as nearly as possible, exclude one or more of the hypotheses;
Carry out the experiment so as to get a clean result;
(Goto 1) - Recycle the procedure, making subhypotheses or sequential hypotheses to refine the problems that remain; and so on.

This seems like a simple restatement of the scientific method. Why does Platt bother to tell us something we already know?

The reason is that many of us have forgotten it. Science is now an everyday business. Equipment, calculations, lectures become ends in themselves. How many of us write down our alternatives and crucial experiments every day, focusing on the exclusion of a hypothesis?

Platt gives us some nice historical anecdotes of strong inference at work. One is from high-energy physics:

[One of the crucial experiments] was thought of one evening at suppertime: by midnight they had arranged the apparatus for it, and by 4am they had picked up the predicted pulses showing the non-conservation of parity.

The paper emphasizes the importance of systematicity and rigor over raw intellectual firepower. Roentgen, proceeding systematically, shows us the meaning of haste:

Within 8 weeks after the discovery of X-rays, Roentgen had identified 17 of their major properties.

Later, Platt argues against the overuse of mathematics:

I think that anyone who asks the question about scientific effectiveness will also conclude that much of the mathematicizing in physics and chemistry today is irrelevant if not misleading.

One of the obstacles to hard science is that hypotheses must come into conflict, and one or the other must eventually win. This creates sociological trouble, but there's a solution:

The conflict and exclusion of alternatives that is necessary to sharp inductive inference has been all too often a conflict between men, each with his single Ruling Theory. But whenever each man begins to have multiple working hypotheses, it becomes purely a conflict between ideas.

Finally, Platt suggests that all scientists continually bear in mind The Question:

But, sir, what experiment could disprove your hypothesis?

If you have time and inclination, try rereading my earlier post on the Compression Rate Method, especially the first part, in the light of Platt's paper.

Edited thanks to feedback from Cupholder.

I would like to observe that you can divide sciences as follows:

1) physical sciences (Physics and Chemistry are models): investigating the fundamentals using the old idea of the scientific method, ie hypothesis to experiment to accepting or rejecting hypothesis. It is a predict and test method.

2) historic sciences (Biology, Geology, Cosmology are models but not Biochemistry and Biophysics which go it the first group); uses the theories of physical science to create historic theories (like evolution or plate tectonics) about how things became what they are. It is only rarely that a hypothesis can be directly tested by experiment and the method sort of proceeds as observation/taxonomy to hypothesis explaining bodies of observations to simple experiments to prove that proposed processes are possible. It is a collect data and try to organize into a integrated story sort of method and very inductive.

3) inventive Science (Engineering, Medicine are examples): uses the theories of physical and historic sciences to create useful and/or profitable things. Here the method is to identify a problem then look for solutions and test proposed solutions in tests/trials.

Associated with these sciences are theoretical areas (Mathematics, Information Theory are examples): These do not test theories and are not in the business of predicting and testing against reality. Their deductive rather then inductive. They create theoretic structures that are logically robust and can be used by the other sciences as powerful tools.

I have left out the social sciences, history proper, linguistics, anthropology and economics because it is not clear to me that they are even sciences and they do not fit the mold of mathematics either. But they are (like the others) large communal scholarships and I am not putting them down when I say they may not be sciences.

My definition of a science is a communal scholarship that

a) is not secretive but public using peer reviewed publication (or its equivalent) in enough detail that the work could be repeated,

b) is concerned with understanding material physical reality and doing so by testing theories in experiments or their equivalents ie no magical or untestable explanations,

c) accepts the consensus of convinced scientists in the appropriate field rather than an authority as a measure of truth. The method by which the scientists are convinced may be anything in principle, but scientists are not likely to be convinced if the math has mistakes, logic is faulty, experiments are without controls, supernatural reasons are used etc. etc.

AI would definitely fall into the inventive sciences. An appropriate method would be to identify a problem, invent solutions using knowledge of physical and historic science and the tools of math etc., see if the solutions work in systematic tests. None of these is simple. To identify a problem, you need to have a vision of what is the end point success and a proposed path to get there. Vision does not come cheap. Invention of solutions is a creative process. Testing takes as much skill and systematic, clear thinking as any other experimental-ish procedure.

AI would definitely fall into the inventive sciences.

It seems that well-established physical or historical sciences invariably serve as the theoretical underpinning for each of the inventive sciences (electrical/mechanical/chemical engineering has the physical sciences, medicine has biology, etc.). What is the theoretical underpinning of AI? Traditionally it has been computer science, but on the face of it CS says little or nothing about the mechanisms of intelligence. Neuroscience isn't quite it either, since neuroscience is focused on describing the h... (read more)

1DanB16y

Excellent analysis. This is the kind of discussion I was looking for. Note that it is necessary to do empirical science before inventive science becomes possible. Engineering depends almost completely on knowledge of physical laws. So a plausible diagnosis of the cause of the limited progress in AI, is that it's an attempt to do invention before the relevant empirical science has become available.

15

Link: Strong Inference

15

15

15

Link: Strong Inference

15

15