NLP is an interesting parallel example. Its founders believed in testing, but the students, not so much. For example, Bandler, in "Using Your Brain For A Change", wrote:
A mathematician doesn't just get an answer and say, "OK, I'm done." He tests his answers carefully, because if he doesn't, other mathematicians will! That kind of rigor has always been missing from therapy and education. People try something and then do a two-year follow-up study to find out if it worked or not. If you test rigorously, you can find out what a technique works for and what it doesn't work for, and you can find out right away. And where you find out that it doesn't work, you need to try some other technology.
It's a transcript from a lecture demonstration where he's just shown how to extinguish someone's urge to smoke, and is testing the result. The volunteer had just claimed she no longer wished to smoke, but Bandler insists that she actually take a cigarette from him, hold it in her hands, and play around with it.
He says:
When you do change work, don't back away from testing it; push it. Events in the world are going to push it, so you may as well do it so you can find out right away. That way you can do something about it. Observing your client's nonverbal responses will give you much more information than the verbal answers to your questions.
He then points out the changed facial expression on the volunteer -- it appears that smelling the cigarette has restored her desire for one. He gives her some modified instructions, to repeat the technique being taught. Afterwards, they verify that the smell no longer acts as a compulsion trigger.
Now, the crazy thing is, Bandler's been teaching this stuff for 20 years, but hardly anybody "gets it"... about testing, or damn near anything else.
Few NLP practitioners do much testing; few NLP books even mention it. Even Bandler's own books don't say that much about it. The formal NLP trainings emphasize "outcome frame" (defining in advance what result you're trying to get), but not so much the process of testing that you've achieved that outcome.
I suspect that this is simply because Bandler is a lousy teacher in some ways. In my personal experience, the most important parts of nearly every Bandler video, audio, or book are in what seem like almost offhand remarks... that happen to reveal volumes if you already happen to be close to figuring out the same thing for yourself. Perhaps this is why he's so insistent that an NLP certification is worthless if it doesn't come from him.
Robyn Dawes, author of one of the original papers from Judgment Under Uncertainty and of the book Rational Choice in an Uncertain World—one of the few who tries really hard to import the results to real life—is also the author of House of Cards: Psychology and Psychotherapy Built on Myth.
From House of Cards, chapter 1:
Remember Rorschach ink-blot tests? It's such an appealing argument: the patient looks at the ink-blot and says what he sees, the psychotherapist interprets their psychological state based on this. There've been hundreds of experiments looking for some evidence that it actually works. Since you're reading this, you can guess the answer is simply "No." Yet the Rorschach is still in use. It's just such a good story that psychotherapists just can't bring themselves to believe the vast mounds of experimental evidence saying it doesn't work—
—which tells you what sort of field we're dealing with here.
And the experimental results on the field as a whole are commensurate. Yes, patients who see psychotherapists have been known to get better faster than patients who simply do nothing. But there is no statistically discernible difference between the many schools of psychotherapy. There is no discernible gain from years of expertise.
And there's also no discernible difference between seeing a psychotherapist and spending the same amount of time talking to a randomly selected college professor from another field. It's just talking to anyone that helps you get better, apparently.
In the entire absence of the slightest experimental evidence for their effectiveness, psychotherapists became licensed by states, their testimony accepted in court, their teaching schools accredited, and their bills paid by health insurance.
And there was also a huge proliferation of "schools", of traditions of practice, in psychotherapy; despite—or perhaps because of—the lack of any experiments showing that one school was better than another...
I should really post more some other time on all the sad things this says about our world; about how the essence of medicine, as recognized by society and the courts, is not a repertoire of procedures with statistical evidence for their healing effectiveness; but, rather, the right air of authority.
But the subject today is the proliferation of traditions in psychotherapy. So far as I can discern, this was the way you picked up prestige in the field—not by discovering an amazing new technique whose effectiveness could be experimentally verified and adopted by all; but, rather, by splitting off your own "school", supported by your charisma as founder, and by the good stories you told about all the reasons your techniques should work.
This was probably, to no small extent, responsible for the existence and continuation of psychotherapy in the first place—the promise of making yourself a Master, like Freud who'd done it first (also without the slightest scrap of experimental evidence). That's the brass ring of success to chase—the prospect of being a guru and having your own adherents. It's the struggle for adherents that keeps the clergy vital.
That's what happens to a field when it unbinds itself from the experimental evidence—though there were other factors that also placed psychotherapists at risk, such as the deference shown them by their patients, the wish of society to believe that mental healing was possible, and, of course, the general dangers of telling people how to think.
The field of hedonic psychology (happiness studies) began, to some extent, with the realization that you could measure happiness—that there was a family of measures that by golly did validate well against each other.
The act of creating a new measurement creates new science; if it's a good measurement, you get good science.
If you're going to create an organized practice of anything, you really do need some way of telling how well you're doing, and a practice of doing serious testing—that means a control group, an experimental group, and statistics—on plausible-sounding techniques that people come up with. You really need it.
Added: Dawes wrote in the 80s and I know that the Rorschach was still in use as recently as the 90s, but it's possible matters have improved since then (as one commenter states). I do remember hearing that there was positive evidence for the greater effectiveness of cognitive-behavioral therapy.