Robyn Dawes, author of one of the original papers from Judgment Under Uncertainty and of the book Rational Choice in an Uncertain World—one of the few who tries really hard to import the results to real life—is also the author of House of Cards: Psychology and Psychotherapy Built on Myth.
From House of Cards, chapter 1:
The ability of these professionals has been subjected to empirical scrutiny—for example, their effectiveness as therapists (Chapter 2), their insight about people (Chapter 3), and the relationship between how well they function and the amount of experience they have had in their field (Chapter 4). Virtually all the research—and this book will reference more than three hundred empirical investigations and summaries of investigations—has found that these professionals' claims to superior intuitive insight, understanding, and skill as therapists are simply invalid...
Remember Rorschach ink-blot tests? It's such an appealing argument: the patient looks at the ink-blot and says what he sees, the psychotherapist interprets their psychological state based on this. There've been hundreds of experiments looking for some evidence that it actually works. Since you're reading this, you can guess the answer is simply "No." Yet the Rorschach is still in use. It's just such a good story that psychotherapists just can't bring themselves to believe the vast mounds of experimental evidence saying it doesn't work—
—which tells you what sort of field we're dealing with here.
And the experimental results on the field as a whole are commensurate. Yes, patients who see psychotherapists have been known to get better faster than patients who simply do nothing. But there is no statistically discernible difference between the many schools of psychotherapy. There is no discernible gain from years of expertise.
And there's also no discernible difference between seeing a psychotherapist and spending the same amount of time talking to a randomly selected college professor from another field. It's just talking to anyone that helps you get better, apparently.
In the entire absence of the slightest experimental evidence for their effectiveness, psychotherapists became licensed by states, their testimony accepted in court, their teaching schools accredited, and their bills paid by health insurance.
And there was also a huge proliferation of "schools", of traditions of practice, in psychotherapy; despite—or perhaps because of—the lack of any experiments showing that one school was better than another...
I should really post more some other time on all the sad things this says about our world; about how the essence of medicine, as recognized by society and the courts, is not a repertoire of procedures with statistical evidence for their healing effectiveness; but, rather, the right air of authority.
But the subject today is the proliferation of traditions in psychotherapy. So far as I can discern, this was the way you picked up prestige in the field—not by discovering an amazing new technique whose effectiveness could be experimentally verified and adopted by all; but, rather, by splitting off your own "school", supported by your charisma as founder, and by the good stories you told about all the reasons your techniques should work.
This was probably, to no small extent, responsible for the existence and continuation of psychotherapy in the first place—the promise of making yourself a Master, like Freud who'd done it first (also without the slightest scrap of experimental evidence). That's the brass ring of success to chase—the prospect of being a guru and having your own adherents. It's the struggle for adherents that keeps the clergy vital.
That's what happens to a field when it unbinds itself from the experimental evidence—though there were other factors that also placed psychotherapists at risk, such as the deference shown them by their patients, the wish of society to believe that mental healing was possible, and, of course, the general dangers of telling people how to think.
The field of hedonic psychology (happiness studies) began, to some extent, with the realization that you could measure happiness—that there was a family of measures that by golly did validate well against each other.
The act of creating a new measurement creates new science; if it's a good measurement, you get good science.
If you're going to create an organized practice of anything, you really do need some way of telling how well you're doing, and a practice of doing serious testing—that means a control group, an experimental group, and statistics—on plausible-sounding techniques that people come up with. You really need it.
Added: Dawes wrote in the 80s and I know that the Rorschach was still in use as recently as the 90s, but it's possible matters have improved since then (as one commenter states). I do remember hearing that there was positive evidence for the greater effectiveness of cognitive-behavioral therapy.
NLP is an interesting parallel example. Its founders believed in testing, but the students, not so much. For example, Bandler, in "Using Your Brain For A Change", wrote:
It's a transcript from a lecture demonstration where he's just shown how to extinguish someone's urge to smoke, and is testing the result. The volunteer had just claimed she no longer wished to smoke, but Bandler insists that she actually take a cigarette from him, hold it in her hands, and play around with it.
He says:
He then points out the changed facial expression on the volunteer -- it appears that smelling the cigarette has restored her desire for one. He gives her some modified instructions, to repeat the technique being taught. Afterwards, they verify that the smell no longer acts as a compulsion trigger.
Now, the crazy thing is, Bandler's been teaching this stuff for 20 years, but hardly anybody "gets it"... about testing, or damn near anything else.
Few NLP practitioners do much testing; few NLP books even mention it. Even Bandler's own books don't say that much about it. The formal NLP trainings emphasize "outcome frame" (defining in advance what result you're trying to get), but not so much the process of testing that you've achieved that outcome.
I suspect that this is simply because Bandler is a lousy teacher in some ways. In my personal experience, the most important parts of nearly every Bandler video, audio, or book are in what seem like almost offhand remarks... that happen to reveal volumes if you already happen to be close to figuring out the same thing for yourself. Perhaps this is why he's so insistent that an NLP certification is worthless if it doesn't come from him.