Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Schools Proliferating Without Evidence

40 Post author: Eliezer_Yudkowsky 15 March 2009 06:43AM

Previously in seriesEpistemic Viciousness

Robyn Dawes, author of one of the original papers from Judgment Under Uncertainty and of the book Rational Choice in an Uncertain World—one of the few who tries really hard to import the results to real life—is also the author of House of Cards: Psychology and Psychotherapy Built on Myth.

From House of Cards, chapter 1:

The ability of these professionals has been subjected to empirical scrutiny—for example, their effectiveness as therapists (Chapter 2), their insight about people (Chapter 3), and the relationship between how well they function and the amount of experience they have had in their field (Chapter 4).  Virtually all the research—and this book will reference more than three hundred empirical investigations and summaries of investigations—has found that these professionals' claims to superior intuitive insight, understanding, and skill as therapists are simply invalid...

Remember Rorschach ink-blot tests?  It's such an appealing argument: the patient looks at the ink-blot and says what he sees, the psychotherapist interprets their psychological state based on this.  There've been hundreds of experiments looking for some evidence that it actually works.  Since you're reading this, you can guess the answer is simply "No."  Yet the Rorschach is still in use.  It's just such a good story that psychotherapists just can't bring themselves to believe the vast mounds of experimental evidence saying it doesn't work—

—which tells you what sort of field we're dealing with here.

And the experimental results on the field as a whole are commensurate.  Yes, patients who see psychotherapists have been known to get better faster than patients who simply do nothing.  But there is no statistically discernible difference between the many schools of psychotherapy.  There is no discernible gain from years of expertise.

And there's also no discernible difference between seeing a psychotherapist and spending the same amount of time talking to a randomly selected college professor from another field.  It's just talking to anyone that helps you get better, apparently.

In the entire absence of the slightest experimental evidence for their effectiveness, psychotherapists became licensed by states, their testimony accepted in court, their teaching schools accredited, and their bills paid by health insurance.

And there was also a huge proliferation of "schools", of traditions of practice, in psychotherapy; despite—or perhaps because of—the lack of any experiments showing that one school was better than another...

I should really post more some other time on all the sad things this says about our world; about how the essence of medicine, as recognized by society and the courts, is not a repertoire of procedures with statistical evidence for their healing effectiveness; but, rather, the right air of authority.

But the subject today is the proliferation of traditions in psychotherapy.  So far as I can discern, this was the way you picked up prestige in the field—not by discovering an amazing new technique whose effectiveness could be experimentally verified and adopted by all; but, rather, by splitting off your own "school", supported by your charisma as founder, and by the good stories you told about all the reasons your techniques should work.

This was probably, to no small extent, responsible for the existence and continuation of psychotherapy in the first place—the promise of making yourself a Master, like Freud who'd done it first (also without the slightest scrap of experimental evidence).  That's the brass ring of success to chase—the prospect of being a guru and having your own adherents.  It's the struggle for adherents that keeps the clergy vital.

That's what happens to a field when it unbinds itself from the experimental evidence—though there were other factors that also placed psychotherapists at risk, such as the deference shown them by their patients, the wish of society to believe that mental healing was possible, and, of course, the general dangers of telling people how to think.

The field of hedonic psychology (happiness studies) began, to some extent, with the realization that you could measure happiness—that there was a family of measures that by golly did validate well against each other.

The act of creating a new measurement creates new science; if it's a good measurement, you get good science.

If you're going to create an organized practice of anything, you really do need some way of telling how well you're doing, and a practice of doing serious testing—that means a control group, an experimental group, and statistics—on plausible-sounding techniques that people come up with.  You really need it.

Added:  Dawes wrote in the 80s and I know that the Rorschach was still in use as recently as the 90s, but it's possible matters have improved since then (as one commenter states).  I do remember hearing that there was positive evidence for the greater effectiveness of cognitive-behavioral therapy.

 

Part of the sequence The Craft and the Community

Next post: "3 Levels of Rationality Verification"

Previous post: "Epistemic Viciousness"

Comments (57)

Comment author: knb 15 March 2009 07:48:34AM *  19 points [-]

You are describing old clinical psychology. Its gotten so much better. Rorschach tests are now only a very marginal measure within psychoanalytic psychology. Psychoanalytic/pscyhodynamic psychologists are themselves outcasts from mainstream clinical psychology, which is increasingly centered around evidence-based practice. For example, behaviorists are using systematic desensitization in novel and effective ways (for treating things like panic disorder), and cognitive-behavioral therapy is quite effective in treating depression: significantly more so than antidepressants.

The important thing to remember is that patients often get the treatment they want. If you're a self-absorbed neurotic, and you want to spend an hour a week for years talking about yourself, you can find someone who will take your money. If you want effective treatment, you can find that too. Most patients don't want to get better, they want to feel like they are doing something, and especially they want to talk about themselves.

Comment author: komponisto 15 March 2009 04:36:16PM *  12 points [-]

The important thing to remember is that patients often get the treatment they want. If you're a self-absorbed neurotic, and you want to spend an hour a week for years talking about yourself, you can find someone who will take your money. [...] Most patients don't want to get better [...] they want to talk about themselves.

Perhaps unwittingly, this comment suggests wherein the value of such psychotherapy lies. There's a social taboo against talking about oneself in this fashion, and a place that is "safe" from this (and other) conversational taboos may well be worth paying for.

Comment author: Eliezer_Yudkowsky 15 March 2009 08:01:05AM 6 points [-]

Well, Dawes wrote in the 80s and I remember being Rorschach'd in the 90s, but I can imagine that things have gotten better in the 00s. Still, I have to ask - have there been any experimental studies showing the improvement?

(I do remember hearing that there was positive experimental validation for cognitive-behavioral therapy doing systematically better than other forms of psychotherapy.)

Comment author: CarlShulman 15 March 2009 08:53:50AM *  15 points [-]

Yes, cognitive-behavioral therapy has come out ahead of other methods in a number of randomized clinical trials.

http://en.wikipedia.org/wiki/Cognitive_behavioral_therapy

[Setting karma to null]

Comment author: RobinHanson 15 March 2009 02:21:34PM 9 points [-]

Carl, following that link to its source brought me here: http://www.nice.org.uk/guidance/index.jsp?action=download&o=33188, where several randomized trials are mentioned. But I see no meta-analysis, so I still worry about publication selection biases, etc. Anyone know of a meta-analysis of this lit?

Comment author: Jesper_Ostman 16 January 2012 05:48:21PM *  5 points [-]
Comment author: MichaelBishop 15 March 2009 04:04:54PM 8 points [-]

I know I'm not teaching Robin anything, but it should be noted that meta-analyses often fail to overcome publication selection biases.

Comment author: Will_Newsome 20 September 2010 11:10:26AM 4 points [-]

I remember being Rorschach'd in the 90s

Were you too young for this to have led to an awesome story?

Comment author: wallowinmaya 02 June 2011 05:38:56PM *  3 points [-]

Psychoanalytic/pscyhodynamic psychologists are themselves outcasts from mainstream clinical psychology

I don't know where you live, but in Germany or Austria this is not the case. This is part of a more general problem in Europe, which is under the terrible reign of continental philosophy.

Comment author: RobinHanson 15 March 2009 02:24:01PM 29 points [-]

"That's what happens to a field when it unbinds itself from the experimental evidence" - so the million dollar question for Less Wrong is: what experimental evidence can this community bind itself to, to avoid the same outcome?

Comment author: Eliezer_Yudkowsky 15 March 2009 05:04:14PM 19 points [-]

Well, yes, that is the fairly heavy subtext here.

Comment author: gwern 13 October 2012 09:20:02PM 5 points [-]

Today I learned that this psychotherapy criticism apparently has a name: http://en.wikipedia.org/wiki/Dodo_bird_verdict

Comment author: PhilGoetz 15 March 2009 03:31:38PM 9 points [-]

Cognitive behavioral therapy, at least, has repeatedly been shown to be effective. I'll cite a ref from Wikipedia: Cooper, Mick (2008). Essential Research Findings in Counselling and Psychotherapy: The Facts are Friendly. SAGE Publications. ISBN 9781847870421. But I haven't read that reference.

The Rorschach test is still in use, and still being taught to students. There are studies claiming to find statistical significance in its results. They don't propose a mechanism; but they do have statistically-significant repeatable correlations between patient responses and their clinical problems. If you want to claim otherwise, get some of those papers and refute their methodology. Again, I haven't been interested enough to read the primary sources myself; but I've been told about them by someone I respect. The key questions are whether they erred by not doing something like a Bonferroni adjustment for the large number of possible hypotheses; and whether the statistically-significant correlations are clinically significant. You would have to read the papers to find out.

Comment author: scientism 15 March 2009 02:32:20PM 9 points [-]

This shouldn't be surprising. Medicine has a longer history than empirical science. For thousands of years it flourished without a second thought for outcome. Clearly whatever medicine is, socially speaking, it isn't reliant on the effectiveness of its methods for its survival. The same is true of education. Schools and universities existed long before there was anything to teach. Whatever social role they may play, imparting skill is a recent development, and clearly not the most central concern.

Comment author: pjeby 15 March 2009 08:58:35PM 13 points [-]

NLP is an interesting parallel example. Its founders believed in testing, but the students, not so much. For example, Bandler, in "Using Your Brain For A Change", wrote:

A mathematician doesn't just get an answer and say, "OK, I'm done." He tests his answers carefully, because if he doesn't, other mathematicians will! That kind of rigor has always been missing from therapy and education. People try something and then do a two-year follow-up study to find out if it worked or not. If you test rigorously, you can find out what a technique works for and what it doesn't work for, and you can find out right away. And where you find out that it doesn't work, you need to try some other technology.

It's a transcript from a lecture demonstration where he's just shown how to extinguish someone's urge to smoke, and is testing the result. The volunteer had just claimed she no longer wished to smoke, but Bandler insists that she actually take a cigarette from him, hold it in her hands, and play around with it.

He says:

When you do change work, don't back away from testing it; push it. Events in the world are going to push it, so you may as well do it so you can find out right away. That way you can do something about it. Observing your client's nonverbal responses will give you much more information than the verbal answers to your questions.

He then points out the changed facial expression on the volunteer -- it appears that smelling the cigarette has restored her desire for one. He gives her some modified instructions, to repeat the technique being taught. Afterwards, they verify that the smell no longer acts as a compulsion trigger.

Now, the crazy thing is, Bandler's been teaching this stuff for 20 years, but hardly anybody "gets it"... about testing, or damn near anything else.

Few NLP practitioners do much testing; few NLP books even mention it. Even Bandler's own books don't say that much about it. The formal NLP trainings emphasize "outcome frame" (defining in advance what result you're trying to get), but not so much the process of testing that you've achieved that outcome.

I suspect that this is simply because Bandler is a lousy teacher in some ways. In my personal experience, the most important parts of nearly every Bandler video, audio, or book are in what seem like almost offhand remarks... that happen to reveal volumes if you already happen to be close to figuring out the same thing for yourself. Perhaps this is why he's so insistent that an NLP certification is worthless if it doesn't come from him.

Comment author: gworley 26 April 2009 11:22:31AM *  3 points [-]

Lest we be too, too hard on these folks, I should add that, as Eli mentions, there is some value in talking to someone about your problems. I have seen a therapist on occasion who mainly just listens and then tries to advise based on human experience. No lying on the couch, no ink blots, no hypnosis. Just a neutral party to talk to who will keep what I say in confidence. Sort of like Catholic confession, but without all the Hail Marys.

Also, just in case anyone gets confused here (I'm sure someone will), psychotherapist are not the same thing as psychiatrist: the latter are actual doctors who can prescribe therapies, be they drug or other. They may not all be great either, but at least one that I've seen really knows his stuff.

Comment author: CronoDAS 15 March 2009 08:01:22AM 9 points [-]

I also have the same kind of unease about economics, especially macroeconomics. It's damn hard to run an experiment to test a hypothesis, and you have also have people running around claiming to be Austrian economists, Keynesian economists, Chicago school economists, etc.

For example: does raising the minimum wage by a few dollars an hour really increase the percentage of people who don't have a job (the sum of the unemployed and those not in the labor force)? Can you prove it?

Comment author: PhilGoetz 15 March 2009 03:33:57PM 5 points [-]

I think it's a no-brainer that we should have taken some of the trillions of dollars we're spending on economic stimulus to study whether economic stimulus works. It would be a good investment to create a National Institute of Economics.

Comment author: Matt_Simpson 15 March 2009 09:46:40PM 6 points [-]

Another example of schools proliferating without evidence: philosophy. Consider all the different schools of ethics which have sprung up: there's utilitarian ethics, deontological ethics, and virtue ethics, with vast numbers of sub categorizations under each school.

Philosophers are more susceptible to this failure mode because on many important philosophical questions, a standard if not unanimous approach is argue that the question cannot be answered by evidence. Modal logicians trying to do metaphysics, for example.

Comment author: thomblake 16 March 2009 02:47:46PM 8 points [-]

Consider all the different schools of ethics which have sprung up

A few thousand years, and we've managed to come up with about three possible answers to the question 'what, in general, does one have most reason to do or want?'. Is your complaint that this is too many to have considered, or that the question isn't completely settled yet?

a standard if not unanimous approach is argue that the question cannot be answered by evidence

I know many philosophers who would be surprised by this assertion - I was under the impression the Empiricists pretty much won. In Ethics, particularly, moral observation is now a standard piece of the toolkit.

Of course, the grain of truth here is that due to the fractured nature of philosophical schools, there are large communities of philosophers who don't realize other large communities of philosophers even exist. In a sense, nobody knows what philosophy doesn't know, even philosophers.

Comment author: Matt_Simpson 16 March 2009 04:25:02PM 3 points [-]

Is your complaint that this is too many to have considered, or that the question isn't completely settled yet?

My complaint is that little progress has been made over many years. There are three general ways to answer the question, sure. But each general answer is really an umbrella term covering a large number of answers. Some sects are similar to others, but they are still different sects.

I know many philosophers who would be surprised by this assertion - I was under the impression the Empiricists pretty much won. In Ethics, particularly, moral observation is now a standard piece of the toolkit.

In retrospect, my experience is probably colored by the small school I go to. From what I can tell, there are still rather large, if minority, groups of philosophers who disagree with the settled answer on many questions.

Comment author: buybuydandavis 25 March 2013 07:50:40PM *  3 points [-]

My obligatory Edwin Jaynes quote on philosophers, quoting a colleague

"Philosophers are free to do whatever they please, because they don't have to do anything right."

Probability Theory http://books.google.com/books?id=tTN4HuUNXjgC&pg=PA144

Comment author: Annoyance 16 March 2009 01:37:05AM -2 points [-]

"Philosophers are more susceptible to this failure mode because on many important philosophical questions, a standard if not unanimous approach is argue that the question cannot be answered by evidence."

No, philosophers are more susceptible because most of them can't recognize that "cannot be answered by evidence" means an answer can't be obtained at all.

To such individuals, reason is merely a passing fad coequal with every other way of asserting something, a fleeting hiccup that they're far too fashionable to consider important.

Comment author: Matt_Simpson 16 March 2009 04:05:05AM -1 points [-]

No, philosophers are more susceptible because most of them can't recognize that "cannot be answered by evidence" means an answer can't be obtained at all.

I would say both. Some things that philosophers think can't be answered by evidence are in fact answered by evidence, such as whether 2 + 2 = 4.

Comment author: whowhowho 25 March 2013 08:44:24PM -1 points [-]

Is any of that avoidable?

Comment author: shminux 25 March 2013 09:25:14PM 0 points [-]
Comment author: whowhowho 25 March 2013 09:33:55PM -1 points [-]

Please provide proof. Please don't point, yet again, to the highly debatable "solution" to FW.

Comment author: shminux 25 March 2013 11:58:21PM 0 points [-]

What kind of proof would you accept?

Comment deleted 26 March 2013 07:50:34PM *  [-]
Comment author: shminux 26 March 2013 09:22:44PM *  7 points [-]

In my limited experience, the "hard problems" in philosophy are the problems which are either poorly defined and so people keep arguing about definitions without admitting it, or poorly analyzed, so people keep mixing decision theory with cognitive science, for example. While the traditional philosophy is good at asking (meta-)questions and noticing broad similarities, it is nearly useless at solving them. When a philosopher tries to honestly analyze a deep question, it usually stops being philosophy and becomes logic, linguistics, decision theory, computer science, physics or something else that qualifies as science. Hence Pearl and Kahneman and Russell, some Wittgenstein, Popper...

Comment author: [deleted] 27 March 2013 11:10:33AM 0 points [-]

In my limited experience, the "hard problems" in philosophy are the problems which are either poorly defined and so people keep arguing about definitions without admitting it, or poorly analyzed, so people keep mixing decision theory with cognitive science, for example.

See also how many of the comments in this thread amounted to “if by sound you mean ‘acoustic wave’ it does, if you mean ‘auditory sensation’ it doesn't”.

Comment author: TheAncientGeek 18 April 2015 10:03:39AM -2 points [-]

There's little evidence of anything else being better at solving them, so that is largely nirvana fallacy,

Comment author: dxu 18 April 2015 07:13:16PM 3 points [-]

Wait, what? There's little evidence of anything better than philosophy at solving problems? How about physics, cognitive science, computer science, mathematics, etc.?

Comment author: Epictetus 19 April 2015 02:40:14AM 2 points [-]

When a branch of philosophy becomes useful at solving problems, people give it a new name and no longer consider it part of philosophy.

Comment author: TheAncientGeek 18 April 2015 07:20:50PM *  0 points [-]

Them="the hard problems in philosophy", not "problems"

How about physics, cognitive science, computer science, mathematics, etc.?

How about philosophy of physics, philosophy of mathematics? Why do they exist?

Comment author: Technologos 15 March 2009 07:11:10AM 3 points [-]

It's my understanding that a good amount of social science does the same--I was recently in a class on organization theory that made few if any testable predictions and ended up being a bunch of just-so stories about how people in organizations think.

One of the side effects of prediciton markets of the Hansonian variety is that people would rapidly see who is doing good science and who isn't: wealth would be rapidly transferred from those who don't know how to make good evidence-centric predictions to those who do.

Comment author: timtyler 26 March 2013 12:18:18AM 1 point [-]

Remember Rorschach ink-blot tests? It's such an appealing argument: the patient looks at the ink-blot and says what he sees, the psychotherapist interprets their psychological state based on this. There've been hundreds of experiments looking for some evidence that it actually works. Since you're reading this, you can guess the answer is simply "No."

I checked. It doesn't say "No".

Comment author: Vaniver 26 March 2013 12:47:17AM 5 points [-]

I checked. It doesn't say "No".

That depends on what Eliezer / you mean by "it." From my reading of the evidence, the claim that the psychotherapist can interpret the patient's psychological state by use of the Rorschach ink-blots has been refuted; any knowledge they get is probably from cold reading, and they fail to notice real evidence they're not looking for. This is an empirical statement about the population of psychotherapists, rather than the best application of the test, though.

Comment author: timtyler 26 March 2013 01:12:19AM 0 points [-]

Cold reading sounds pretty negative. Cold reading is a technique used by mentalists, psychics, fortune-tellers, mediums and illusionists to dupe their marks. If you want to go with a negative comparison, perhaps consider Tasseography ;-)

Comment author: Vaniver 26 March 2013 01:59:05AM 3 points [-]

Cold reading sounds pretty negative. Cold reading is a technique used by mentalists, psychics, fortune-tellers, mediums and illusionists to dupe their marks. If you want to go with a negative comparison, perhaps consider Tasseography ;-)

I chose that phrase for its precision, not its emotional valence; were there a more neutral yet readily understandable term, I would have picked it instead.

Comment author: timtyler 26 March 2013 11:04:32AM -2 points [-]

For a start, "cold" seems as though it would often be wrong. It isn't "cold" reading when you have a medical history in front of you.

Comment author: Jayson_Virissimo 26 March 2013 12:48:21AM *  2 points [-]

I looked at the original 10 inkblots and saw pelvic bones in 9 of them. I wonder what that says about me...

Comment author: gwern 16 July 2012 09:12:05PM 1 point [-]

And there's also no discernible difference between seeing a psychotherapist and spending the same amount of time talking to a randomly selected college professor from another field. It's just talking to anyone that helps you get better, apparently.

See also http://lesswrong.com/lw/94t/meta_analysis_of_writing_therapy/

Comment author: sonic 15 March 2009 08:11:55AM 1 point [-]

Excellent analysis. I think your historical explanation is important- not the totality of the situation, but important. Problems with psychotherapy- 1) What are the goals? and 2) "I like what is happening" is a valid reason to consider it successful. Having to answer 1) above is not that unusual for a scientific endeavor. The utter subjective nature of 2) above makes it very unusual for a scientific study.

Comment author: CWG 18 April 2015 07:03:29AM 0 points [-]

And there's also no discernible difference between seeing a psychotherapist and spending the same amount of time talking to a randomly selected college professor from another field. It's just talking to anyone that helps you get better, apparently.

Unless this has been tested for random people other than just college professors, there's a stronger case for saying that talking to a person of a certain intelligence and education level helps you get better. And I suspect that it doesn't generalise to "talking to anyone that helps you get better" but I haven't looked into it.

(I'm sure there are other factors, but I'm just going by what was said about college professors.)