My recollection of my conversation with Geoff, at a Berkeley LW meetup, in Berkeley University in a building that had a statue of a dinosaur in it, is that it went like this. Disclaimer: My episodic memory for non-repeated conversations is terrible and it is entirely possible that there are major inaccuracies here. Disclaimer 2: This is not detailed enough to count as an engaged critique, and Geoff is not obliged to respond to it since I put very little effort into it myself (it is logically rude to demand ever-more conversational effort from other people while putting in very little yourself).
Geoff: I've been working on an incredible new mental theory that explains everything.
Eliezer (internally): That's not a good sign, but he seems earnest and intelligent. Maybe it's something innocuous or even actually interesting.
Eliezer (out loud): And what does it say?
Geoff: Well, I haven't really practiced it explaining it, and I don't expect you to believe it, but (explains CT)
Eliezer (internally): Well this is obviously wrong. Minds just don't work by those sorts of bright-line psychoanalytic rules written out in English, and proposing them doesn't get you anywhere near the level of an interesting cognitive algorithm. Maybe if he's read enough of the Sequences and hasn't invested too many sunk costs / bound up too many hopes in it, I can snap him out of it in fairly short order?
Eliezer (out loud): Where does CT make a different prediction from the cognitive science I already know that I couldn't get without CT?
Geoff: It predicts that people will change their belief to believe that their desires will be fulfilled...
Eliezer (internally): Which sounds a lot like standard cognitive dissonance theory, which itself has been modified in various ways, but we aren't even at the point of talking about that until we get out of the abstract-belief trap.
Eliezer (out loud): No, I mean some sort of sensory experience. Like your eyes seeing an apple fall from a tree, or something like that. What does CT say I should experience seeing, that existing cognitive science wouldn't tell me to expect?
Geoff: (Something along the lines of "CT isn't there yet", I forget the exact reply.)
Eliezer (internally): This is exactly the sort of blind alley that the Sequences are supposed to prevent smart people from wasting their emotional investments on. I wish I'd gotten this person to read the Belief and Anticipation sequence before CT popped into his head, but there's no way I can rescue him from the outside at this point.
Eliezer (out loud): Okay, then I don't believe in CT because without evidence there's no way you could know it even if it was true.
I think there might've also been something about me trying to provide a counterexample like "It is psychologically possible for mothers to believe their children have cancer" but I don't recall what Geoff said to that. I'm not sure whether or not I gave him any advice along the lines of, "Try to explain one thing before explaining everything."
If I recall correctly, I was saying that I didn't know how to use CT to predict simple things of the form "Xs will always Y" or "Xs will Y at rate Z", where X and Y refer to simple observables like "human", "blush", etc. It would be great if I could do this, but unfortunately I can't.
Instead, what I can do is use the CT charting procedure to generate a CT chart for someone and then use CT to derive predictions from the chart. This yields predictions of the form "if a person with chart X does Y, Z will occur". These predictions frequently do not overlap with what existing cognitive science would have one expect.
The way I could have evidence in favor of CT would be if I had created CT charts using the CT procedure, used CT to derive predictions from the charts, and then tested the predictions. And I've done this.
The way I could have evidence in favor of CT would be if I had created CT charts using the CT procedure, used CT to derive predictions from the charts, and then tested the predictions. And I've done this.
See, this is an example of what I mean about the CT website equaling "not understanding 'evidence'".
What you've described is primarily evidence for "more detailed models of a specific human make more accurate and surprising predictions than using a generic model of humanity."
It is almost no evidence for CT's actual theory.
By comparison, consider The Secret and such - "law of attraction". If you follow some of their procedures, you actually stand a good chance of obtaining some of their results... but this does not actually lend any evidential weight to the idea that a "law of attraction" actually exists. Richard Wiseman's "luck" research (showing a link between self-perceived "luck" and ability to notice lucky opportunities) provides a much better theory to explain such results.
In the case of CT, other practical and theoretical models involving mapping of a person's beliefs exist. One model (which isn't even a psychological theory, mind you) is a ToC "Current Reality Tree" (CRT) diagram based solely on elementary cause-and-effect logic. You can use a CRT to map and predict the behavior of extremely complex businesses (that is pretty much what it's for) and make all sorts of useful predictions from one, and I've used them in the past with beliefs as well.
But really, a CRT is just a visualization of elementary logic, and a CT chart is shorthand for a CRT, so in effect all your "evidence" is proving is that CT's practical approach is maybe as good as elementary logic... without providing any evidence left over for the theory itself. ;-)
That being said, building good theories involving the mind is hard; building workable techniques is much easier by comparison. (Though still no picnic!) I have long ago given up on trying to do the former, and stick with the latter, using theories now only as mnemonics and intuition pumps to drive techniques. You might be better off doing the same.
These predictions frequently do not overlap with what existing cognitive science would have one expect.
What is an example of a case you've actually observed where CT made a falsifiable, bold, successful prediction? ("Falsifiable" - say what would have made the prediction fail. "Bold" - explain what a cogsci guy or random good human psychologist would have falsifiably predicted differently.)
For at least 2 years prior to January 2009, I procrastinated between 1-3 hours a day reading random internet news sites. After I created my first CT chart, I made the following prediction: "If I design a way to gain information about the world that does not involve reading internet news sites that also does not alter my way of achieving my other intrinsic goods, then I will stop spending time reading these internet news sites." The "does not alter my way of achieving my other intrinsic goods" was unpacked. It included: "does not alter my way of gaining social acceptance", "does not alter my relationships with my family members", etc. The specifics were unpacked there as well.
This was prediction was falsifiable - it would have failed if I had kept reading internet news sites. It was also bold - cogsci folk and good random human psychologists would have predicted no change in my internet news reading behavior. And it was also successful - after implementing the recommendation in January 2009, I stopped procrastinating as predicted. Now, of course there are multiple explanations for the success of the prediction, including "CT is true" and "you just used your willpower". Nevertheless, this is an example of a faisifiable, bold, successful prediction.
cogsci folk and good random human psychologists would have predicted no change in my internet news reading behavior.
Your model of human psychologists needs updating, then. Books on hypnotism that I read when I was 11 discuss needs substitution, secondary gain, etc. that would be relevant to making such a prediction. Any good human psychologist knows to look for what gains a behavior produces.
Of course, maybe you meant "good (random human) psychologists", not "good, random (human psychologists)" - i.e., psychologists who study the behavior of random humans, rather than people who help individual humans... in which case, that's a really low bar for CT to leap over.
Also:
it would have failed if I had kept reading internet news sites.
This is also a really low bar, unless you specify how long you would stay away from them. In this case, three years is pretty good, but just getting somebody to stop for a few days or even a couple months is still a relatively low bar.
The keywords in psychology for this distinction are nomothetic vs. idiographic (which are useful as search terms, or for talking with a small subset of people). Nomothetic approaches deal with general trends among a large number of people, and cover most psychology research (e.g. people who are high in Conscientiousness tend to have more successful careers). Idiographic approaches try to engage with a particular individual's psychology in detail. From what I've read, I'd call CT an idiographic approach to motivated reasoning and defensiveness, with promising potential applications.
Stanovich on science-by-anecdote:
As psychologist Ray Nickerson (1998) has said in his review of the cognitive processes we use to deceive ourselves, "Every practitioner of a form of pseudomedicine can point to a cadre of patients who will testify, in all sincerity, to have benefited from the treatment" (p. 192). For example, subliminal self-help audiotapes... that are purported to raise memory performance or self-esteem generate plenty of testimonials despite the fact that controlled studies indicate that they have absolutely no effect on memory or self-esteem (Moore, 1995).
Hey Peter,
Thanks for writing this.
I’m the primary researcher working on Connection Theory at Leverage. I don’t have time to give an in-depth argument for why I consider CT to be worth investigating at the moment, but I will briefly respond to your post:
Objections I & II:
I think that your skeptical position is reasonable given your current state of knowledge. I agree that the existing CT documents do not make a persuasive case.
The CT research program has not yet begun. The evidence presented in the CT documents is from preliminary investigations carried out shortly after the theory’s creation when Geoff was working on his own.
My current plan is a follows: Come to understand CT and how to apply it well enough to design (and be able to carry out) a testing methodology that will provide high quality evidence. Perform some preliminary experiments. If the results are promising, create training material and programs that produce researchers who reliably create the same charts, predictions and recommendations from the same data. Recruit many aspiring researchers. Train many researchers. Begin large-scale testing.
Objection III:
I agree that a casual reading of CT suggests that it conflicts with existing science. I thought so as well and initially dismissed the theory for just that reason. Several extended conversations with Geoff and the experience of having my CT chart created convinced me otherwise. Very briefly:
The brain is complicated and the relationship between brain processes and our everyday experience of acting and updating is poorly understood. Since CT is trying to be a maximally elegant theory of just these things, CT does not attempt to say anything one way or the other about the brain and so strictly speaking does not predict that beliefs can be changed by modifying the brain. That said, it is easy to specify a theory, which we might call CT’, that is identical in every respect except that it allows beliefs to be modified directly by altering the brain.
“Elegant updating” is imprecisely defined in the current version of CT. This is definitely a problem with the theory. That said, I don’t think the concept is hopelessly imprecise. For one, elegant updating as defined by CT does not mean ideal Bayesian updating. One of the criteria of elegance is that the update involve the fewest changes from the previous set of beliefs. This means that a less globally-elegant theory may be favored over a more globally-elegant theory due to path-dependence. This introduces another source of less-than-optimally-rational beliefs. If we imagine a newly formed CT-compliant mind with a very minimal belief system updating in accordance with this conception of elegance and the constraints from their intrinsic goods (IGs), I think we should actually expects its beliefs to be totally insane, even more so than the H&B literature would suggest. Of course, we will need to do research in developmental psychology to confirm this suspicion.
It is surprisingly easy to explain many common biases within the CT framework. The first bias you mentioned, scope insensitivity, is an excellent example. Studies have shown that the amount people are willing to donate to save 2,000 birds is about the same as what they are willing to donate to save 200,000 birds. Why might that be?
According to CT, people only care about something if it is part of a path to one of their IGs. The IGs we’ve observed so far are mostly about particular relationships with other people, group membership, social acceptance, pleasure and sometimes ideal states of the world (world-scale IGs or WSIGs) such as world peace, universal harmony or universal human flourishing. Whether or not many birds (or even humans!) die in the short term is likely to be totally irrelevant to whether or not a person’s IGs are eventually fulfilled. Even WSIGs are unlikely to compel donation unless the person believes their donation action to be a necessary part of a strategy in which a very large number of people donate (and thus produce the desired state). It just isn’t very plausible that your individual attempt to save a small number of lives through donation will be critical to the eventual achievement of universal flourishing (for example). That leaves social acceptance as the next most likely explanation for donation. Since the number of social points people get from donating tends not to scale very well, there is no reason to expect the amount that they donate to scale. This is not the only possible CT-compliant explanation for scope insensitivity, but my guess is that it is the most commonly applicable.
I’ll close by saying that, like Geoff, I do not believe that CT is literally true. My current belief is that it is worthy of serious investigation and that the approach to psychology that it has inspired - that of mapping out individual beliefs and actions in a detailed and systematic manner, will be of great value even if the theory itself turns out to not be.
I’ll close by saying that, like Geoff, I do not believe that CT is literally true. My current belief is that it is worthy of serious investigation and that the approach to psychology that it has inspired - that of mapping out individual beliefs and actions in a detailed and systematic manner, will be of great value even if the theory itself turns out to not be.
FWIW, the idea of a person's belief system having some fundamental criteria has been influential on my work with what I call EPIC beliefs (beliefs that are necessary for a person to see themselves and the world as having Esteem, Predict/ability, and being an Independent Collaborator).
However, I have not found intermediate mapping of beliefs to be all that helpful, since the kinds of things I help people with tend to center on concrete childhood experiences (or patterns thereof), and most all of the relevant beliefs can be found by inspecting the six layers or belief classes I call SAMMSA (Surface, Attitude, Model, Mirror, Shadow, and Assumptions).
EPIC and SAMMSA are the result of spending a lot of time mapping my own and others' beliefs and looking for common structural patterns, and mostly noticing that even among people with experience at RMI or Gendlin's Focusing, it's hard to keep the mapping exercise tied to concrete emotional beliefs without wandering off into all sorts of random distractions.
Based on my own experience, I would suggest that EPIC is a more basic or fundamental model than IG's are, and that IGs are simply things people believe are necessary conditions for themselves or the world to be EPIC - i.e., basically good, fair, predictable/masterable, with a place for them to participate in a meaningful "higher" purpose than just existing, by their own free choice.
I also don't think that "elegant updating" is or should be a premise in CT: compare with PCT's model of "reorganization", and you'll see that their model is more basic: stuff gets rewired until the conflict goes away, and "elegant" is defined evolutionarily, i.e., what you get is what's easiest for your brain to rewire to, not what necessarily involves the most human-level elegance of belief update.
By the way, when I was first referred to the CT website, my first reaction to the "evidence" put forth on the site amounted to, "ugh... these guys don't understand what 'evidence' means". It was very off-putting. Nonetheless, as I said, I did manage to take away the idea of IG-as-belief-mediator; I just didn't find your approach to be radically different from what's already in PCT and the Method of Levels, which also assume we have a hierarchy of goods determining our behavior, and suggest mapping some portion of that hierarchy in order to change behavior. I'd strongly suggest, before embarking on new research in that area, that you check out what already exists for PCT theory and its clinical spin-off, the Method of Levels.
Thanks for the info PJ!
PCT looks very interesting and your EPIC goal framework strikes me as intuitively plausible. The current list of IGs that we reference is not so much part of CT as an empirical finding from our limited experience building CT charts. Neither Geoff nor I believe that all of them are actually intrinsic. It is entirely possible that we and our subjects are simply insufficiently experienced to penetrate below them. It looks like I've got a lot reading to do :-)
I'm being charted by Jasen, and I may be wrong about this, but it seems that it is comparatively harder to find my Intrinsic Goods than it was to find the intrinsic goods of most people who were CT-charted, even though Jasen called my chart very straightforward to do.
For one thing, I know that CT can't literally be true because evolution is true and CT is not evolutionary. the comment above on nomothetic vs. idiographic partly explains that.
But maybe CT can help people think about themselves by creating a model of them more complete. Maybe it works for some kinds of people, like super-goal-oriented-rationalists, but not for the average joe (or me).
If the update system of CT encompassed evolutionarily easy mind-paths for change, and saw genetic goals besides intrinsic goods, I think it could become an outstanding idographic theory of mind, and that is why I'm being charted!
Geoff Anders has said that of CT that "I believe it is false; my concern is with how useful it is."
In many ways, Connection Theory reminds me of Neuro-linguistic programming. For example, here's a quote from one of the scientific reviews of NLP, Von Bergen (1997):
Instead of being grounded in contemporary, scientifically derived neurological theory, NLP is based on outdated metaphors of brain functioning...
[The originators] stated that they were not interested in establishing scientific validation of NLP but intended to portray what works. Hence, the authors only present anecdotal and testimonial data to support their [claims]. Even an elementary text on scientific method (for example, [How to Think Straight About Psychology]) details the myriad pitfalls of such a methodology and describes its irrelevance to legitimate theory building.
Similarly, the documents describing Connection Theory make almost no use whatsoever of the details of "contemporary, scientifically derived neurological theory" (compare to my own Crash Course in the Neuroscience of Human Motivation). Instead, CT seems to be derived from the outdated folk theory sometimes called "belief-desire theory." (To see why it's outdated, I again refer readers to my Crash Course in the Neuroscience of Human Motivation).
Also similar is the emphasis on "what works" rather than "what is true." If a technique works for you, great! But there are advantages that practical advice backed by deep theories has over practical advice backed by wrong theories. For example, you're better able to predict what will work not just for you but for other people, too.
As far as I can tell, what "works" about CT is the "CT-charting" method, which is actually just a particularly thorough application of consequentialism and goal clarification. I doubt that much of the benefit (if any) is coming from the special CT component, which seems to be unnecessary. Twelve hours spent clarifying your goals with an experienced guide does sound like something that should be valuable to people, but Leverage could take this useful exercise to the masses more effectively if they disconnected it from their unnecessary and weird-looking theory of psychology that hangs alone in theory space, apart from the grand edifice of modern science.
If it were up to me, I'd give it a non-kooky Cool Name for marketing purposes (like Getting Things Done™), do almost exactly the same thing, and let whatever psychology comes into it be the best of mainstream psychology, not Connection Theory.
Also similar is the emphasis on "what works" rather than "what is true." If a technique works for you, great! But there are advantages that practical advice backed by deep theories has over practical advice backed by wrong theories. For example, you're better able to predict what will work not just for you but for other people, too.
I don't think that traditionally trained academic psychologists are very good at predicting outcomes of interventions. I think the difference between a therapists who can predict outcomes and a therapist who doesn't isn't in the amount of academic backing of his practice but in whether he actually measures the outcomes of his own interventions and follows up with patients.
In the NLP world Roberts Dilt for example suggests a technique for curing allergies. Part of his advice is that every patient with a serious allergy is supposed to go through an allergy test by a real doctor after the intervention.
If you design decent feedback circles deep theory isn't need to be able to predict.
Twelve hours spent clarifying your goals with an experienced guide does sound like something that should be valuable to people, but Leverage could take this useful exercise to the masses more effectively if they disconnected it from their unnecessary and weird-looking theory of psychology that hangs alone in theory space, apart from the grand edifice of modern science.
I don't think that NLP has a bad track record when it comes to reaching people. Basing yourself on a weird-looking theory of psychology isn't much of a problem for reaching most people.
Has Leverage done anything recently? Their website doesn't look like it's been updated in a while.
They do seem permanently stuck in February, but a friend still asked me to write up my critique of CT, so I obliged and thought it would be worthwhile to put it here on LessWrong. I know that Anders, the author of CT and founder of Leverage, is a LessWrong user.
I don't represent Leverage but I've done a lot of work for THINK and am aware of the gist of their goings-ons.
The short answer is that their most noticeable efforts have happened under the THINK brand in the past several months, and their other projects (including CT research) are proceeding, just quietly. Updating the website on a regular basis takes effort and isn't worthwhile until they're ready to go public in a more official capacity.
I don't know offhand, but I don't think it's for a while. My impression is that they're currently doing word-of-mouth / in-person networking.
Most of what we are doing on Leverage is related to learning/developing disciplines testing theories etc... so most of it is publicly invisible. I work from Brazil, and my institute is very Leverage related, we are more public facing... www.ierfh.org (portuguese). THINK will be the most visible output for a while.
CT was not supposed to be connected with actual cognitive science? Objection #3 seems conclusive. However, I'm curious to know what's the initial motivation to create something original.
If the founder(s) of CT can consistently make better recommendations/predictions for people they interview than an average therapist (impartially judged), that's something. But say that their method has some pre-scientific (or yet to be proven scientific) charting practice - that doesn't mean that the theories behind it have any value; it may be that they're just gifted therapists with a superstition, or more likely, that the use of this particular formal structure is helpful, but could be replaced by, say, a checklist of things to inquire about.
An over-simple, over-confident science-y theory is exactly what some therapy subjects need in order to become compliant (for those who aren't more receptive to astrology or auras). I assume this is some of what pjeby does for his clients.
Or perhaps the terms are actually effective for analyzing and communicating what goes on in the human mind. Presumably that's what the researchers are excited about demonstrating.
I agree with the conclusion that LR likely need to show that CT charting->predicting is more effective than the many competing non-physically-grounded theories/practices that are popularly used. But perhaps if they're lucky they can find a way to demonstrate some new concrete stand-alone claim that they imagined with CT as inspiration.
If the founder(s) of CT can consistently make better recommendations/predictions for people they interview than an average therapist (impartially judged), that's something.
It would be something, if they did that, but there's definitely no indication that they did. The improvements mentioned are simply asserted, not demonstrated; among a far too small and certainly not representative sample; and definitely not controlled against average therapists or impartially judged.
Leverage Research is a recently formed New York-based research group "dedicate[d] to making the world a better place by the best and most effective means possible". I personally empathize and connect with this mission statement a lot, and I myself have made more or less an identical dedication personally, and through my involvement with the organizations GivingWhatWeCan and 80000 Hours.
One of the pieces of research developed by Leverage Research founder Geoff Anders (and presumably tested and developed by Leverage Research) is Connection Theory. Broadly, Connection Theory (CT) is a theory of mind that proports to explain and predict nearly all mental phenomena -- such as why people believe, desire, think, and act the way they do.
CT, if true, would be an amazing and profound success for psychology as a discipline. However, I don't think CT is true. Or rather, even if CT is true, I don't think the case made for CT is persuasive. In this essay, I intend to articulate my case against CT.
Disclaimer
I don't write this essay with the intent to "hurt" Leverage Research or Geoff Anders, but rather with the intent to either:
(1) be persuaded for the truth of CT and understand how to use it to better make the world a better place, or
(2) successfully criticize CT and help an organization with a very laudable mission better accomplish it by correcting the errors they've made.
In many ways, I expect my criticism to "miss the mark" and misunderstand CT. In doing so, I hope to allow Leverage Research to clarify their theory in light of my objections to make it less misunderstood. In other ways, I expect my criticism to reveal flaws in CT that require revision and reassessment.
(Lastly, I make this essay with the disclaimer that I have no specialized knowledge or formal training in psychology, beyond pursuing an undergraduate degree in the field. I am currently in my Junior Year at Denison University.)
Connection Theory in More Detail
Connection Theory is a combination of the following four premises:
Put more simply, CT asserts that every single person has certain fundamental desires about the way they want the world to be (usually just a few, like social acceptance and world peace), people always believe that their fundamental desires will eventually be satisfied, and that people will change their beliefs in completely rational ways except if such a change would cause them to stop believing their fundamental beliefs will be satisfied. CT also asserts that people will always act to best satisfy their fundamental desires, according to their beliefs.
The Evidence for Connection Theory
The Evidence for Connection Theory has been gathered in "Connection Theory: The Current Evidence" (link to an automatic download of a PDF file). The article mentions six pieces of evidence for CT: (1) Recommendation Tests, (2) Mind Mapping Tests, (3) Results from Everyday Use, (4) Results from Practical Application, (5) Explanation of Sociological Phenomena, and (6) Philosophical Arguments.
Recommendation Tests
The recommendation test starts with a CT chart -- that is, using Connection Theory to create a chart for a person that aims to explain their thoughts, feelings, and behavior by reference to their "intrinsic goods" (goals they fundamentally want to accomplish), "modes" (different attitudes toward life that can be triggered by various events), and "path diagrams" which connect various actions to instrumental goals that can eventually accomplish intrinsic goods.
The process for making a chart is given lengthy treatment in "Connection Theory: Theory and Practice" (link to an automatic download of the PDF) starting from page 32, though reading the whole document is probably necessary to understanding what is going on.
To make a long story short, the Recommendation Test involves creating the CT chart, then reviewing it to create a prediction in the form "If the participant takes {specified actions}, the participant will change in {specified way}". Using two different CT charts (one for Anders by Anders, the author of CT; and second for a friend of Anders by Anders), 60 predictions were made. Of those, 22 were tested, 16 came true, 5 were ruled too difficult to confirm, and 1 was ruled false but because of an error made in the first chart. This predictive success is taken as "strong evidence of CT".
Mind Mapping Tests
Mind Mapping Tests are a bit different -- for this test, one needs to generate a CT chart for a person again, but this time look through all the beliefs of the participant. For each belief, the CT theorist should be able to provide a "CT-compliant answer", ie an answer that shows the belief was derived either through "elegant updating" (being completely and ideally rational) or through a need to believe in order to believe one can still accomplish one's "intrinsic goods". Anders claims to have done CT charts for seven different people and has always been able to describe beliefs in CT-compliant ways. This is taken as "a moderate amount of evidence" for CT.
"Other"
Unfortunately for the remaining four types of evidence for CT, very little elaboration is provided; instead, the existence of results is simply declared. For example:
More distressingly, the existence of sociological explanations and philosophical arguments are not mentioned at all! Instead, there are promises that we'll see more, later...:
Objection #1: The Case For CT Is Very Incomplete and Inadequate
So what's wrong with CT? Well, first it's clear that a lot of the evidence provided for CT doesn't count. Obviously, we cannot accept the existence of "high-quality philosophical arguments" or the ability to "explain sociological phenomena" without that information actually being provided. Likewise, CT's "noteworthy results in [...] everyday lives" aren't actually documented -- we have to take Ander's word for it that these things occurred.
Mind Mapping and Recommendation Testing are more thorough, but still incomplete. For Mind Mapping, again Anders merely asserts that he's yet to find a belief that couldn't be explained in a CT-compliant way, though the beliefs surveyed and CT-compliant responses given are not listed.
Recommendation Testing is thus the only evidence for which usable data is given, however there is a lot of patchiness: Why were 38 predictions made left completely untested? What should we make of the 5 predictions ruled too difficult to confirm or deny? Why should 16 successes out of 60 predictions be taken as phenomenal results?
Objection #2: The Case For CT Uses Flawed Methodology
A second objection to CT is that the methodology used to produce CT is flawed. The gold standard for demonstrating a psychological theory is demonstrating explanatory power through a psychological experiment. For CT, no psychological experiments were provided and many confounding factors were left uncontrolled. For instance, in the Reccomendation Testing, one's improvement could be for a wide variety of reasons unrelated to the CT map.
The CT methodology also lends itself to significant issues of bias. The sample size in the Reccomendation Testing was only two participants -- the author of the study and someone closely related to the author. The participants could easily suffer from confirmation bias when constructing and using CT charts, guided by a conscious and/or subconscious wish for CT to be true.
A better standard for CT would be to evaluate over a dozen different charts of people who are unassociated with the author of the study and unaware of CT.
Objection #3: CT Conflicts With Established Science
Lastly, CT conflicts with established science. "Connection Theory: Current Evidence" concedes that documentation that beliefs can be changed from brain damage is one outstanding, yet to be investigated problem. Additional studies pile on in this direction to the point where postulating that people are ideally rational in all cases unrelated to their internal beliefs is quite implausible.
Between scope insensitivity, the affect heuristic, the anchoring effect, the priming effect, confirmation bias, clustering illusions, pareidolia, attentional biases, the availability heuristic, illusory correlations, the Dunning-Kuger effect, the Forer Effect, etc., enough cognitive biases have been documented to fill a book -- indeed, with Daniel Kahneman's Thinking Fast and Slow, they actually have! The myth of the rational person has been so thoroughly shattered in so many ways that don't ostensibly relate to intrinsic beliefs, that I don't think CT has much to stand on.
Conclusion
Thus CT should not be taken as a candidate for a psychological theory without significant improvement -- it is incomplete and inadequate, has flawed methodology, and conflicts well established science. For what it's worth, Leverage Research has had an appropriate lack of confidence in CT. None of their documentation asserts that CT is true, but rather makes it clear that the current evidence is circumstantial, and more data needs to be collected.
It's also worth noting that CT could be tested via replication, a process in which independent researchers seek to duplicate the results -- if the results can be duplicated, the theory is more likely to be true, and vice versa. Someone could follow the instructions and create a CT chart and see if it could be used to make accurate psychological predictions, especially under controlled circumstances. Personally, I don't want to invest the time needed to preform the replication until CT has stronger legs to stand on and has made an adequate case.
(Note: This was cross-posted from my blog.)