= 27d567bc2d48d9f40e9ccc20ea370b15)
Yvain comments on The usefulness of correlations - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (52)
But social science doesn't respect a correlation of .6 because they think it's a good way to measure something that could be measured directly. They find correlations either as an important step in establishing causation, a way to get large-scale trends, or a good way to measure something that can't be measured directly.
The correlation between smoking and lung cancer is only .7, but that's a very interesting fact. True, just picking out smokers is a terrible way to predict who has lung cancer when compared to even a so-so screening test, which is what I interpreted the point of Richard's article as being. But knowing that there's a high correlation there is useful for other reasons. Since we now know it's causative, we can use it to convince people not to smoke. Even if we didn't know there was causation, it would at least help us to pick out who needs more frequent lung cancer screening tests.. So I am not prepared to immediately accept that someone is doing something wrong if they call a correlation of .6 pretty high.
Can you or Richard give an example of something the people investigating lung cancer could have done with direct measurement that would have been more productive than analyzing the cigarettes-smoking correlation? If not, can you provide a situation where people did overuse correlations when they'd have been better off using a measurement?
I wasn't aware that this was considered either psychology or social science; those are the fields towards which the criticism I pointed out was addressed, not medicine. (Medicine has a rather different set of statistics-based, politics-based, and payola-based errors to deal with.)
Correlation's a useful tool when that's all you have; the PCT criticism is that we now have more to go on than that where humans' and other organisms' behavior are concerned, so it's time to become dissatisfied with the old way and get started on improving things.
(Edit to add: WTF? This is the most baffling downvote I've ever seen OR received here, and I've seen some pretty weird ones in the past.)
Really, I'm not hostile to PCT, just skeptical— but given your claims about the predictive power of PCT, and given that it's been studied for 35 years by a large group including several former academics, I think it's fair to ask this: Can you direct me to an experiment such that
Note the importance of step 2. The results you've so far pointed out to me (can't find them within LW, sorry) concern a person manipulating a dial to keep a dot in the center of the screen while acted on by unknown, varying forces, and a rat varying the pressure on a lever it needs to hold down in response to varying counterforces. Since these are cases in which 'acting like a controller' is a simple strategy that produces near-optimal results, it doesn't surprise other theories of cognition that the agents arrived at this strategy. (I find it quite probable, in fact, that some form of control theory governs much of our motor impulses, since that's a fairly simple and elegant solution to recurring problems of balance, varying strain, etc.) The point where PCT really diverges from mainstream theories of cognition is in the description of cognitive content, not motor response; and that's where PCT's burden of proof lies.
If PCT is as well-developed across levels as you claim (and well-developed enough to make diagnoses and prescriptions for, say, emotional issues), then it should be easy to make and test such a prediction in a cognitive domain. If you can present me with an experiment that clearly meets those four conditions, I'll be very interested in whatever PCT book you recommend. If 30 years haven't produced such results, then that counts as evidence too.
'Standard theories of cognition' is a broad class that includes so many conflicting and open-ended models that I'm not sure I could come up with an experiment/experimental result pair that fulfills this requirement, even without the requirement that the experiment actually have that result.
That's a good point. I'll have to think carefully about what kind of results would constitute a "surprising" result to theories of mind that include basic modeling capacities and preferences in the usual fashion. Any good suggestions for emending requirement 2 would be appreciated.
And when you do, what you'll discover is that none of them really predict anything we don't already know about human behavior, or provide a reductionistic model of it.
What's different about PCT is that it gives us a framework for making and testing reductionist hypotheses about what is causing an individual's behavior. We can postulate variables they're controlling, do things to disturb the values of those variables, and observe whether the values are indeed being controlled by the person's behavior.
For example, if we want to know whether someone's "Bruce"-like behavior is due to a fear of success or a desire for failure, we could artificially induce success or failure experiences and observe whether they adjust their behavior to compensate.
Now try that with the standard cognitive theories, which will only give us ways to describe what the person actually does, or make probabilistic estimates about what people usually do in that situation, rather than any way to reduce or compress our description of the person's behavior, so that it becomes a more general predictive principle, instead of just a lengthy description of events.
OK, excellent; since you assert that PCT has so much more predictive power, I'm sure you can show me many impressive, quantitative PCT-driven experimental results that aren't in a domain (like motor response or game strategy) where I already expect to see control-system-like behavior.
For example, if you could get a mean squared error of 10% in predicting a response that balances ethical impulses against selfish ones (say, the amount that a person is willing to donate to a charity, given some sort of priming stimuli), then I'd consider that good evidence. That's the sort of result that would get me to pick up a PCT textbook.
Seriously, please point me to these results.
You've just crossed over two different definitions of "predictive" -- not to mention two different definitions of "science". What I described was something that would give you a "hard", strictly falsifiable fact: is the person controlling variable X or not?
That's actual science. But what you've asked for instead is precisely the sort of probabilistic mush that is being critiqued here in the first place. You are saying, "yes, it's all very well that science can be used to determine the actual facts, but I want some probabilities! Give me some uncertainty, dammit!"
And as a result, you seem to be under the mistaken impression that PCT has some sort of evidence deficiency I need to fix, when it's actually psychology that has a modeling deficiency that needs fixing. How about you show me a genuinely reductionistic (as opposed to merely descriptive) model of human psychology that's been proposed since Skinner?
I only mentioned PCT in this thread in the context of Yvain's request for an example of people making the mistake Richard wrote this post about. And you responded to my criticism of psychology (i.e., it's not a "hard" science) by raising criticisms of PCT that are in fact off-topic to the discussion at hand.
Are you claiming that, if PCT is flawed, then everything in psychology is just jim-dandy fine? Because that's a pretty ludicrous position. Check your logic, and address the topic actually at hand: the complete failure of cognitive-level psychology to come up with a halfway decent reduction of human behavior, instead of just cataloging examples of it.
Otherwise, you are in the exact same position as an intelligent-design advocate pretending that gaps in evolutionary biology mean you don't have to consider the gaps in your own theory, or lack thereof.
Because PCT could be ludicrously wrong, and it would still be a huge advance in the current state of psychology to be able to nail down with any precision why or how it was wrong.
Which is why critique of PCT is irrelevant to this topic: you could disprove PCT utterly, and the given criticism of psychology would still stand, just like disproving evolution wouldn't make "God did it" any more plausible or useful of a theory.
So let's say, for the sake of argument, that I utterly recant of PCT and say it's all gibberish. How would that improve the shoddy state of psychology in the slightest? What would you propose to replace PCT as an actual model of human innards?
Let's hear it. Name for us the very best that modern psychology has given us since Skinner, of any attempt to actually define an executable model of human behavior. Has anyone even tried, who wasn't an outsider to the field?
I'll give this one last try.
You've given me the two results I mentioned above, in the area of motor response. They sound like good experiments to me: you can take a model with relatively few free parameters, and find that most subjects' behavior will fit that model extremely well for some particular values of the parameters. That is the kind of experiment I'd take as good evidence that control theory operates in motor response. (Incidentally, if you could give me a link to those experiments, I'd much appreciate it.)
You've been claiming for months that this is just the tip of the iceberg, that PCT is able to isolate variables that subjects are controlling in cognitive contexts like belief. I would be very interested in this claim if I saw some evidence for it; fortunately, your claim that PCT is able to diagnose and treat cognitive conditions implies that it's strong enough to do the same kind of experiments as in the case of motor response. So I began by asking for references to such results, and gave an example of the kind of result that would definitely move me to look into PCT.
Experimental verification seems to me like the obvious thing for PCT advocates to do if they're confident in their theory and frustrated by its lack of academic respect. I would therefore find it highly unlikely, given that your claims are true, that in 35 years there hasn't been a single positive experimental result in a cognitive context, of the same form as the "controlling the position of the dot" or "varying the force on the bar" experiments. That you meet my question with outrage, rather than with citations, is thus Bayesian evidence against the validity of PCT.
Nope. I'm just claiming that if PCT doesn't have the kind of evidence it claims, then I probably shouldn't bother investigating it. The problems with mainstream psychology are manifold, but the discipline seems to be making (slow) progress by scientific criteria: Tversky and Kahneman, for instance, were making novel and unexpected experimental predictions that turned out to be correct. If your discipline does that much better than mainstream psychology, there should be some strong experimental results that show it.
I really can't imagine that's too much to ask, and that's why I've made this challenge. Point me to experimental results that validate PCT in a cognitive context, and I'll pick up the textbook of your choice. Keep grandstanding against the very kind of evidence you presented as evidence of PCT in motor response, and I'll have to conclude that you're peddling woo-woo. The ball is in your court.
I take it you skipped reading Marken's references then, since I believe one of the cited papers was on how physicians' errors in prescribing medicine match a PCT model of the situation, but fail to match an intuitive model of how such errors would respond to environmental changes.
You've apparently also been ignoring my repeated mention of time-averaged perceptual variables like "the amount of work I've done today" or "how recently I got laid" - you can have a "feel" for such values, and how they change over time, as well as respond to changes in them. Do you claim to not perceive -- and control -- such variables? Or are you going to say that since "work" and "getting laid" involve physical activity, they are somehow therefore "motor" rather than "cognitive"?
Finally, you seem to have put me in the strange position of a passing physics student being harangued by a young earth creationist, insisting that I prove the age of the universe to your satisfaction, before you will study any physics, whereas I assert that if you were to go and study some physics, it will be obvious to you why YEC-ism is wrong.
But after being harangued at some length, I relent and attempt to begin with some basic equations, which you then argue are not in the Bible and thus not valid evidence. It is at this point, I begin to question who you're trying to convince by your diatribe, and why, if you genuinely want to learn something, why you're spending more time writing than reading. Don't you have a library anywhere near you?
I'm not sure I follow you here, since I've only referred to neuroanatomy evidence -- i.e. evidence from a "hard" science. You may be confusing me with one of the other PCTers here who've been talking about the joystick perception experiments, which I consider only relevant for debunking Skinner... which isn't really as useful as it used to be.
A correlation of 0.6 is a bad measurement, period. It does not become a good one for want of a better.
I don't know what you mean by "analysing" a correlation, but this is some of what they did do.
I could have mentioned epidemiology in my intro. The reason it depends on statistics is that it is often much more difficult to discern the actual mechanism of a disease process than to do statistical studies. Googling turns up this study which is claimed (by the scientist doing the work) to be the very first demonstration of a causal link between smoking and lung cancer -- in April of this year (and not the 1st of the month).
But the correlations remain what they are, and it still takes a lot of work to get somewhere with them.
A bad measurement can still be the best there is.
But it is useful. I think Yvain asked the wrong question. You can do better than correlations, but do you deny that you can draw from them the conclusions that Yvain does? (ie, the population effect of smoking)
The MN scientist is lying. No, I didn't click on the link. Yes, I mean lying, not mistaken.
The conclusion he draws is:
Sure, standard statistics. No problem, for want of anything better.
On the other hand, if you want to know how the link between smoking and lung cancer works, the epidemiology can do no more than suggest places to look.
On closer reading, the actual scientific claim is less than I thought. It's a statistical study correlating the presence of a nitrosamine compound in the urine with lung cancer, and finding a higher correlation than with self-reported smoking. Original paper (full text requires subscription) here and blogged here. So just more statistical epidemiology and not at all epoch-making.
ETA: Extra links, just because these things are worth knowing.