= 27d567bc2d48d9f40e9ccc20ea370b15)
Yvain comments on The usefulness of correlations - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (52)
Can you give some concrete examples of people who you think are making this mistake?
When I was starting out in trading I worked at a company where most of the traders were "spread traders" in the futures markets. They would trade either cash vs. futures or different futures expirations against each other. So, for instance, if you had future F1 that expired in september, and future F2 on the same underlying product that expired in december, they would define the spread between them F1-F2, and basically, try to buy that spread and sell it (or sell it and then buy it) over and over again. While F1 and F2 were whipping around, F1-F2 would tend to be pretty steady. The bid/ask spread of F1-F2 was determined by the volatility of F1 and F2, but the volatility of F1-F2 was much lower, so the bid/ask of F1-F2 was large compared to its volatility which is a recipe for juicy trading. So, anyway, these people wanted to me trade equities this way, but I had a super hard time with it. I would take two equities (E1 and E2) that were say 90% correlated, do the regression to get the ratio (r), and then start thinking about E1-rE2 as a spread just like I was used to. What I realized is that for 90% correlated instruments the spread volatility is 31% of the volatility of the naked instrument, which is still large compared bid/ask. The individual instruments that the futures traders were spreading were nearly 100% correlated, which is basically the requirement to have a spread market that you can reasonably talk about.
He already did:
This is a standard PCT criticism of psychology and social science, i.e., that these low correlation levels are an indicator that they're measuring the wrong things, and indeed using the wrong model of what kinds of things to measure.
(Specifically, the complaint is that those sciences assume organisms are open-loop stimulus-response reactors rather than closed-loop controllers, so they try to measure input-output correlations in their experiments, instead of using their experiments to identify the variables that organisms are able to control or tend to control.)
But social science doesn't respect a correlation of .6 because they think it's a good way to measure something that could be measured directly. They find correlations either as an important step in establishing causation, a way to get large-scale trends, or a good way to measure something that can't be measured directly.
The correlation between smoking and lung cancer is only .7, but that's a very interesting fact. True, just picking out smokers is a terrible way to predict who has lung cancer when compared to even a so-so screening test, which is what I interpreted the point of Richard's article as being. But knowing that there's a high correlation there is useful for other reasons. Since we now know it's causative, we can use it to convince people not to smoke. Even if we didn't know there was causation, it would at least help us to pick out who needs more frequent lung cancer screening tests.. So I am not prepared to immediately accept that someone is doing something wrong if they call a correlation of .6 pretty high.
Can you or Richard give an example of something the people investigating lung cancer could have done with direct measurement that would have been more productive than analyzing the cigarettes-smoking correlation? If not, can you provide a situation where people did overuse correlations when they'd have been better off using a measurement?
I wasn't aware that this was considered either psychology or social science; those are the fields towards which the criticism I pointed out was addressed, not medicine. (Medicine has a rather different set of statistics-based, politics-based, and payola-based errors to deal with.)
Correlation's a useful tool when that's all you have; the PCT criticism is that we now have more to go on than that where humans' and other organisms' behavior are concerned, so it's time to become dissatisfied with the old way and get started on improving things.
(Edit to add: WTF? This is the most baffling downvote I've ever seen OR received here, and I've seen some pretty weird ones in the past.)
Really, I'm not hostile to PCT, just skeptical— but given your claims about the predictive power of PCT, and given that it's been studied for 35 years by a large group including several former academics, I think it's fair to ask this: Can you direct me to an experiment such that
Note the importance of step 2. The results you've so far pointed out to me (can't find them within LW, sorry) concern a person manipulating a dial to keep a dot in the center of the screen while acted on by unknown, varying forces, and a rat varying the pressure on a lever it needs to hold down in response to varying counterforces. Since these are cases in which 'acting like a controller' is a simple strategy that produces near-optimal results, it doesn't surprise other theories of cognition that the agents arrived at this strategy. (I find it quite probable, in fact, that some form of control theory governs much of our motor impulses, since that's a fairly simple and elegant solution to recurring problems of balance, varying strain, etc.) The point where PCT really diverges from mainstream theories of cognition is in the description of cognitive content, not motor response; and that's where PCT's burden of proof lies.
If PCT is as well-developed across levels as you claim (and well-developed enough to make diagnoses and prescriptions for, say, emotional issues), then it should be easy to make and test such a prediction in a cognitive domain. If you can present me with an experiment that clearly meets those four conditions, I'll be very interested in whatever PCT book you recommend. If 30 years haven't produced such results, then that counts as evidence too.
'Standard theories of cognition' is a broad class that includes so many conflicting and open-ended models that I'm not sure I could come up with an experiment/experimental result pair that fulfills this requirement, even without the requirement that the experiment actually have that result.
That's a good point. I'll have to think carefully about what kind of results would constitute a "surprising" result to theories of mind that include basic modeling capacities and preferences in the usual fashion. Any good suggestions for emending requirement 2 would be appreciated.
And when you do, what you'll discover is that none of them really predict anything we don't already know about human behavior, or provide a reductionistic model of it.
What's different about PCT is that it gives us a framework for making and testing reductionist hypotheses about what is causing an individual's behavior. We can postulate variables they're controlling, do things to disturb the values of those variables, and observe whether the values are indeed being controlled by the person's behavior.
For example, if we want to know whether someone's "Bruce"-like behavior is due to a fear of success or a desire for failure, we could artificially induce success or failure experiences and observe whether they adjust their behavior to compensate.
Now try that with the standard cognitive theories, which will only give us ways to describe what the person actually does, or make probabilistic estimates about what people usually do in that situation, rather than any way to reduce or compress our description of the person's behavior, so that it becomes a more general predictive principle, instead of just a lengthy description of events.
OK, excellent; since you assert that PCT has so much more predictive power, I'm sure you can show me many impressive, quantitative PCT-driven experimental results that aren't in a domain (like motor response or game strategy) where I already expect to see control-system-like behavior.
For example, if you could get a mean squared error of 10% in predicting a response that balances ethical impulses against selfish ones (say, the amount that a person is willing to donate to a charity, given some sort of priming stimuli), then I'd consider that good evidence. That's the sort of result that would get me to pick up a PCT textbook.
Seriously, please point me to these results.
You've just crossed over two different definitions of "predictive" -- not to mention two different definitions of "science". What I described was something that would give you a "hard", strictly falsifiable fact: is the person controlling variable X or not?
That's actual science. But what you've asked for instead is precisely the sort of probabilistic mush that is being critiqued here in the first place. You are saying, "yes, it's all very well that science can be used to determine the actual facts, but I want some probabilities! Give me some uncertainty, dammit!"
And as a result, you seem to be under the mistaken impression that PCT has some sort of evidence deficiency I need to fix, when it's actually psychology that has a modeling deficiency that needs fixing. How about you show me a genuinely reductionistic (as opposed to merely descriptive) model of human psychology that's been proposed since Skinner?
I only mentioned PCT in this thread in the context of Yvain's request for an example of people making the mistake Richard wrote this post about. And you responded to my criticism of psychology (i.e., it's not a "hard" science) by raising criticisms of PCT that are in fact off-topic to the discussion at hand.
Are you claiming that, if PCT is flawed, then everything in psychology is just jim-dandy fine? Because that's a pretty ludicrous position. Check your logic, and address the topic actually at hand: the complete failure of cognitive-level psychology to come up with a halfway decent reduction of human behavior, instead of just cataloging examples of it.
Otherwise, you are in the exact same position as an intelligent-design advocate pretending that gaps in evolutionary biology mean you don't have to consider the gaps in your own theory, or lack thereof.
Because PCT could be ludicrously wrong, and it would still be a huge advance in the current state of psychology to be able to nail down with any precision why or how it was wrong.
Which is why critique of PCT is irrelevant to this topic: you could disprove PCT utterly, and the given criticism of psychology would still stand, just like disproving evolution wouldn't make "God did it" any more plausible or useful of a theory.
So let's say, for the sake of argument, that I utterly recant of PCT and say it's all gibberish. How would that improve the shoddy state of psychology in the slightest? What would you propose to replace PCT as an actual model of human innards?
Let's hear it. Name for us the very best that modern psychology has given us since Skinner, of any attempt to actually define an executable model of human behavior. Has anyone even tried, who wasn't an outsider to the field?
A correlation of 0.6 is a bad measurement, period. It does not become a good one for want of a better.
I don't know what you mean by "analysing" a correlation, but this is some of what they did do.
I could have mentioned epidemiology in my intro. The reason it depends on statistics is that it is often much more difficult to discern the actual mechanism of a disease process than to do statistical studies. Googling turns up this study which is claimed (by the scientist doing the work) to be the very first demonstration of a causal link between smoking and lung cancer -- in April of this year (and not the 1st of the month).
But the correlations remain what they are, and it still takes a lot of work to get somewhere with them.
A bad measurement can still be the best there is.
But it is useful. I think Yvain asked the wrong question. You can do better than correlations, but do you deny that you can draw from them the conclusions that Yvain does? (ie, the population effect of smoking)
The MN scientist is lying. No, I didn't click on the link. Yes, I mean lying, not mistaken.
The conclusion he draws is:
Sure, standard statistics. No problem, for want of anything better.
On the other hand, if you want to know how the link between smoking and lung cancer works, the epidemiology can do no more than suggest places to look.
On closer reading, the actual scientific claim is less than I thought. It's a statistical study correlating the presence of a nitrosamine compound in the urine with lung cancer, and finding a higher correlation than with self-reported smoking. Original paper (full text requires subscription) here and blogged here. So just more statistical epidemiology and not at all epoch-making.
ETA: Extra links, just because these things are worth knowing.
As pjeby points out, I gave the examples of "psychology and social science". Look at reports that summarise statistical results by claims of the form "X's are Y's", sometimes by the scientists themselves, not journalists. If you want something more concrete than those generalities, see the context of this comment.