= 27d567bc2d48d9f40e9ccc20ea370b15)
DanielLC comments on This is why we can't have social science - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (82)
I sort of side with Mitchel on this.
A mentor of mine once told me that replication is useful, but not the most useful thing you could be doing because it's often better to do a followup experiment that rests on the premises established by the initial experiment. If the first experiment was wrong, the second experiment will end up wrong too. Science should not go even slower than it already does - just update and move on, don't obsess.
It's kind of how some of the landmark studies on priming failed to replicate, but there are so many followup studies which are explained by priming really well that it seems a bit silly to throw out the notion of priming just because of that.
Keep in mind, while you are unlikely to hit statistically significance where there is no real result, it's not statistically unlikely to have a real result that doesn't hit significance the next time you do it. Significance tests are attuned to get false negatives more often than false positives.
Emotionally though... when you get a positive result in breast cancer screening even when you're not at risk, you don't just shrug and say "probably a false positive" even though it is. Instead, you irrationally do more screenings and possibly get a needless operation. Similarly, when the experiment fails to replicate, people don't shrug and say "probably a false negative", even though that is, in fact, very likely. Instead, they start questioning the reputation of the experimenter. Understandably, this whole process is nerve wracking for the original experimenter. Which I think is where Mitchel was - admittedly clumsily - groping towards with the talk of "impugning scientific integrity".
If you get a positive result, you run another test. If you keep getting positive results, you probably have breast cancer.
Similarly, if an experiment fails to replicate, you try again. If it replicates this time, then it's probably fine. If it keeps failing to replicate, then there's a problem.
At the very least, you need to try to replicate a random sample of studies, just to make sure there aren't more false studies than you've been assuming.
Not an expert on cancer, but I don't think it works that way .I think the cancer test accurately tests a variable wihch is a proxy for cancer risk. So a patient who doesn't have cancer but tests positive will continue testing positive, because the variable that the cancer test measures as a proxy for cancer is elevated in that patient.
Experiments do work that way, but I'm not arguing against that. I'm only arguing that direct replication isn't a better use of resources than just going on to a followup experiment with a different methodology (unless direct replication is really easy and you can just have some students do it or something).
Is there only one kind of test? Couldn't they find another proxy?
If the followup is testing the same thing with a different methodology, then the metaphor works. If you run followup experiments just to find more detail, it would be like someone testing positive for cancer so then you run a test for what kind of cancer. You're assuming they have cancer when you run the second test, so the results could be misleading.
Generally an idea is considered well supported when multiple methodologies support it, yes. In the psychology lab I used to work in, at least, we never try to replicate, but we do try to show the same thing in multiple different ways. There are maybe 15 different experiments a year, but they're generally all centered around proving or dis-proving a cluster of 2 or 3 broad, conceptually linked hypotheses.
Biology labs I've worked with do often do the whole "okay, the results are in and this is established now, let's find additional detail' thing, but that's because they were usually looking at much simpler systems, like a single protein or something, so they could afford to take liberties and not be so paranoid about experimental methods.