someonewrongonthenet comments on This is why we can't have social science - Less Wrong

36 Post author: Costanza 13 July 2014 09:04PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (82)

You are viewing a single comment's thread. Show more comments above.

Comment author: someonewrongonthenet 14 July 2014 02:54:06AM *  3 points [-]

I sort of side with Mitchel on this.

A mentor of mine once told me that replication is useful, but not the most useful thing you could be doing because it's often better to do a followup experiment that rests on the premises established by the initial experiment. If the first experiment was wrong, the second experiment will end up wrong too. Science should not go even slower than it already does - just update and move on, don't obsess.

It's kind of how some of the landmark studies on priming failed to replicate, but there are so many followup studies which are explained by priming really well that it seems a bit silly to throw out the notion of priming just because of that.

Keep in mind, while you are unlikely to hit statistically significance where there is no real result, it's not statistically unlikely to have a real result that doesn't hit significance the next time you do it. Significance tests are attuned to get false negatives more often than false positives.

Emotionally though... when you get a positive result in breast cancer screening even when you're not at risk, you don't just shrug and say "probably a false positive" even though it is. Instead, you irrationally do more screenings and possibly get a needless operation. Similarly, when the experiment fails to replicate, people don't shrug and say "probably a false negative", even though that is, in fact, very likely. Instead, they start questioning the reputation of the experimenter. Understandably, this whole process is nerve wracking for the original experimenter. Which I think is where Mitchel was - admittedly clumsily - groping towards with the talk of "impugning scientific integrity".

Comment author: Lumifer 28 August 2014 07:24:27PM 1 point [-]

If the first experiment was wrong, the second experiment will end up wrong too.

...and now you have two problems X-)

Science should not go even slower than it already does

It's not a matter of speed, it's a matter of velocity. Going fast in the wrong direction is (much) worse than useless.

you are unlikely to hit statistically significance where there is no real result

You are quite likely. You start with a 5% chance under ideal circumstances and that chance only climbs from there. P-hacking is very widespread.

Instead, you irrationally do more screenings

8-0 You think getting additional screenings after testing positive for cancer is "irrational"??

Comment author: someonewrongonthenet 28 August 2014 07:53:56PM -1 points [-]

The process of screening itself involves risks, not to mention the misplaced stress and possibility of unnecessary surgery.

Comment author: Lumifer 28 August 2014 09:17:04PM 1 point [-]

This is true for e.g. any visit to the doctor. Are you saying that it's irrational to go for medical checkups?

In the cancer screening case, what do you think does the cost-benefit analysis say?

Comment author: someonewrongonthenet 28 August 2014 09:55:04PM *  -1 points [-]

It would be irrational to go for medical check ups when they aren't necessary - if you did it every 3 days, for example.

I'm looking at this from a birds eye view. A lot of people get unnecessary screenings, which tell them information which is not worth acting upon no matter whether it says that it is positive or negative, and then start worrying and getting unnecessary testing and treatment. Information is only useful to the extent that you can act upon it.

Comment author: Lumifer 29 August 2014 06:01:51AM *  1 point [-]

I'm looking at this from a birds eye view.

And from up there you take it upon yourself to judge whether personal decisions are rational or not? I think you're way too far away for that.

A lot of people get unnecessary screenings

That's a different issue. In a post upstream you made a rather amazing claim that additional tests after testing positive for cancer on a screening would be irrational. Do you stand by that claim?

Comment author: someonewrongonthenet 30 August 2014 03:22:06PM -1 points [-]

And from up there you take it upon yourself to judge whether personal decisions are rational or not? I think you're way too far away for that.

Er...I think that's a little harsh of you. Overscreening is recognized as a problem among epidemiologists. When I say overscreening is a problem, I'm mostly just trusting expert consensus on the matter.

That's a different issue. In a post upstream you made a rather amazing claim that additional tests after testing positive for cancer on a screening would be irrational. Do you stand by that claim?

I stand by that a lot of smart people who study this issue believe that in actual medical practice, these screenings are either a problem in themselves, or that the information from the screenings can lead people to irrational behavior, and I do trust them.

But really, that was just an illustrative example used to steelman Michael. You don't have to except the actual example, just the general concept that this sort of thing can happen.

Comment author: Lumifer 02 September 2014 06:38:45PM *  1 point [-]

Overscreening is recognized as a problem among epidemiologists.

Rationality does not specify values. I rather suspect that the cost-benefit analysis that epidemiologists look at is quite different from the cost-benefit analysis that individuals look at.

these screenings are either a problem in themselves, or that the information from the screenings can lead people to irrational behavior

LOL. Don't bother you pretty little head with too much information. No, you don't need to know that. No, you can't decide what you need to know and what you don't need to know. X-/