You're looking at Less Wrong's discussion board. This includes all posts, including those that haven't been promoted to the front page yet. For more information, see About Less Wrong.

Lumifer comments on In Defence of Simple Ideas That Explain Everything But Are Wrong - Less Wrong Discussion

8 Post author: johnlawrenceaspden 22 March 2016 03:46PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (52)

You are viewing a single comment's thread. Show more comments above.

Comment author: Lumifer 31 March 2016 08:35:19PM *  1 point [-]

Well, fine, but in order to get two p<0.05 results by chance if there's no effect, there'd have had to be roughly forty failed experiments done and never mentioned. Is that likely?

First, that forty number comes from the spherical-cow land where everything is independent, normally distributed, gardens of forking paths do not exist, etc. etc. Look at the replication crisis in psychology which has been getting a lot of press recently. They have dozens of papers showing highly significant results for some effect which, as it turns out now, does not exist.

Second, consider the incentives. Say, you ran a small trial, got zero results, what are you going to do with it? No journal will be very excited about the "we tried a weird thing and it didn't work" paper. Or, say, you got negative results, your patients started dying. Would you be terribly interested in writing up "we tried a weird thing and ended up killing some people" results?

Wouldn't there?

You need to convince not me, but people who are good at writing grant proposals and passing ethics boards :-/

Comment author: johnlawrenceaspden 31 March 2016 09:17:59PM *  0 points [-]

You need to convince not me, but people who are good at writing grant proposals and passing ethics boards :-/

Sure, but I have now managed to convince myself, so I am practising. Please be the most evil opponent you can be!

Second, consider the incentives.

Incentives-wise, doctors are regularly getting struck off for believing and practising this, all over the world. And the NICE guidelines specifically say that thyroxine is not to be used for the treatment of CFS. You'd think they'd be overjoyed to have a reference to quote when striking people off/writing guidelines.

Would you be terribly interested in writing up "we tried a weird thing and ended up killing some people" results?

Well, I think that's exactly what the Scottish GPs thought they were writing up! They found 'no difference from placebo' in their patient group, and 'harmful' in their control group.

Where we differ is that I think that means that they must actually have done a fair bit of good in their patient group (specifically, in that portion of their patient group who actually had type 2 hypothyroidism, and for whom 100mg thyroxine/day was roughly the right amount. According to Skinner, that would have been too much for many of them, and too little for many others).