gjm comments on Open thread, Nov. 16 - Nov. 22, 2015 - Less Wrong Discussion
You are viewing a comment permalink. View the original post to see all comments and the full post content.
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (185)
I don't spend enough of my time reading the results of studies that you should necessarily pay much attention to what I think. But: you want to know what information it gives you that the study found (say) a trend with p=0.1, given that the authors may have been looking for such a trend and (deliberately or not) data-mining/p-hacking and that publication filters out most studies that don't find interesting results.
So here's a crude heuristic:
You might (might!) want to apply a bit less discounting in cases where the result doesn't seem like one the researchers would have been expecting or wanting, and/or doesn't substantially enhance the publishability of the paper, because such results are less likely to be produced by the usual biases. E.g., if that p=0.1 trend is an incidental thing they happen to have found while looking at something else, you maybe don't need to treat it as zero evidence.
This is likely to leave you with lots of little updates. How do you handle that given your limited human brain? What I do is to file them away as "there's some reason to suspect that X might be true" and otherwise ignore it until other evidence comes along. At some point there may be enough evidence that it's worth looking properly, so then go back and find the individual bits of evidence and make an explicit attempt to combine them. Until then, you don't have enough evidence to affect your behaviour much so you should try to ignore it. (In practice it will probably have some influence, and that's probably OK. Unless it's making you treat other people badly, in which case I suggest that the benefits of niceness probably outweigh those of correctness until the evidence gets really quite strong.)
Thank you! This is exactly what I was looking for. Thinking in terms of bits of information is still not quite intuitive to me, but it seems the right way to go. I've been away from LW for quite a while and I forgot how nice it is to get answers like this to questions.