I don't spend enough of my time reading the results of studies that you should necessarily pay much attention to what I think. But: you want to know what information it gives you that the study found (say) a trend with p=0.1, given that the authors may have been looking for such a trend and (deliberately or not) data-mining/p-hacking and that publication filters out most studies that don't find interesting results.
So here's a crude heuristic:
You might (might!) want to apply a bit less discounting in cases where the result doesn't seem like one the researchers would have been expecting or wanting, and/or doesn't substantially enhance the publishability of the paper, because such results are less likely to be produced by the usual biases. E.g., if that p=0.1 trend is an incidental thing they happen to have found while looking at something else, you maybe don't need to treat it as zero evidence.
This is likely to leave you with lots of little updates. How do you handle that given your limited human brain? What I do is to file them away as "there's some reason to suspect that X might be true" and otherwise ignore it until other evidence comes along. At some point there may be enough evidence that it's worth looking properly, so then go back and find the individual bits of evidence and make an explicit attempt to combine them. Until then, you don't have enough evidence to affect your behaviour much so you should try to ignore it. (In practice it will probably have some influence, and that's probably OK. Unless it's making you treat other people badly, in which case I suggest that the benefits of niceness probably outweigh those of correctness until the evidence gets really quite strong.)
Thank you! This is exactly what I was looking for. Thinking in terms of bits of information is still not quite intuitive to me, but it seems the right way to go. I've been away from LW for quite a while and I forgot how nice it is to get answers like this to questions.
If it's worth saying, but not worth its own post (even in Discussion), then it goes here.
Notes for future OT posters:
1. Please add the 'open_thread' tag.
2. Check if there is an active Open Thread before posting a new one. (Immediately before; refresh the list-of-threads page before posting.)
3. Open Threads should be posted in Discussion, and not Main.
4. Open Threads should start on Monday, and end on Sunday.