You're still ignoring the difference between a failed experiment and a failed replication. Edison did not publish 999 papers each of them claiming that this is the way to build the lightbulb (at p=0.05).
So? What does this have to do with my point about optimizing return from experimentation?
And what exactly prevents the researchers from considering the prior odds when they are trying to figure out whether their results are really statistically significant?
Nothing. But no one does that because to point out that a normal experiment has resulted in a posterior probability of <5% is not helpful since that could be said of all experiments, and to run a single experiment so high-powered that it could single-handedly overcome the prior probability is ludicrously wasteful. You don't run a $50m clinical trial enrolling 50,000 people just because some drug looks interesting.
I disagree with you -- if a researcher consistently publishes research that cannot be replicated I will call him a bad researcher.
Too bad. You should get over that.
I think our disagreement comes (at least partially) from the different views on what does publishing research mean.
I see your position as looking on publishing as something like "We did A, B, and C. We got the results X and Y. Take it for what it is. The end."
I'm looking on publishing more like this: "We did multiple experiments which did not give us the magical 0.05 number so we won't tell you about them. But hey, try #39 succeeded and we can publish it: we did A39, B39, and C39 and got the results X39 and Y39. The results are significant s...
For those who haven't heard, NIH and NSF are no longer processing grants, leading to many negative downstream effects.
I've been directing my attention elsewhere lately and don't have anything informative to say about this. However, my uninformed intuition is that people who care about effective altruism (research in general, infrastructure development, X-risk mitigation, life-extension...basically everything, actually) or have transhumanist leanings should be very concerned.
The consequences have already been pretty disastrous. To provide just one, immediate example, the article says that the Center for Disease Control and Prevention has shut down. I think that this is almost certain to directly cause a nontrivial number of deaths. Each additional day that this continues could have huge negative impact down the line, perhaps delaying some key future discoveries by years. This event *might* be a small window of opportunity to prevent a lot of harm very cheaply.
So the question is:
1) Can we do anything to remedy the situation?
2) If so, is it worth doing it? (Opportunity costs, etc)