= 27d567bc2d48d9f40e9ccc20ea370b15)
ChristianKl comments on Problems with Academia and the Rising Sea - Less Wrong Discussion
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (39)
I think a core problem is the way scientists cite papers. There no real reason why one should always cite the first paper that makes a given claim.
You could change that habit and instead cite the paper that does the first replication.
that's... actually a really good idea.
somehow incentivising a rule of always citing a replication of an experiment would make a massive difference.
I don't think it about incentives. It's roughly the same matter of work to cite a replication than the original experiment. It's rather about convincing the scientists that it's a hallmark of good science to cite the replication.
You start by convincing the cool people. Let them signal that they are cool by citing replications.
Afterwards you go to a journal and ask: "Do you support good science? If so, please add a rule to your paper review process that submitters have to cite first replications instead of the original paper."
Or both.
The goal of the endeveour is to move academic resources from publishing out-there-research to verifiable research. The best way to do that is to reduce the incentives for out-there-research and increase those for verification.
Sometimes it might make sense to cite every study on a given topic. The default should be to cite the first replication or a meta-analysis when available. If I'm a reader of an academic paper I profit from reading the paper that provides the evidence that the claim is actually true.
There just no good reason to give out prestige to the first person who finds an effect.
In short term, we might want less research and more verification. But in long term, we don't want to discourage the research completely.
Also, people will always try to game the system. How will you get points for the first verification of the original idea, when the original ideas become scarce? Split the work with your friend: one of you will write the original article, another will write the verification article; then publish both at the same time. Would that be a big improvement? I would prefer the verification to be done by someone who is not a friend, and especially does not owe a favor for getting such opportunity.
Note: In ice hockey, players get points for goals and assists. Sure, it's not completely the same situation, but it is a way to encourage two behaviors that are both needed for a success. In science we need to encourage both research and verification.
Yes, two people getting a P<0.05 and both setting up the experiment are better than one.
I don't believe that original ideas become scarce. Researching original ideas is fun. There will be still grant giving agencies that give out grands to persue original ideas.
How many false-positives get published as opposed to negatives (or rarely even false-negatives)? If the ratio is too high then you'll need more than just two positive studies. If, as claimed in the quoted article, 70% of papers in a field are not reproducible that implies finding papers at random would require about nine positive studies to reach a true P<0.05, and that's only if each paper is statistically independent from the others.
If there is financial incentive to reproduce existing studies when there is a ready-made template to copy and paste into a grant-funding paper I think the overall quality of published research could decline. At least in the current model there's a financial incentive to invent novel ideas and then test them, versus just publishing a false reproduction.
People cite several if they can. But they do try to cite the first.
Textbooks replace each other on clarity of explanation as well as adherence to modern standards of notation and concepts.
Maybe just cite the version of an experiment that explains it the best? Replications have a natural advantage because you can write them later when more of the details and relationships are worked out.