I feel this would be very necessary for the general idea of an archive of refuted research to be effective. I say this primarily because even in the science and engineering domain (with which I am more familiar, as opposed to medicine and the soft sciences) there are many degrees of possible wrongness. When a field is just opening up, it is inevitable and even okay for people to pulish things that are a bit wrong. We know that the Cosmological Constant isn't "right" but we don't call Einstein's original papers "refuted."
Someone may say, "That's fine, we will only put research that is meaningfully refutable in this archive," but then you invite endless argument about whether string theory is refutable, etc., not to mention providing a platform for researchers with vendettas and ulterior agendas to invalidate the research of rivals through sheer nitpicking and willful misrepresentation.
Someone may say, "That's fine, we will only put research that is meaningfully refutable in this archive," but then you invite endless argument about whether string theory is refutable,
That's why I mostly just concentrated on medicine, where you can simply ask whether replication studies on "thing X helps people" manage to reproduce the original result or not. Yes, there's room for interpretation and disagreement even here, but hopefully less than for "is string theory refutable".
Once such an archive existed, medical journals could adopt a policy of checking all citations in a proposed article against the archive, rejecting submissions that tried to cite refuted research as valid.
The problem with this is that power corrupts. Medical science is already suffering from group think problems this would likely make it worse.
Nancy mentioned Polifact here, I'm pretty sure I've seen controversy over some of their pronouncements. In any case I'm sure they'd be much worse if they had actual gate-keeping power.
This is quite true, but it doesn't seem obvious that it'd be worse than incorrect results being cited for decades after being refuted.
While I agree in principle that this could be useful, still I wonder -- researchers that spend months, years on some subject are apparently unable to double-check the shoulders they stand on -- would this really make a difference?
Now, if the respected journals would automatically reject any paper citing a refuted paper as support, that might bring about the needed cultural change...
What do you mean? As I understand it, LexisNexis is just a database of legal articles. PubMed is a similar database of medical articles. PM lacks the full text of the articles and even the bibliographies, but it seems to me adequate for this purpose. The difference seems to me to be a matter of culture, of making a point of doing the search. Also, the hierarchical nature of law makes it easy to tell if one opinion trumps another.
Math Reviews is a database of journal articles that includes complete bibliographies for the recent articles. I suppose that must require a deal with the journals, but I wonder why PubMed didn't make the same deal? Google Scholar is an automatically generated citation database.
this could be a good project, any takers. We need to collect the information and put it all together on 1 site, we don't have to release our own studies.
In some fields most of the studies are refuted within 2 years of publishing, i wonder if it might be better to include any and all studies we can find and then rate them on a reliability scale like this one: http://www.cebm.net/index.aspx?o=1025 / http://chronopause.com/index.php/2011/08/12/interventive-gerontology-101-01-the-basics/ .
that scale might not work for all fields but we can probably think of a scale that does work. And then say any rating below X should be never be cited.
I do not however have a solution for how to become a respected agency.
I do not however have a solution for how to become a respected agency.
If journals start to reject publishing researched because it relies on 'poor' citations, that should have the effect of making this proposed archive-of-study-quality respected.
So maybe a more specific question: how could we get journals to use this archive as part of their review process?
The obvious method is to start with people who are credentialed in the fields to join in. If you have relevant PhDs who are working on this people will likely pay a lot more attention. Maybe talk to people are the local university, see if you can get them interested.
From The Atlantic's Lies, Damned Lies and Medical Science:
Here's a suggested solution to the problem of refuted research still being cited. Have some respected agency maintain an archive of studies that have failed to replicate or that have otherwise been found lacking. Once such an archive existed, medical journals could adopt a policy of checking all citations in a proposed article against the archive, rejecting submissions that tried to cite refuted research as valid. This might also alleviate the problem of people not doing replications because replications don't get many cites. Once such an archive was established, getting your results there might become quite prestigious.
The one major problem that I can see with this proposal is that it's not always obvious when a study should be considered refuted. But even erring on the side of only including firmly refuted studies should be much better than nothing at all.
Such a fix seems obvious and simple to me, and while maintaining the archive and keeping it up to date would be expensive, it should be easily affordable for an organization such as a major university or the NIH. Similar archives could also be used for fields other than medicine. Is there some reason that I'm missing for why this isn't done?