Insightful, accurate and depressing.
On a related note:
Ben Goldacre along with colleagues has recently set up COMPare.
For the last few years it’s been the norm that research in humans should be preregistered before the trial starts to avoid the file-drawer effect where negative trials don’t get published.
Without preregistration it’s hard to tell when someone has thrown a dart at a wall then built the dartboard around it.
It’s gradually been improving with a lot of research being preregistered but still published trials often don’t report what they said they were going to report or report things they didn’t preregister.
The dartboard is now there beforehand but people are still quietly building new dartboards around wherever the dart hits without mentioning it or mentioning that they were aiming at the original dartboard.
The COMPare project is doing something incredibly simple: Reading the paper. Reading the preregistered plan. Posting a public note on their website and sending a letter to the publishing journal pointing it out.
It’s embarrassing for journals because in theory they should have made sure that the papers matched what was preregistered during peer review.
They’ve had a range of responses, some journal editors like the ones at the BMJ have posted corrections while others have doubled-down like the editors at Annals of Internal Medecine, it’s really quite entertaining.
I would regard projects like COMPare, which rate studies after publication, as much more valuable than preregistration. Yes, preregistration reduces researcher degrees of freedom, but it also increases red tape. Ioannidis mentions how researchers are spending too much time chasing funds. Preregistration increases costs (in terms of extra work) to the researcher; encouraging them to chase more funding. Increasing quality will likely require reducing the cost of doing higher quality research; not increasing it. Yes, I'm aware COMPare is using the prereg...
John Ioannidis has written a very insightful and entertaining article about the current state of the movement which calls itself "Evidence-Based Medicine". The paper is available ahead of print at http://www.jclinepi.com/article/S0895-4356(16)00147-5/pdf.
As far as I can tell there is currently no paywall, that may change later, send me an e-mail if you are unable to access it.
Retractionwatch interviews John about the paper here: http://retractionwatch.com/2016/03/16/evidence-based-medicine-has-been-hijacked-a-confession-from-john-ioannidis/
(Full disclosure: John Ioannidis is a co-director of the Meta-Research Innovation Center at Stanford (METRICS), where I am an employee. I am posting this not in an effort to promote METRICS, but because I believe the links will be of interest to the community)