Insightful, accurate and depressing.

On a related note:

Ben Goldacre along with colleagues has recently set up COMPare.

http://compare-trials.org/

For the last few years it’s been the norm that research in humans should be preregistered before the trial starts to avoid the file-drawer effect where negative trials don’t get published.

Without preregistration it’s hard to tell when someone has thrown a dart at a wall then built the dartboard around it.

It’s gradually been improving with a lot of research being preregistered but still published trials often don’t report what they said they were going to report or report things they didn’t preregister.

The dartboard is now there beforehand but people are still quietly building new dartboards around wherever the dart hits without mentioning it or mentioning that they were aiming at the original dartboard.

The COMPare project is doing something incredibly simple: Reading the paper. Reading the preregistered plan. Posting a public note on their website and sending a letter to the publishing journal pointing it out.

It’s embarrassing for journals because in theory they should have made sure that the papers matched what was preregistered during peer review.

They’ve had a range of responses, some journal editors like the ones at the BMJ have posted corrections while others have doubled-down like the editors at Annals of Internal Medecine, it’s really quite entertaining.