Admiral Nelson had multiple surgeries and was in serious danger of infection and death afterwards, but he would have been a goner for sure without surgery.
This argument seems exactly identical to the argument for trepanning, even including the survivorship bias. (One of the suspected uses of trepanning was to revive people otherwise thought dead.)
While we're looking at anecdotes, this bit of Nelson's experience with surgery seems relevant:
Although surgeons had been unable to remove the central ligature in his amputated arm, which had caused considerable inflammation and poisoning, in early December it came out of its own accord and Nelson rapidly began to recover.
I'm not sure I'd count that as a win for surgery, or evidence that he couldn't have survived without it!
Gwern, peer review is my life. My tenure case will be decided by peer review, ultimately. I do peer review myself as a service, constantly. I know all about peer review.
But this means that, unless you're particularly good at distancing yourself from your work, you should expect to be worse at judging it than a disinterested observer. The classic anecdote about "which half?" comes to mind, or the reaction of other obstetricians to Semmelweis's concerns.
Regardless, we would expect that, if studies are better than anecdotes, studies on peer review will outperform anecdotes on peer review, right?
This argument seems exactly identical to the argument for trepanning, even including the survivorship bias.
It's not identical because we know, with benefit of hindsight, that amputating potentially gangrenous limbs is a good idea. The folks in the past had solid empirical basis for amputations, even if they did not fully understand gangrene. Medicine was mostly, but not always nonsense in the past. A lot of the stuff was not based on the scientific method, because they had no scientific method. But there were isolated communities that came up with s...
If it's worth saying, but not worth its own post (even in Discussion), then it goes here.
Notes for future OT posters:
1. Please add the 'open_thread' tag.
2. Check if there is an active Open Thread before posting a new one. (Immediately before; refresh the list-of-threads page before posting.)
3. Open Threads should be posted in Discussion, and not Main.
4. Open Threads should start on Monday, and end on Sunday.