Hi
I am new to this website so please excuse me if I diverge from the rules of discourse.
I think there is an inaccurate assumption in this text. Even though scientific ideas become official through publishing in scientific journals it is not the only platform to discuss science. I will suggest here other mechanisms to root out errors and discuss scientific ideas based on my experience as a researcher in Earth Sciences. I know other fields use different methods though, to my understanding, these methods are fundamentally similar.
Since Earth Sciences suffer from chronicle lack of evidence Earth scientists are bound to come up with contradicting ideas. Most of these ideas are rooted out in the initial stage of fitting the model to existing evidence. The rest are then presented not in papers but as presentations in seminars, workshops and conferences. Since most of the problems are only handled by a small amount of researchers these meetings are very effective in highlighting all the existing models of the discussed issue. Naturally basic flaws will rise at that stage. Only afterwards ideas tend to be submitted to publication. Much can be written about the peer review system but if it is done well it will also point out weak claims and flaws in logic. Though, due to the aforementioned stages, it is usually focused on technical details or gaps in background and references rather than the baseline logic. Lastly, a published work is not the bottom line of an argument. Other papers may be published with other models or new evidence that will make older publications irrelevant. Since those papers will have to go through all the previous stages as well it is likely that they will result in either new agreed baseline or agreed disagreement that requires further evidence. In both cases it is unlikely that the argument will generate further publications (and thus may explain your findings). The exception is when new evidence emerge in a case of agreed disagreement. I find it likely that all sides of the argument will try to use their models to explain the new evidence with varying degrees of success. Normally when a model stop fitting the evidence it is simply ignored in future publications. So it may still appear to exist as a possible explanation but it is effectively not there. In most cases to find the agreed models for a problem researchers turn to review papers. These are meant to present the issue, all possible explanations and the strong and weak points of each model.
So, to sum things up, I think the process you are looking for is the one done under less official interactions. Theories are confronted in meetings and such. Less accurate theories are simply ignored in future discourse.
There are clear disadvantages to this system. The main one is that it is clearly sensitive to manipulation by strong people on key positions in their field. A theory can be ignored not because it is wrong but because its publisher is not popular. This may be the strongest argument to why publications that are agreed to be wrong should not be changed or commented after the fact.
I hope this makes sense and answers your question.
Lior
This may be the strongest argument to why publications that are agreed to be wrong should not be changed or commented after the fact.
Could you elaborate? How one should know which publications are agreed to be wrong?
Not a full answer, but I would expect most of this kind of debate to be in more informal channels rather than journals (as in LiorSuchoy's answer).
Einstein, for example, was a prolific letter writer, and corresponded with many of the great physicists and mathematicians of the day, e.g. Born, Cartan and Schrödinger (from a quick google it looks like the Schrödinger letters are still not published as a collection, so I haven't linked them).
I read the Cartan letters, some time ago. I don't have access to a copy now, but IIRC they get much more into picking at disagreements/clearing up confusions than anything you'd find in journals. For example, I opened up the Google Books preview, and immediately found the following from Einstein (on page 13):
I am sending to you my articles on the subject, published so far by the Academy. The second, on the approximate field equations, suffers, however, from the drawback that, with the choice made there for the Hamiltonian, a spherically symmetric electric field is impossible...
Then as well as letters, there'd be conversations at conferences, gossip over lunch and in department common rooms, question sessions after lectures. This stuff is mostly lost, though, whereas the letters can still be read now, so that's where I'd look.
All of this still goes on between researchers now, of course, and that's still how news travels in individual research areas. If you want to know what's wrong with published papers you're much better off talking people in that field than trying to find retractions in the published literature. But academia was so much smaller then that informal networks of correspondence might plausibly cover large areas of science rather than a small research speciality.
My expectation is that the fourth alternative, or some variation thereof, is the dominant answer. This is less a reflection of the quality of the papers, and more a reflection of the limited bandwidth of scientists for reading them.
This problem has been discussed in the modern context because of the explosion in the number of publications and the administrative responsibilities of scientists (for example, teaching and grant writing). But it is also noticed that deep reading of papers is both time consuming and cognitively intensive; troubling to write up a correction still more so. I argue there is still a fundamental bandwidth limit, and the early 20th century scientists still had to abide by it.
Following on the argument that reading papers deeply enough to correct errors and publish those corrections is difficult, I posit that the 'publish or perish' mechanism is responsible for corrections being published at all. I expect that even though there are errors, if the objective is to produce the best original work possible it is more efficient to correct them for yourself and then go on to use the corrected version for yourself; it could even be argued that leaving the errors publicly uncorrected is advantageous for being first. I also expect that if the objective shifts to total number of publications, it becomes more efficient to publish corrections because writing up a correction is less difficult than producing original work.
If my expectation is correct, then we should see very few corrections published leading up to World War II, and then an increasing number afterward as the professionalization of science progresses.
One good source for this kind of question would be histories of science and/or math. They do a pretty good job of disentangling what scientists thought and when, because they do the difficult work of going through notes, correspondence, and the published work. The downside is it will usually be from the subject's perspective, ie thermodynamics, instead of focusing on academia per se.
You start with the assumption that academia works by ensuring that there are no errors in papers. This sounds like the Popperian hypothesis that falsification is what drives science forward. Historians of science like Thomas Khun seem to constistently find that science doesn't work the way Popper assumes it does.
Given how diseased the field of Neuroscience is it might be a easy target, but apart from their ability to consistently get results that are better then possible, they also managed as a community to make a certain statistical error in 50% of the papers where such an error could be made.
At the same time, the field seems to make progress. It provides demand for fMRI to be build and as a result fMRIs with higher resolution get as time goes on. There are now even designs for MEG scanners that don't need to be supercooled.
You start with the assumption that academia works by ensuring that there are no errors in papers.
I think you misunderstood the question. The assumption of the post is explicitly that we expect that there will be errors in papers, and we are just asking how the errors in those papers are being discovered, and how those discoveries are communicated.
The question is: Given that a lot of ideas will be wrong and confused, how does academia notice and communicate those confusions?
Oli's comment is right. Another way of phrasing the question is:
Why do journals purport to contain no errors in any of the papers? Is it because they're lying about the quality of the papers? Or if there's a reliable process that removed errors from 100% of papers, can someone tell me what that process was?
Edit: I've added this to the OP.
I upvoted this post as strongly as I could with my Karma, and I'm putting this comment here to reinforce: this is a great question, and I learned some things about the 19th century from it.
I would love to see more things on Less Wrong on the topics of:
I wrote this post, and at the time I just wrote it because... well, I thought I'd be able to write a post with a grand conclusion about how science used to check the truth, and then point to how it changed, but I was so surprised to find that journals had not one sentence of criticism in them at all. So I wrote it up as a question post instead, framing my failure to answer the question as 'partial work' that 'helped define the question'.
In retrospect, I'm really glad I wrote the post, because it is a clear datapoint about how science does not work. I have updated down on the need for large amounts of public critique in figuring out what's true. I think about the datapoint from time to time, and expect to think about it more.
This post:
This makes this suitable for a nomination.
It's a really important question, and the answers actually helped me answer it (though they were far from comprehensive).
Something I've seen annoyingly often when digging into the literature is (often critical) reviews of the document I'm after. This is annoying because everything is behind a paywall, and the title often makes it look like I'm getting the document I'm really after, when in fact I'm just getting a 2-page review written by a different person (which I only discover after I've paid for it). I very rarely pay for documents behind paywalls; when I do, it's because the document in question just can't be obtained another way. According to my unreliable memory, at least as often as not, it's turned out that I'm making this mistake, and only obtain a short review of the document I'm really after.
It seems to me that these reviews are mostly written for books.
I might be wrong, but I may have seen some back and forth in some of these? Like, an author response.
This is an interesting historical question, but I'd like to challenge your initial motivation ;) So the idea that sciences used to be pursued more effectively a century ago. Intuitively speaking, I don't see why this would be the case, so I'd first have to see some evidence (including the measure of effectiveness) for this claim. My impression is rather that due to immense fragmentation of today's science into sub-disciplines, there are more people working on particular problems who are effective in their own domains, while remaining largely unknown to the wider audience.
In fact, I would link a lower degree of interaction in the past science, in comparison to today's science (we have peer-review system, there are more conferences, there is an easier access to publications, etc.) with a lower degree of effectiveness. But of course, how exactly interaction and effectiveness/efficiency are related is an empirical question, so I'm open to be surprised :)
In the post 'Four layers of Intellectual Conversation', Eliezer says that both the writer of an idea, and the person writing a critique of that idea, need to expect to have to publicly defend what they say at least one time. Otherwise they can write something stupid and never lose status because they don't have to respond to the criticism.
I was wondering about where this sort of dialogue happens in academia. I have been told by many people that current journals are quite terrible, but I've also heard a romantic notion that science (especially physics and math) used to be more effectively pursued in the early 20th century (Einstein, Turing, Shannon, etc). So Oliver and I thought we'd look at the journals to see if they had real conversations.
We looked at two data points, and didn't find any.
First, Oliver looked through Einstein's publication history (Oli is German and could read it). Einstein has lots of 'reviews' of others' work in his list of publications, sometimes multiple of the same person, which seemed like a promising example of conversation. Alas, it turned out that Einstein had merely helped German journals write summaries of papers that had been written in English, and there was no real dialogue.
Second, I looked through a volume of the London Mathematical Society, in particular, the volume where Turing published his groundbreaking paper proving that not all mathematical propositions are decidable (thanks to sci-hub for making it possible for me to read the papers!). My eyes looked at about 60% of the pages in the journal (about 12 papers), and not one of them disagreed with any prior work. There was :
There were lots of discussions of people's work but always building, or extending, or finding a neater way of achieving the same results. Never disagreement, correction, or the finding of errors.
One thing that really confuses me about this is that it's really hard to get all the details right. Lots of great works are filled with tiny flaws (e.g. Donald Knuth reliably has people find errors in his texts). So I'd expect any discussion of old papers to bring up flaws, or that journals would require a section at the end for corrections of the previous volume. There were of course reviewers, but they can't be experts in all the areas.
But more importantly where did/does the dialogue happen if not in the journals?
If I try to be concrete about what I'm curious about:
There's also the third alternative, that no progress was made. And there's the fourth alternative, that most papers were bad and the thing scientists did was to just never read or build on them. I'm happy to get evidence for any of these, or a fifth alternative.
Added: Maybe this is a more crisp statement: