This is a linkpost for https://thewaywardaxolotl.blogspot.com/2019/05/why-most-academic-research-is-fake.html
Why Most Academic Research is Fake
5Hastings
-2Zero Contradictions
2Seth Herd
0ProgramCrafter
New Comment
To say that most academic research is anything, you’re going to have to pick a measure over research. Uniform measure is not going to be exciting – you’re going to get almost entirely undergraduate assignments and Third World paper mills. If your weighted sampler is “papers linked in articles about how academia is woke” you’re going to find a high %fake. If your weighed measure is “papers read during work hours by employees at F500 companies” you’ll find a lower, nonzero %fake.
Handwringing over public, vitriolic retractions spats is going to fuck your epistemology via sampling bias. There is no replication crisis in underwater basket weaving
I already know that. I addressed these considerations in greater depth in my sequel essay: https://zerocontradictions.net/epistemology/critique-of-academic-research
Regardless, you're missing the point of the essay. The overall academic enterprise is not designed to seek the truth. It's clearly explained that ideological bias, perverse incentives, social circularity, naive/fake empiricism, and misleading statistics (e.g. p-hacking) compromise the production of truthful research. The sequel essay elaborates on all of these ongoing issues.
1
1Hastings is responding to the claim in the title, so if he's missed the point of the essay, you've mistitled it.
I think this would've done much better with a more modest title something like "academia is mostly not truthseeking" or something similar. LW is highly suspicious of clickbait titles and inflated claims - the goal to "inform not persuade" is almost the opposite of essay writing elsewhere.
I think you come off as condescending and defensive in the above response, and it probably earned the post an extra downvote or two. I upvoted it, because I think it's an important topic, and I agree with the spirit if not the literal claim in the title. I think many LWers would roughly agree, they just wouldn't state it this hyperbolically.
Having been a professional academic for 23 years and considering the epistemology pretty closely, I think the title isn't far off. As Hastings said, you'd have to quantify what's academic research, but most of the top-journal stuff I read wouldn't deserve the commonsense meaning of "fake". But it is so low-quality as a result of conflicting incentives that calling most of it "confused" or "wrong" wouldn't be a stretch.
For the record, there is still some valid, good-faith scientific research out there, but most of the academic “research” produced in more recent decades is either fabricated, dishonest, plagiarized, redundant, outdated, and/or useless.
This claim could be restated as: most of the academic "research" is either false because authors did not intend to be honest (or conformed to unrelated biases), or false because authors did not have more accurate data (this kind of research becomes superseded over time, when more belief depth is acquired). This might be true.
However, I don't know what redundant articles do in that list; I suppose you're claiming more articles stating the same point do not provide more evidence to it, but replication and more experiments in good faith are always good.
To say that “most” academic research is “fake” also implies that we can quantify how much of it is fake or not. I can’t precisely estimate, quantify, and judge every academic paper that gets published out there, so I don’t claim to know exactly how much of the current research being published is reliable. It probably also varies by fields, and it’s possible for papers to include a mixture of true and fake data, reasoning, and conclusions.
And thereon the essay goes to saying "most" without any description what "research" is taken as a sample set. Hastings' comment, on the other hand, suggested some alternatives:
Uniform measure is not going to be exciting – you’re going to get almost entirely undergraduate assignments and Third World paper mills. If your weighted sampler is “papers linked in articles about how academia is woke” you’re going to find a high %fake. If your weighed measure is “papers read during work hours by employees at F500 companies” you’ll find a lower, nonzero %fake.
However, we can generally say that most recent research in humanities (or human-centric sciences) is fake, redundant, or useless; most research in earth-centric sciences is true, fake, or questionable; and most research in STEM fields is true, fake, outdated, or redundant.
Major nitpicking here. If most true claims of human-centric studies are replicated, then each of the corresponding papers are redundant (as it'd have a duplicate); therefore, almost all research would be "fake, redundant, or useless". Moreover, for STEM fields "true, fake, outdated, or redundant" seems to describe universal set - that is, that statement is of no substance. I'd suggest clarifying what claims you had in mind, if you are not using them for emphasis only.
The best rule is not to assume that because an academic paper says X, that X is true.
The best rule known to us - i.e. Bayesian reasoning - mandates that we simply treat "research" as stream of characters, and assess probabilities of each stream being shown to you if X were true and if X were false. That is intractable; after some fallback, you get at "correct for authors' biases, and assume that paper's claims represent average of what happens". I have the impression LessWrong does pretty much that.
Academic research can be fake in different ways. It can simply be false. It can be emotionally manipulative propaganda masquerading as knowledge. It can be irrelevant or meaningless.
Specific claim being true or false necessarily screens off being "emotionally manipulative propaganda". A weaker point that would stand, though: "often papers are emotionally manipulative, even when the claims presented in them are inappicable to most real situations or meaningless outside of academia".
I believe the further parts of Sections 1 and 2 are not of much interest for LessWrong, except that they attempt establishing common knowledge that academic "research" is commonly fake. Section 3, with specific suggestions, could be positively received when posted separately.
The PDF version can be read here.
In this essay, I will argue that most academic research is fake. The modern academy is not a reliable source of knowledge. Instead, it produces the pretense of knowledge.
Academic research can be fake in different ways. It can simply be false. It can be emotionally manipulative propaganda masquerading as knowledge. It can be irrelevant or meaningless.
These are the main causes of fake research:
I’ll describe each and explain how it causes fake research.
(see the rest of the post in the link)