Vladimir_M comments on Some Heuristics for Evaluating the Soundness of the Academic Mainstream in Unfamiliar Fields - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (272)
I do agree that there are fields where the overall standards of the academic mainstream are not that high, but I'm not sure about the heuristics - I tend to use a different set.
One confusing factor is that in almost any field, the academic level of an arbitrary academic paper is not that high - average academic papers are published by average scientists, and are generally averagely brilliant - in other words, not that good. The preferred route is typically to prove something that's actually already well known, but there are also plenty of flawed papers. There are also plenty of papers that are perhaps interesting if you're interested in some particularly small niche of some particularly minor topic, but are of no relevance to the average reader. None of this says anything much about the quality of the mainstream orthodoxy, which can be very much higher than the quality of the average paper.
My main principle is that human beings are just not that intelligent. They are intelligent enough to follow a logical argument that is set into a system where there are tightly defined rules from which one can reason. They are NOT intelligent enough to reason sensibly AT ALL in regions where such rules are not defined. Well, perhaps a logical step or two is plausible, but anything beyond that becomes very dubious indeed - it is like trying to build a tower on a foundation of jello.
Reasoning based on vague definitions is a red flag - it encourages people to come up with any answer they want, and believe they've logically arrived at it. Reasoning based on a complicated set of not particularly related facts is a red flag, as nobody is intelligent enough to do it correctly.
Someone once said that all science is either physics or stamp collecting. It's close - you have to have some organising principles of decent mathematical quality to do reasoning with any certainty. Without that, stamp collecting is the limit of the possible.
Equally, maths is not a panacea. It's quite possible, in an academic paper, to spend a great deal of time developing a mathematical argument based on assumptions that aren't really connected to the question you're trying to answer - the maths is probably correct, but the vague and fuzzy bit where the maths is trying to connect to the problem is where it all goes wrong. To take the example everyone knows, financial models that assume average house prices can't go down as well as up may have perfectly correct mathematics, but will not predict well what will happen to those investments when house prices do go down.
In summary, those fields with widely accepted logical systems are probably doing something right. Those fields where there are multiple logical systems that are competing are probably also doing something right - the worst they can do is to reason correctly about the wrong thing. Fields where there is an incumbent system which is vague are bad, as are those fields where freeform reason is the order of the day.
DuncanS:
I disagree with this. In many areas there are methodologies that don't approach a mathematical level of formalization, and nevertheless yield rock-solid insight. One case in point is the example of historical linguistics I cited. These people have managed to reach non-obvious conclusions as reliable as anything else in science using a methodology that boils down to assembling a large web of heterogeneous common-sense evidence carefully and according to established systematic guidelines. Their results are a marvelous example of what some people call "traditional rationality" here.
In a way making a forum post is an example of the very kind of thing that I'm criticising - it's a piece of freeform expression, and it's a medium in which mistakes creep in easily.
I think you're right to disagree with my statement there. The key thing isn't the presence of mathematics - it's the existence of some kind of set rational process - the "established systematic guidelines" that you mentioned.