Perhaps the main tool of rationality is simply to use explicity reasoning where others don't, as Jacob Falcovich suggests:
New York Times reporter Cade Metz interviewed me and other Rationalists mostly about how we were ahead of the curve on COVID and what others can learn from us. I told him that Rationality has a simple message: “people can use explicit reason to figure things out, but they rarely do”
However, I also think a big chunk of the value of rationality-as-it-exists-today is in its corrections to common mistakes of explicit reasoning. (To be clear, I'm not accusing Jacob of ignoring that.) For example, bayesian probability theory is one explicit theory which helps push a lot of bad explicit reasoning to the side.
The point of this question, however, is not to point to the good ways of reasoning. The point here is, rather, to point at bad concepts which are in widespread use.
For example:
- Fact vs opinion. There are several reasons why this is an awful concept.
- The common usage suggests that there are "matters of fact" vs "matters of opinion"; eg, I like hummus (opinion) but 1+1=2 (matter of fact). But common usage also suggests that probabilistic reasoning gives mere opinions, while other modes of reasoning (such as direct observation, and logical reasoning) yield facts. This is inconsistent; it suggests that we can tell whether a belief is an opinion or a fact by examining what it is about (beliefs about subjective things = opinions; beliefs about objective things = facts), while also seeming to need the mode of reasoning by which we arrived at the belief (eg, if I saw a black hole myself, it would be a fact, but if I derived one's existence from unproven physics, it would be opinion).
- Calling something a fact generally indicates that others are epistemically obligated to believe it. But if it is contentious, then this is precisely what's at issue. So calling something a fact like this is generally useless.
- We could take "fact" to mean something like "true opinion". But from the inside, this is no different from a strong belief. So again, to call something a fact rather than a strong opinion seems to add no information (whereas, I take it, it's supposed to according to common usage).
- "Purpose" as an inherent property. In common usage, it makes sense to ask "the purpose of life" because a purpose is a property which lots of objects have. In reality, it only makes sense to think of "purpose" relative to some agent, as in "I made this for this purpose". Common usage allows purpose to be agent-independent because there are lots of things (tables, chairs, silverware, etc) which have purposes largely independent of agent (most people use tables to set things on for convenient reach, chairs to sit on, silverware to eat, etc). However, in cases which aren't like that, the language doesn't make sense without explanation (but people treat it like it does).
These are intended to be the sort of thing which people use unthinkingly -- IE, not popular beliefs like astrology. While astrology has some pretty bad concepts, it is explicitly bundled as a belief package which people consider believing/disbelieving. Very few people have mental categories like "fact-ist" for someone who believes in a fact/opinion divide. It's therefore useful to make explicit belief-bundles for these things, so that we can realize when we are choosing whether to use that belief-bundle.
My hope is that when you encounter a pretty bad (but common) concept out there in the wild, you'll think to return here and add it to the list as a new answer. (IE, as with all LW Questions, I hope this can become a timeless list, rather than just something people interact with once when it is on the front page.)
Properly dissolving the concept by explaining why people (mis)use it is encouraged, but not required for an entry.
Feel free to critique entries in the comments (and critique my above two proposals in the comments to this post), but as a contributor, don't stress out about responding to critiques (particularly if stressing about this makes you not post suggestions -- the voting should keep the worst ones at the top, so don't worry about submitting concepts that aren't literally the worst!).
Ideally, this would become a useful resource for beginners to come and get de-confused about some of the most common confusions.
I think that we'll need to figure this out at some point to be able to clearly distinguish these things to children. Not being able to distinguish fact from fiction can lead to the propagation of non-factual information that will be then be discerned as fact as dictated by the overall consensus of a said individual's community. Our present tools on the internet are very good at determining relevancy (which is debatable of course), but they are not the best at determining verity or distinguishing bias.
We're have to assume that a source of factual information is altruistic and that they actually do the research in order to distinguish one from another. But this in itself is very fragile as if it violates the facts that a community already believes (such as when said altruistic person is in actuality not altruistic), there is no simple way to revert the information that has already propagated throughout said communities. Being able to clearly distinguish facts from opinions, enables one to reinforce the idea of what's factual...and avoid the potential issue of disputing a fact when comparing it against a differing opinion which is being misinterpreted as a fact.
I think this is critical, because certain structures rely on being able to discern fact from opinion, which has manifested itself in the national policies of some countries, but they of course can be easily misrepresented if an individual only consumes information (factual or otherwise) from a similar source of said information.