For example discovering that two phenomenon we thought were separate are actually sub-cases of some more basic phenomenon.
I like to do this a lot in mathematics, but fortunately mathematical language is both rich and rigorous enough that I can avoid false alarms in that context (category theory in particular is full of examples of phenomena that look separate but can rigorously be shown to be subcases of a more basic phenomenon).
I think of mathematics as being a conspiracy theorist's fantasy land: it works nearly the way a conspiracy theorist thinks reality works.
So I guess what I'm basically asking is why aren't we spending a lot more time improving the checklist of rationality habits, especially via empiricism.
Well, that's something like what CFAR is trying to do.
See also: Boring Advice Repository, Solved Problems Repository, Grad Student Advice Repository
I often find that my understanding of the world is strongly informed by a few key concepts. For example, I've repeatedly found the concept of opportunity cost to be a useful frame. My previous post on privileging the question is in some sense about the opportunity cost of paying attention to certain kinds of questions (namely that you don't get to use that attention on other kinds of questions). Efficient charity can also be thought of in terms of the opportunity cost of donating inefficiently to charity. I've also found the concept of incentive structure very useful for thinking about the behavior of groups of people in aggregate (see perverse incentive).
I'd like people to use this thread to post examples of concepts they've found particularly useful for understanding the world. I'm personally more interested in concepts that don't come from the Sequences, but comments describing a concept from the Sequences and explaining why you've found it useful may help people new to the Sequences. ("Useful" should be interpreted broadly: a concept specific to a particular field might be useful more generally as a metaphor.)