To begin, here are some Fun Psychology Facts:
People who were asked to describe a face after seeing it are worse at recognizing the same face later.
People who are asked to describe a wine after drinking it are worse at recognizing the same wine later.
People who are asked to give reasons for their preferences among a collection of jellies are worse at identifying their own preferences among those jellies.
This effect, known as Verbal Overshadowing, occurs primarily when a principally non-verbal process is disrupted by a task which involves verbalization. The above generalizations (and Verbal Overshadowing effects more generally), do not occur among what we can term "Verbal Experts": individuals who are as good at verbalizing the relevant process as they are at doing it implicitly or automatically. This seems like it will be very important to keep in mind when cultivating our own Rationality.
Here's an oversimplified picture of what this means: We've got an implicit facial recognition process, IFRP, which is pretty good. We've also got a generalized explicit verbal thinking process, GEVTP, which is good for lots of things, but isn't especially good at recognizing faces. Normally, IFRP is in charge of facial recognition, but there are some things we can do, like, trying to put a face into words, that wakes up GEVTP, which then muscles IFRP out of the way, and all of a sudden, we are a lot worse at recognizing faces.
The good news is that GEVTP can be trained. To take the wine case, people who put in the time and effort can become verbal experts about wine. This isn't to say they automatically have better judgments about wine. Rather, it means that their GEVTP is on par with their implicit wine recognition, because it has been trained to do the same quality job as the the implicit process.
As a crude metaphor, imagine the difference between the natural process by which you go about walking, versus having to keep track of each and every instruction that needs to be sent to different joints and muscles if you had to consciously issue each one.
Now, obviously the specific studies mentioned are important for wine tasting, eye-witness identification, or determining one's own jelly preferences, but the phenomenon of Verbal Overshadowing has a much larger, more systematic importance for th Art of Rationality.
Let's bridge to the broader point with a quote from David Hume, a man whose insights were often far ahead of their time: "I shall add [...] that, as this operation of the mind, by which we infer like effects from like causes, and vice versa, is so essential to the subsistence of all human creatures, it is not probable, that it could be trusted to the fallacious deductions of our reason, which is slow in its operations; appears not, in any degree, during the first years of infancy; and at best is, in every age and period of human life, extremely liable to error and mistake. It is more conformable to the ordinary wisdom of nature to secure so necessary an act of the mind, by some instinct or mechanical tendency, which may be infallible in its operations, may discover itself at the first appearance of life and thought, and may be independent of all the laboured deductions of the understanding. As nature has taught us the use of our limbs, without giving us the knowledge of the muscles and nerves, by which they are actuated; so has she implanted in us an instinct, which carries forward the thought in a correspondent course to that which she has established among external objects; though we are ignorant of those powers and forces, on which this regular course and succession of objects totally depends."
In short, Hume is saying, in the field of inference and reasoning, our Implicit Reasoning Process often outpaces our GEVTP. I'm not suggesting that our implicit reasoning is perfect (it is, after all, fraught with its own biases), but, supposing that Verbal Overshadowing is a general phenomenon, it would appear that, with respect to our reasoning and inferences more generally, our situation is one in which trying to talk about what we are doing is liable to mess us up.
The obvious suggestion, then, is that we become verbal experts on the subject, so that our thinking about rationality doesn't mess up our thinking rationally.
"Aha," I hear you all say, "then your advice is unnecessary, for what is it that we Rationalists are already doing, if not training ourselves to think explicitly about rationality?" And that would be a good reply, but for one crucial fact: we are not training ourselves correctly to become verbal experts.
One does not become a verbal expert about wine by tasting only strange vintages or the wine of abnormal grapes. One does not become a verbal expert about facial recognition by practicing only on the stunningly gorgeous or the hideously deformed. And likewise, one does not become a verbal expert on Rational thinking by focusing on the edge cases (i.e. The Epistemic Prisoner's dilemmas, The Gettier Cases, The High Stakes scenarios, etc.). Verbal Experts get trained, primarily, on the paradigms.
In fact, the studies on Insight Puzzles in particular (i.e. verbal overshadowing with respect to explaining the actual process by which one achieved the solution to a problem), suggest that those of us who engage in verbalization tasks relating to our reasoning and inferences (say, those of us dedicating a lot of time and energy to writing posts or comments about it), had better figure out how to train our Generalized Explicit Verbal Thinking Process not to drop the ball when it comes to thinking about reasoning.
I am not a psychologist, but I do know that our current plan (of, for example, thinking about the brainteaser cases), is definitely not the way to develop actual expertise.
(Edit) You can pretty much ignore this comment, but you should read pangloss' responses below. I found them enlightening.
Personally, I think there are two extremes at studying something like this. One is to work from the edge cases toward the general/common cases and the other is to work from the general/common cases toward the edges. Some gradients in the middle certainly apply.
My hunch is that our perceptions of the common cases have been painted over by too many other ways of life/thinking. Rationality can have trouble getting its point through the layers. The edge cases make more sense to flesh out because the edge cases are explicitly designed to force specific ways of life/thinking into choosing between one or more well-defined options.
Edge cases also have the advantage of Jargon. We can make up our own words to mean something that we were not able to express previously and, by definition, be correct. While this is less amusing than describing it in laymen's terms it certainly gets the problem defined. Working from the edges with new descriptions is a long process and one easy enough to mess up. Pigeonholing is a great risk and we can describing large swaths of general/common cases with a new word is dangerous.
General cases have the advantage that they can provide a foundation that will instantaneously apply to related cases. The network can build faster but the definitions may not be as clear-cut. A word used to describe wine can be used to describe faces (e.g. bitter) and people will be able to follow the conversation. The danger this time is generalizing terms and cases so that we are no longer able to distinguish between specific concepts.
All that being said, language is nifty. Learning how to work with it correctly would certainly be useful.
(Note) I am making a fine distinction between pigeonholing and generalizing that may exist only in my head. I see pigeonholing as taking large numbers of varied things and trying to separate them into ill-defined categories that force more variance than is really necessary. I see generalizing as taking large numbers of varied things and slopping them into categories that begin to erase the variances. If someone else knows better terms for this, please let me know.
I think the question about which cases to focus on when forming theories is different from the question of which cases to use to train oneself to verbalize one's thoughts without interfering with one's thinking. The latter requires us to train on paradigms, the former may be something we can pursue in either direction.
This is crucial: The thought isn't to presuppose which direction our theorizing should go, but rather to make sure that when we theorize, we aren't tripping ourselves up.