I previously discussed the potential relevance of therapeutic and instructional models of metacognitive training to LW-style rationality skills. As an attempted concrete realization of what this connection could look like, I ran a self-experiment in which I counted instances of noticing confusion. Below I elaborate on the motivation and design of the experiment, then discuss some quantitative results and qualitative reflections.

What can existing research tell us about teaching and learning key rationalist skills? One central idea is the monitoring and adjustment of mental processes, which appears in various forms in the LW community, debiasing research, education research concerning metacognition, and cognitive-behavioral and related therapies. I expect that rationality training can benefit from fully realizing the connections between these fields, which some mainstream research is already beginning to do. In another post, I discuss a self-experiment in noticing confusion based on these ideas.

As promised, I review and point-by-point summarize How Learning Works: 7 Research-Based Principles for Smart Teaching by Susan A. Ambrose, Michael W. Bridges, Michele DiPietro, Marsha C. Lovett, and Marie K. Norman (2010), hereafter HLW as I scratch in futility at the sprawling length of this post.

Review

The authors aim to provide "a bridge between research and practice" for teaching and learning, very much in the spirit of Practical Advice Backed by Deep Theories. They concentrate on widely-supported results that are independent of subject matter and environment, so while the discussion is directed towards instructors in K-12 and college classrooms, there are also implications for essentially anyone in a teaching or learning role.

Let me restate that a little more strongly: any student, autodidact or not, would be well-served by internalizing the models and recommendations presented here. Teachers have even less of an excuse not to read the book, which is written very clearly and without sinking to punchy popularization. This is basic stuff, in the best possible way.

Sure, there are more sophisticated ideas out there; there exist subgenres of domain-specific research (especially for math and physics education); you can find diverse perspectives in homeschooling communities or in philosophy of education. There's even some controversy in the depths of the research on some of the points in this book (though for the most part the scope of disagreements is still contained within the boundaries drawn by the authors). But as far as most people need concern themselves, HLW is an earnest and accurate if not quite comprehensive account of What We Know about learning.

[I do wish there were a similar account of And How We Think We Know It, looking into common research techniques, metrics of learning outcomes, systematic errors to guard against, reliability of longitudinal studies, statistics about replicability and retractions, and so on, but this isn't it. The book lightly describes methods when it sees fit, and my scattered checks of unfamiliar studies leave me fairly confident that the research does in fact bear the claims the book makes.]

The book organizes research on teaching and learning into seven principles in order to "provide instructors with an understanding of student learning that can help them (a) see why certain teaching approaches are or are not supporting students ’ learning, (b) generate or refine teaching approaches and strategies that more effectively foster student learning in specific contexts, and (c) transfer and apply these principles to new courses." The principles are

1. Students' prior knowledge can help or hinder learning.
2. How students organize knowledge influences how they learn and how they apply what they know.
3. Students' motivation determines, directs, and sustains what they do to learn.
4. To develop mastery, students must acquire component skills, practice integrating them, and know when to apply what they have learned.
5. Goal-directed practice coupled with targeted feedback enhances the quality of students' learning.
6. Students' current level of development interacts with the social, emotional, and intellectual climate of the course to impact learning.
7. To become self-directed learners, students must learn to monitor and adjust their approaches to learning.

Hopefully these ideas are not surprising to you. They are not meant to be; they stand mostly to organize diverse research findings into a coherent model (see principle #2). And if many of those research findings are old news to you as well, I also take that to be a point in favor of the book, and I trust that you will understand why.

Each chapter begins with two stories meant to illustrate the principle, a discussion of the principle itself, a discussion of the research related to that principle, and recommendations that take the principle into account. The chapters are interconnected but stand on their own. If you don't plan to teach, you might get most of your value from Chapters 4, 5, and 7. There's some fluff to the book, but not much. My summary, though long, leaves out the stories and examples, useful repetitions and rephrasings, detailed explanations, and specific recommendations, not to mention descriptions and citations of the relevant studies. I do not consider it a substitute for reading the book, which isn't really that long to begin with.

Before I summarize HLW, I'll make a couple brief comparisons. Why Don't Students Like School: A Cognitive Scientist Answers Questions About How the Mind Works and What It Means for the Classroom by Daniel T. Willingham (2009) looks pretty similar, down to the format in which chapter titles ask questions which are then answered by Principles of Learning, followed by a discussion of the principle, followed by recommendations for the classroom. It's written at a more popular level, with less discussion of actual research and lots more fluff. Only occasionally does it draw connections directly to a study, rather than use that as the chief mode of exposition (as in HLW). Each chapter does have a short annotated bibliography divided into less and more technical texts, which is nice. Willingham comes down strongly in favor of drilling and factual knowledge preceding skill. While that's something I've approvingly polemicized about at some length, it needs a mountain of caveats. In general he optimizes (explicitly, in fact) for counterintuitive punchiness, and it's not always clear how well-supported his advice really is. The organization and coverage feels haphazard to me, but where he hits on topics covered by HLW, he seems to agree.

The 25 Principles of Learning [pdf] from the University of Memphis learning group is a short document with a similar aim: a few sentences describing each principle, a couple sentences describing the implications, and a couple of references. It covers important points that HLW addresses only indirectly or that it inexplicably leaves out entirely (spaced repetition, testing, and generation effects, for example). It's worth looking over to fill in those gaps. But it's really "25 Important Findings on Learning": it doesn't give examples, offer very specific advice, or attempt to organize these principles into a causal model of learning. Consider them exercises for the reader.

(Cross-posted from my personal site.)

Several months ago I began a list of "things to try," which I share at the bottom of this post. It suggests many mundane, trivial-to-medium-cost changes to lifestyle and routine. Now that I've spent some time with most of them and pursued at least as many more personal items in the same spirit, I'll suggest you do something similar. Why?

• Raise the temperature in your optimization algorithm: avoid the trap of doing too much analysis on too little data and escape local optima.
• You can think of this as a system for self-improvement; something that operates on a meta level, unlike an object-level goal or technique; something that helps you fail at almost everything but still win big.
• Variety of experience is an intrinsic pleasure to many, and it may make you feel less that time has flown as you look back on your life.
• Practice implementing small life changes, practice observing the effects of the changes, practice noticing further opportunities for changes, practice value of information calculations, and reinforce your self-image as an empiricist working to improve your life. Build small skills in the right order and you'll have better chances at bigger wins in the future.
• Advice often falls prey to the typical-mind (or typical-body) fallacy. That doesn't mean you should dismiss it out of hand. Think about not just how likely it is to work for you, but how beneficial it would be if it worked, how much it would cost to try, and how likely it is that trying it would give you enough information to change your behavior. Then just try it anyway if it's cheap enough, because you forgot to account for uncertainty in your model inputs.
• Speaking of value of information: don't ignore tweakable variables just because you don't yet have a gwern-tier tracking and evaluation apparatus for the perfect self-experiment. Sometimes you can expect consciously noticeable non-placebo effects from a successful trial. You might do better picking the low hanging fruit to gain momentum before you invest in a Zeo and a statistics textbook.
• You know what, if there's an effect, it may not even need to be non-placebo. C.f. "Lampshading," as well as the often-observed "honeymoon" period of success with new productivity systems.
• It's very tempting, especially in certain communities, to focus exclusively on shiny, counterintuitive, "rational," tech-based, hackeresque, or otherwise clever interventions and grand personal development schemes. Some of these are even good, but one suspects that some are optimized for punchiness, not effectiveness. Conversely, mundane ideas may not propagate as well, despite being potentially equally or more likely to succeed.
• If you were already convinced of all of the above, then great! I hope you have the agency to try stuff like this all the time. If not, you might find it useful, as I did, just to have a list like this available. It's one less trivial inconvenience between thinking "I should try more things" and actually trying something. I've also found that I'm more likely to notice and remember optimization opportunities now that I have a place to capture them. And having spent the time to write them down and occasionally look over them, I'm more likely to notice when I'm in a position to enact something context-dependent on the list.

I removed the terribly personal items from my list, but what remains is still somewhat tailored to my own situation and habits. These are not recommendations; they are just things that struck me as having enough potential value to try for a week or two. The list isn't not remotely comprehensive, even as far as mundane self-experiments are concerned, but it's left as an exercise to the reader to find and fill the gaps. Take this list as an example or as a starting point, and brainstorm ideas of your own in the comments. The usual recommendation applies against going overboard in domains where you're currently impulsive or unreflective.

Related posts: Boring Advice Repository, Break your habits: Be more empirical, On saying the obvious, Value of Information: Four Examples, Spend money on ergonomics, Go try things, Don't fear failure, Just try it: Quantity trumps quality, No, seriously, just try it, etc.