In 2018, I was a bright-eyed grad student who was freaking out about AI alignment. I guess I'm still a bright-eyed grad student freaking out about AI alignment, but that's beside the point. 

I wanted to help, and so I started levelling up. While I'd read Nate Soares's self-teaching posts, there were a few key lessons I'd either failed to internalize or failed to consider at all. I think that implementing these might have doubled the benefit I drew from my studies. 

I can't usefully write a letter to my past self, so let me write a letter to you instead, keeping in mind that good advice for past-me may not be good advice for you.

Make Sure You Remember The Content

TL;DR: use a spaced repetition system like Anki. Put in cards for key concepts and practice using the concepts. Review the cards every day without fail. This is the most important piece of advice.

The first few months of 2018 were a dream: I was learning math, having fun, and remaking myself. I read and reviewed about one textbook a month. I was learning how to math, how to write proofs and read equations fluently and think rigorously. 

I had so much fun that I hurt my wrists typing up my thoughts on impact measures. This turned a lot of my life upside-down. My wrists wouldn't fully heal for two years, and a lot happened during that time. After I hurt my wrists, I became somewhat depressed, posted less frequently, and read fewer books. 

When I looked back in 2019/2020 and asked "when and why did my love for textbooks sputter out?", the obvious answer was "when I hurt my hands and lost my sense of autonomy and became depressed, perchance? And maybe I just became averse to reading that way?"

The obvious answer was wrong, but its obvious-ness stopped me from finding the truth until late last year. It felt right, but my introspection had failed me.

The real answer is: when I started learning math, I gained a lot of implicit knowledge, like how to write proofs and read math (relatively) quickly. However, I'm no Hermione Granger: left unaided, I'm bad at remembering explicit facts / theorem statements / etc. 

I gained implicit knowledge but I didn't remember the actual definitions, unless I actually used them regularly (e.g. as I did for real analysis, which I remained quite fluent in and which I regularly use in my research). Furthermore, I think I coincidentally hit steeply diminishing returns on the implicit knowledge around when I injured myself

So basically I'm reading these math textbooks, doing the problems, getting a bit better at writing proofs but not really durably remembering 95% of the content. Maybe part of my subconscious noticed that I seem to be wasting time, that when I come back four months after reading a third of a graph theory textbook, I barely remember the new content I had "learned." I thought I was doing things right. I was doing dozens of exercises and thinking deeply about why each definition was the way it was, thinking about how I could apply these theorems to better reason about my own life and my own research, etc.

I explicitly noticed this problem in late 2020 and thought, 

is there any way I know of to better retain content? 

... gee, what about that thing I did in college that let me learn how to read 2,136 standard-use Japanese characters in 90 days? you know, Anki spaced repetition, that thing I never tried for math because once I tried and failed to memorize dozens of lines of MergeSort pseudocode with it? 

hm... 

This was the moment I started feeling extremely silly (the exact thought was "there's no possible way that my hand is big enough for how facepalm this moment is", IIRC), but also extremely excited. I could fix my problem! 

And a problem this was. In early 2020, I had an interview where I was asked to compute . I was stumped, even though this was simple high school calculus (just integrate by parts!). I failed the interview and then went back to learning algebraic topology and functional analysis and representation theory. You know, nothing difficult like high school calculus. 

I was pretty frustrated with myself.

It's not that I didn't understand it. I just didn't remember it, especially on the spot. The worst part was that I had brushed up on calculus the previous spring, and I still didn't remember it. Turns out that my brain won't remember material it doesn't use for months on end, even if forgetting that material would be embarrassing.

Enter Anki, an amazing spaced repetition system ($20 for iOS, free for computer). The way I like to explain Anki is: 

Anki is a flashcard application into which you can enter a constant number of cards each day while retaining a constant average daily workload. You can add cards each day, without having to study longer and longer to get through all of the cards.

I currently think that unless you have really good memory or you're not learning content you want to remember months from now, you're making a mistake by not using a spaced repetition system. Read Gwern for more on spaced repetition.

Spaced repetition seems especially useful for students. In college, I ran an experiment: for an upper-level French class, I put things I didn't know how to say into Anki, reviewed daily, and otherwise didn't study at all. I got an A. 

How powerful might a bright 6th grader grow, were they to use Anki every day for their whole life? The best time to plant a tree may have been in sixth grade, and the second-best time may have been in seventh grade, but you should still plant the tree now rather than never. 

I've been using Anki for math for the last 71 days, and I currently have a deck of about 900 cards which I study for ~30 minutes daily. In 2018, I spent about 10 minutes daily reviewing a deck of nearly ten thousand French cards.
If I were to add no more cards, daily reviews drop off quickly. 

(Once completed, the reviews in the next few days will be pushed into some of the future days, so this projection is slightly optimistic, but you get the point.)

I love Anki, and I was foolish to circumscribe it to language-learning. I now use Anki to remember key concepts from academic talks, LessWrong blogposts, and yes - textbooks. Which I now love again, and which I read for ~an hour daily again, because I'm actually retaining the content

Measure theory, ring theory, random stuff about taxonomy and epidemiology, quantum mechanics, basic physics, deep RL papers, they all go into Anki, and Anki cards go into my brain - and stay there! 

What a wonderful time to be alive. 

Random Anki tips

I know quite a bit about how to best use Anki, so if you try this and it doesn't seem to work, please message me instead of banging your head against the wall or giving up!

  • Use cloze deletions on short cards. Cloze deletions hide a small part of the text, and make you remember it given the rest of the content. They are fast to make and fast to review. 
    • The vast, vast majority of my cards are cloze now, and they weren't when I first wrote this post. I think that was a mistake.
  • Study every day, preferably at the same time so you aren't always scrambling to get it done before bedtime.
  • Sync with AnkiWeb so you don't lose all your cards if your device dies.
  • Save time by just screenshotting theorem statements and/or proofs. 
  • On iPad, I like using MarginNote to read. I can just draw rectangles around key parts of the pdf, make cloze deletions in the app, and then export the cards to an Anki deck.
Memorizing definitions can be useful: when reading a text, it saves you from having to constantly check what the concept is. Make sure to include examples to work through - don't just toss in random definitions you're barely interested in and will never think about again. 
  • Don't just memorize proofs, focus on the key ideas. Don't just memorize definitions, throw in several example problems which are small enough to actually do in your head (or with a scrap of paper).
    • For example, if I'm trying to really ingrain the concept of an efficient pseudorandom number generator, I have cards where I reason about it by completing short proofs:
The math here isn't important. The key thing is, I want to remember the "pseudorandom number generator" concept, and so I find an interesting result and make myself prove it. The proof isn't too long, which is key—just a few cloze deletions. Don't try to memorize essays with Anki. 

Am I ever going to actually use this math for my research? Probably not. Doesn't matter. Anki makes it cheap to learn and retain things.
  • If you get a card wrong more than 4 times in the first week, it's a bad card. Remake it.
  • Use MathJax instead of Latex, because MathJax renders instantly.
If you want custom commands, use the card type editor.
  • My trigger for "I should add a new card" is reading something and thinking, "this is a cool concept!"
    • I recommend adding cards liberally. Don't worry about getting the formatting or phrasing perfect at first. Just add cards and you'll develop a taste for what should be added, and how.

Read Several Textbooks Concurrently

TL;DR study several topics at once so that your brain has time to cement the concepts you're learning, before the text builds on those concepts further.

AllAmericanBreakfast's recent post is great, so I'll refer you to that to make this point:

Wait, what? You want me to make life easier on myself by, instead of studying calculus...studying calculus, linear algebra, and statistics all at once?

~ AllAmericanBreakfast, The Multi-Tower Study Strategy

The basic idea is that your brain needs time to really cement a new idea in, and so you should study several topics at once in dependency-heavy areas like mathematics. This advice matches up with both my recent personal experience and with advice I received early on from Qiaochu Yuan, but which I had unfortunately ignored. 

For example, right now I'm reading Nielsen and Chuang's Quantum Computation and Quantum Information, Evan Chen's The Infinite Napkin, and a ridiculously easy physics book, Kuhn and Noschese's A Self-Teaching Guide: Basic Physics (more on this later). Previously, I'd been going back through Wasserman's All of Statistics and Pearl's Causality after reading Pearl's Book of Why

I always feel like I'm learning something new instead of banging my head against the wall. Sometimes you should just read one book, but if you don't need to cram, I recommend diversifying. 

Completing The Whole Textbook Is Usually a Big Waste of Time, Please Don't Do It

TL;DR extract the most useful / central concepts and remember them forever via Anki. This doesn't require grasping every arcanum, every detail of a textbook. 

When I started reading textbooks, I completed the whole book. I didn't want to miss a crucial concept. But the thing about crucial concepts is that they pop up everywhere. If you missed a crucial concept, you'll know. You're not going to wake up 20 years from now and be like, "OH NO! I forgot to learn about 'force' when I self-studied physics! And I forgot to learn about injectivity in linear algebra!"

As you read more, you'll get a taste for what's probably important, and what's details you can reference later if need be. You can also ask experts if you should study part of a book - this is one key benefit of having an actual teacher. 

But don't complete the whole book and all of its exercises, if you just want to become a polymath. Leverage the Pareto principle, get 80% of the benefit out of the key 20/30/40% of the concepts and exercises, and then move on. 

Another reason I used to do this a lot was that I wanted to look good by being able to mention on e.g. my resume that I had read the whole book. It's a lot more impressive to say "I read these 10 textbooks" than "I read large parts of this book, and some of this book, and a little of this one, and a lot of this other book." I've found that learning well requires keeping an eye out for these instincts. My brain is not my friend in this battle.

I now often ask myself "am I doing this, at least in part, in order to look good?". Sometimes I answer 'yes', and sometimes I do it anyways - wanting to look good isn't always bad. But there are sometimes things more important than looking good

Read Easier Textbooks Instead of Struggling Valiantly

TL;DR even though slogging through tough textbooks makes you feel sophisticated and smart, don't. 

Imagine I'm learning how to program, but I've never used a computer before. Learning to program while learning to operate a computer, will probably take longer than learning to operate a computer and then learning to program. Learning time is superadditive in terms of your ignorance / the dependencies you're missing. 

But it's worse than that. Imagine I'm learning quantum mechanics, but I don't know any linear algebra either. I'm now trying to do three things:

  1. Learn linear algebra,
  2. Learn the formal postulates of quantum mechanics, and
  3. Tie all of this into the real world.

Similarly, if I'm trying to learn fluid mechanics without knowing how to manipulate partial differential equations (PDEs), it might look trying to simultaneously

  1. Learn PDEs,
  2. Learn the physical equations, such as Navier-Stokes, and
  3. Tie all of this into the real world to explain what I already know about e.g. water and rivers and blood pressure.

But what if instead I picked up some dumb book that doesn't even have any calculus, and let it give me approximate explanations via e.g. Archimedes' principle:

Archimedes' principle states that the upward buoyant force that is exerted on a body immersed in a fluid, whether fully or partially, is equal to the weight of the fluid that the body displaces. (Wikipedia)

I breeze through this book no problem, and I can see how to tie in these laws to explain my intuitive models: "logs float more easily than rocks because rock is denser than wood, and so the buoyant force from Archimedes' principle is enough to support the weight of a log." So I'm taking care of point #3, "tie this content into the real world." 

Then, suppose I learn about PDEs and become comfortable with them. Now all I need to do to learn a piece of fluid mechanics is to learn the relevant physical equation, and then think about how it implies things like Archimedes' principle. Crucially, via this method, I'm only confused about one thing at a time. Build models in the right order!

Be Comfortable with Approximate Models

TL;DR allow yourself to learn things in order of comprehensibility: don't try to learn general relativity before Newton's law of gravitation.

In 2019, I wanted to learn a bit of chemistry. I got my hands on a high school chemistry textbook. I got stuck on chapter two because I was being too strict about forming gears-level models.

I was stuck because I was thinking about electron shells. The book acknowledged the Bohr model of the atom was wrong, electrons aren't really discrete particles orbiting the nucleus:

A wrong model.

But I got nerd-sniped into trying to understand how the electron standing wave only has solutions for certain energy levels, which is a result of quantum mechanics (or so I remember reading). I couldn't understand why, and so the knowledge felt "fake." 

It wasn't like I explicitly reasoned "this model is wrong and so I'm not going to keep reading this book", it felt more like... I just wasn't that hungry to learn this, because it wasn't "real." And I gradually stopped returning to that book.

Learn things quickly, note your confusions, and correct them later when the Anki cards show up again. Let yourself learn approximate models of reality.

Conclusion

I knew about the "read easier textbooks" advice already, but I didn't apply it. Perhaps I just didn't recognize a chance to apply it. The same forces of chaos and entropy and madness which prevented my applying e.g. Luke Muelhauser's advice, may prevent you from applying this post's advice. If you think any of this advice might help you, I recommend setting up a plan now for how and when you'll implement it.

New Comment
75 comments, sorted by Click to highlight new comments since:
Some comments are truncated due to high volume. (⌘F to expand all)Change truncation settings

use a spaced repetition system like Anki.

Spaced repetition/Anki is one of those things that seems obvious and like it makes perfect sense. However, my experiences have been pushing me further and further away from thinking that it is usually something that is useful.

I spent a year or so self-studying full time. I also spent a few months studying Leetcode stuff for FAANG interviews half-time. In both experiences I spent months using Anki, perhaps 100-200 cards where I would study the cards pretty consistently every day.

I know this might go against the research on spaced repetition, but I've found that the stuff I used to study often just doesn't stick. On the other hand, there are things that I never really did spaced repetition on that have stuck remarkably well. My impression and experiences is that obtaining a deep understanding is a lot more fruitful than doing spaced repetition.

Also, thinking along the lines of a sanity check and the outside view, are there examples of people becoming very successful due to Anki? My impression is that it doesn't happen all too often, and when it does it is because Anki happens to be a great fit for the situation, eg. passing med school exams.&n... (read more)

"are there examples of people becoming very successful due to Anki?"

It's hard to answer that kind of question, because there are plenty of examples of people becoming successful without SRS, and people who use SRS don't only use SRS.

Personally, I use Anki for professional development (so not to pass exams, but for long-term mastery).  My biggest topics are math, algorithms (ex. machine learning research topics), and programming.

It's got a number of advantages.  Sometimes it helps by keeping technical details (like trig identities or programming syntax) close to the surface, so I can finish a project faster or plow through whiteboarding a problem with a colleague.  Other times it helps by cementing a deep conceptual intuition strongly, so that I can focus on doing more advanced things (instead of rehashing the same slippery basics).  Or by just increasing the breadth of what I know is available, so I don't have to agonize over the right Google terms. Sometimes it helps by making it easier to remember people's names, accelerating my ability to network (I recommend mnemonics here, since names are rote memorization, and rote memorization is difficult with Anki).

But ... (read more)

2Adam Zerner
Hm, I seem to have different experiences than you have had. My experience as a programmer (from myself and from talking to others) is that you pick up syntax pretty quickly. For a new language there's a learning curve. Perhaps SRS would help with that, although that doesn't seem too useful. Eg. I use JavaScript every day and rarely use Python. I could use SRS for Python. Alternatively, I could wait until I need to use Python, take a few days getting familiar with the syntax, and then be fine, which seems better. Perhaps my memory is failing me, but I can't recall doing SRS over and over again ever cementing a deep conceptual intuition for me. I think looking at the same thing from different angles is often helpful, but with SRS, a given card is largely looking at the concept from one angle. Here's an example. I'm learning functional programming right now. In functional programming, operators like + and * are just functions. Instead of 2 + 3 you can do (+) 2 3 just like you'd do add 2 3. Similarly, you could call functions using infix notations by using backticks like this 2 `add` 3 and it behaves like an operator. This really cemented for me what an operator is in such a way that I don't think SRS ever could, and it did so much faster. It sounds like your point is moreso that it's hard to be as productive without SRS, but I'd like to note that there are other ways you can use a spare 15 minutes to learn rather than SRS.
5DirectedEvolution
There's definitely a tradeoff between breadth and depth/speed. On a real-world project, you can attain great speed and/or great depth on the narrow set of techniques/concepts that are directly relevant to the work you do every day. It's very expensive to maintain that kind of fluency. I've never done any programming in a language that uses infix vs. prefix operators, yet I know the difference just from a couple minutes total studying the concept at a few different widely-spaced points in time. This concept would not pose a challenge to me were I to pick up a functional programming language, although it would take time to build a habit of using them. When you overinvest in study-through-practice in a concept you've already mastered, you will forget a lot of the more broad knowledge. Anki gives you a way to retain it, or to get it back when you're ready for it. If you see yourself as having no need for a broad knowledge-base -- if you see your career as being a happy code monkey banging out programs that are within the wheelhouse you've already established, then you're probably good. But if you have a different vision for your career, it's possible that broad knowledge will be really helpful. And spaced repetition/Anki gives you a set of tools to build and maintain that broad knowledge-base. I'm really glad you posted this objection to Anki, because I think it's probably common. It's also a fair point: sometimes, we don't care about building a broad knowledge base. We're just trying to become fluent in the narrow set of skills that let us execute a technical project. Clarifying that distinction is very valuable from the perspective of budgeting your study time wisely.
6Vaniver
Interestingly, this comment made me more excited about using Anki again (my one great success with it was memorizing student names, which it's well-suited for, and I found it pretty useless for other things), because this comment has a great idea with a citation that I probably won't be able to find again unless I remember some ancillary keywords (searching "blurry to sharp" on Google won't help very much). But if I have it in an Anki deck, not only will be it more likely to be remembered, but also I'll have the citation recorded somewhere easy to search.
2TurnTrout
+1, I used Anki before my CFAR workshop. had to remind myself not to call people by their names before they even had their nametags on. This was great, because remembering names is generally a source of very mild social anxiety for me. 
2FinalFormal2
How do you do that? Do you take their picture beforehand or something?
3TurnTrout
There was a names/faces doc circulated before the workshop.
4TurnTrout
Thanks for all of these thoughts! I don't see why these should be mutually exclusive, or even significantly trade off against each other. This sounds like saying "drinking water is more important than getting good sleep", which is also true in a literal sense, but which implies some kind of strange tradeoff that I haven't encountered. I aim to get a deep / intuitive understanding, and then use Anki to make myself use that understanding on a regular basis so that I retain it. I agree, and I'm not trying to tell people to not sweat the details. It's more like, you don't have to do every single practice problem, or complete all the bonus sections / applications which don't interest you. Which is a thing I did - e.g. for Linear Algebra Done Right, and then I forgot a lot of linear algebra anyways. Another approach is, you start with the papers you want to understand, and then backchain into the concepts you have to learn in a big tree. Then you're guaranteed not to waste time. I haven't tried this yet, but it sounds very sensible and I mean to try it soon.
2Adam Zerner
Because that time you spend using Anki to retain it is time that you could spend seeking deep understandings elsewhere. Yeah, that makes sense. Sounds like we made opposite mistakes :) I've tried that. I think it's an approach to have in your toolkit, but for me it's not really my style. Eg. in learning Haskell I've been trying to skim a book, then work on practice problems, then backtrack when I notice a concept I haven't learned yet that I need to solve the problem. I always seem to get lost/overwhelmed in doing the backtracking. So I pivoted to reading the 1000+ page Haskell Programming from First Principles book, which is exactly what the title implies, and it's going well for me. Also from a more motivational perspective rather than a pedagogical one, I find that I can get into a flow state more easily with the first principles approach. With the backtracking approach, I often get stuck. With the first principles approach, it may be slow at times, but at least I'm continuing to take small steps forward and not getting stuck.
3TurnTrout
Right, but part of having deep understandings of things is to have the deep understanding, and if I just keep acquiring deep understanding without working on retention, I'll lose all but an echo. If I don't use Anki, I won't have the deep understanding later. I therefore consider Anki to help with the long-term goal of having deep understanding, even if moments spent making Anki cards could be used on object-level study.
4Adam Zerner
I guess the crux is about Anki helping with retention. If you'd lose a lot of your understanding without Anki, I agree it's worthwhile. If you'd only lose a little, it seems like a better use of time to seek deep understandings elsewhere.
1sinitsa
In my personal experience, a deep-understanding seeking study session is not portable (hard to take notes when you don't have a surface) and if it's less than an hour long, it's usually not very good (need time to 'warm up' and fully focus). Anki, on the other hand, is perfect for quick study during a break/short commute/being in a queue.  Also creating flashcards in itself makes a great interactive reread of things.
1manunamz
+1 for this response. I particularly like the blurry-to-sharp nugget, to which I wanted to add:  "low-res-breath > high-res-depth"  

There is a lot of material I like in this post.

  • The idea to use Anki for math is something I've considered but never figured out a good way to do because the natural fact unit of advanced math is more complicated than a the definition of a natural language word. I would be happy to hear details more about how you did this.
  • The tips on textbooks are useful. I have never read a textbook cover-to-cover (except athletic training textbooks) even though I have tried many times. It is reassuring to hear this is not important. It is also useful to hear the advice about reading easy textbooks. The value of reading easy textbooks is obvious in retrospect but something I never figured out on my own before reading this post.

EDIT: I now consider these example cards to be too long, although somewhat functional (pun intended). I recommend using cloze deletions instead, with small cards that are a couple of sentences long. The overlapping cloze add-on is also useful for remembering proofs in order, or sequential definitions which are hard to break into several cards. 

The idea to use Anki for math is something I've considered but never figured out a good way to do because the natural fact unit of advanced math is more complicated than a the definition of a natural language word. I would be happy to hear details more about how you did this.

I didn't really have trial and error when I started doing this in November, I just dove in and it worked. I had, after all, experience making well over ten thousand Anki cards. The key is to only make new cards composed of concepts which you already understand. I'll try to unpack how I make math cards, though, by making a new card and introspecting on what my instincts are.

Let's say I'm trying to make a card for "what is a mathematical function?" (this isn't "advanced math", but I find that my card-entry approach remains effective as the material gets harder). I... (read more)

I'm really surprised at how big your cards are! When I did anki regularly, I remember getting a big ugh-feeling from cards much smaller than yours, just because there were so many things that I had to consciously recapitulate. It was also fairly common that I missed some little detail and had to choose between starting the whole card over from scratch (which is a big time sink since the card takes so much time at every repeat) or accept that I might never remember that detail.

I'm super curious about your experience of e.g. encountering the function question. Do you try to generate both an example and a formalism, or just the formalism? Do you consciously recite a definition in words, or check some feeling of remembering what the definition is, or mumble something in your mind about how a function is a set of ordered pairs? Is the domain/range-definitions just there as a reminder when you read it, or do you aim to remember them every time? Do you reset or accept if you forget to mention a detail?

I'm really surprised at how big your cards are! When I did anki regularly, I remember getting a big ugh-feeling from cards much smaller than yours, just because there were so many things that I had to consciously recapitulate. It was also fairly common that I missed some little detail and had to choose between starting the whole card over from scratch (which is a big time sink since the card takes so much time at every repeat) or accept that I might never remember that detail.

First, the Lebesgue integration card is one of my is one of my largest; I selected it to show Isusr how I might handle a non-decomposable question. I recommend that new Anki users be careful not to do this kind of thing: find a way to decompose, whenever possible! 

I generally tolerate missing details that I would have worked out were I actually expanding the mental sketch into formal writing; for example, if I make an arithmetical error but perform the correct kinds of calculations, I count that. If I make a sign error that I wouldn't make if I had been writing it out pencil and paper, I'll usually count it. If I think momentum is a scalar quantity or something crazy, though, I fail that card. 

I want... (read more)

5voider1
Interesting. I have used multiple different approaches with Anki. Last year I had to learn all pre-university math to enter uni, and I decided to use Anki which worked out pretty well. For a lot of the problems it's about remembering what steps to take in general. So I put in practice problems so I could remember the steps. The numbers don't matter, they could've been anything.  I also tend to put in definitions, rules, theorems, standard and harder derivatives and primitives.   Now I'm in uni and I used Anki for learning linear algebra, I mostly tended towards clozes. For linear algebra I tended to put in all theorems and facts, tried to keep them short so they're easier to remember and then do complementary exercises handed out by the course. Being an autodidact myself, I must say that if the course is done right it can make learning a lot easier. Sadly many of the courses weren't as good as the linear algebra course.
6lsusr
I don't have much to contribute to your comment except to acknowledge that it's useful, quality information. An upvote didn't feel sufficient.
3Gurkenglas
(Why not simply define the integral of f as the LM of {(x,r)|x in Omega, r <= f(x)}?)
4TurnTrout
The definition I gave is the standard one, but that's an interesting question. I'm not sure I know enough measure theory to find a hole in this right away; presumably there is one, because this would be a much more succinct definition.  EDIT: TheMajor pointed me to this SE question; looks like this can work if you can prove that the aforementioned set is measurable in the product sigma-algebra. I like this definition a lot more.
4TheMajor
Mathoverflow has discussion on it. In short: * This area definition is equivalent to the standard definition, although this was (to me) not immediately obvious. * Some statements (linearity of integrals, for example) are obvious from the one definition, while others (the Monotone Convergence Theorem) are obvious from the other definition. Unfortunately, proving that the two definitions are equivalent is pretty much the proof for these statements (assuming the other definition). * The general approach of "given a claim, test it on indicator functions, then simple functions, then all integrable positive functions, then all integrable functions, then (if desired) integrable complex functions" is called the standard machine of measure theory, so there is educational benefit to seeing it.
2gjm
I haven't thought it through, but I suspect the theorems you want to prove about integrals are probably easier to prove with the definition TurnTrout gave than with that one. 
1ErrethAkbe
What process do you use to review cards? Do you look at a prompt until you can say exactly what is on the card? Or if not verbatim what tolerance do you have for missing details/small mistakes?
3TurnTrout
Not verbatim. I mark the card 'correct' if I conjured the concept in the right way. If I was supposed to do a physics calculation, I don't pass it if I could recite the answer, I pass it if I went through the correct reasoning steps. "Calculate the gravitational force of a .1kg apple on a 70kg person, when they're 1m apart" -> ".1 * 70 * G / (1^2) = ?" is marked correct.
3ErrethAkbe
How do you find doing problems/exercises from these textbooks when you have prepared using Anki? And are you finding that earlier material seems obvious when reread? Sorry if this is all coming across as critical and /or doubtful. I've tried to use Anki for theory before and dismally failed; the success you claim is very exciting and I'm trying to understand where I was going wrong. So far I think I have focused too much on creating cards that can be memorised exactly (formulae and what-not), rather than having general concept cards that are used to develop fluency and familiarity (and later understanding), which sounds like what you are doing.
4TurnTrout
I think I do find Anki’d subjects much easier when I go back to them, yes. I think that’s probably it - focus on concepts. If you like, I’d be happy to take a look at a deck you make / otherwise give feedback!
3Mo Putera
On using Anki for math, Michael Nielsen wrote up his own experience trying it in this essay: http://cognitivemedium.com/srs-mathematics  I was previously skeptical that Anki could be used for anything more complex than "basic facts" (whatever I thought that meant), but Nielsen's essay changed my mind. 

Curated.

There's a lot of nuggets to appreciate here, but I'm particularly grateful to a reminder about "Hey, you know you can just go out and learn a bunch of things? It's not mysterious, here are some concrete ideas for how to go about that."

7Ben Pace
I also really like the concrete info in the comments about how TurnTrout made math anki work for him.

Very interesting article! A big part of the outlined techniques I also kind of discovered by trial and error.

I recently also started to work more on my math skills and putting in the mathjax was getting very time consuming. To reduce that bottleneck I can recommend this small app: https://mathpix.com

PS: I stumbled once over this collection of proofs without words and they make great Anki cards to build up intuition: https://mathoverflow.net/questions/8846/proofs-without-words

2TurnTrout
thank you! This is amazing. I wish I'd known about this sooner.
1Sting
It is worth noting that Mathpix only allows 10 free snips per month. Of course, they do not tell you this until you have installed the program and created an account. 

This post is a solid list of advice on self-studying. It matches the Review criterion of posts that continue to affect my behavior. (I don't explicitly think back to the post, but I do occasionally bring up its advice in my mind, like "as yes, it's actually good to read multiple textbooks concurrently and not necessarily finish them".)

I actually disagree with the "most important piece of advice", which is to use spaced repetition software. Multiple times in my life, I have attempted to incorporate an SRS habit into my life, reflecting on why it previously ... (read more)

This is awesome! That's exactly the kind of post I wanted you to make!

About the lessons themselves:

  • I agree with the value of anki, but I have trouble finding things to ankify. My first impulse what to ankify everything under the sun, which lead to an anki burnout. Now I have the inverse problem of not finding much to put in Anki, mostly because I want to know/understand the concept before ankifying it. Or maybe I just don't read enough maths these days.
  • I want to disagree with the "read several textbook at once", but I think you're right. It's just that I'm
... (read more)
8TurnTrout
I agree; I should have written "don't complete textbooks", and have edited that part accordingly. I'm thinking some combination of "don't do all of the exercises" and "don't spend time learning every single concept in the book."

I've edited the post to incorporate a bit of wisdom I've gained over the last year. 

  • Use cloze deletions for Anki cards. 
    • Cloze deletions hide a small part of the text, and make you remember it given the rest of the content. They are fast to make and fast to review. 
    • The vast, vast majority of my new cards are cloze now, and they weren't when I first wrote this post. I think that was a mistake.
  • In the post, I used my representation theory cards as examples. 
    • What a bad choice. Those cards were too long. I ended up forgetting them (although n
... (read more)

Last year I tried re-learning calculus, and later learning some statistics, and both went kinda okay but I feel like I don't really have the scaffolding to make them actually useful longterm. I have a vague sense that it's good for me to have mathematical literacy but I'm never quite sure what to do with that belief.

This... I guess this comment is me saying "my default action is still to probably not invest much more effort here, but I am interested in anyone pitching me on a concrete strategy they think I would benefit from" (I think this is most useful coming from people who know me reasonably well and have some sense of what my deal is)

[-]Xor50

I am excited to see this sort of content here. I am currently finishing up my senior year of high school and making plans for the summer. I have decided to focus much of my free time on learning, and rationality as well as filing out my knowledge base on math, physics, and writing. These will be a valuable set of skills for collage and the rest of my life. This summer I plan to build a course on learning (free stuff on youtube) first because I want to be rigorous in my understanding of learning and teaching ensures that. Second I am looking forward to the ... (read more)

I'm glad I started using Anki. I did Anki 362/365 days this year. 

I averaged 19 minutes of review a day (although I really think review tended to take longer), which nominally means I spent 4.75 clock-days studying Anki. 

Seems very worth it, in my experience. 

Nice article. Like Nielsen, you seem to have discovered Incremental Learning for yourself.

https://supermemo.guru/wiki/Michael_Nielsen_re-discovers_incremental_reading_with_Anki

If you want the whole package in software (i.e. not only the SRS part) give SuperMemo a try :)

https://help.supermemo.org/wiki/Incremental_learning

Instead of reading a textbook with SRS and notes, skim more books (not just textbooks), solve problems, read wikis and papers, talk to people, watch talks and lectures, solve more problems from diverse sources. Instead of memorizing definitions, figure out how different possible definitions work out, how motivations of an area select definitions that allow constructing useful things and formulating important facts. The best exercise is to reconstruct main definitions and results on a topic you've mostly forgotten or never learned that well. Math is self-he... (read more)

3Eric 'Siggy' Scott
Why "instead?" I find that all of those activities enhance my SRS experience, and my SRS experience (sometimes dramatically) enhances all of those activities! Particularly since I use SRS to capture concepts, intuitions, and rich relationships, not just isolated facts.  Rote memorization doesn't work very well with SRS anyway--as with everything, it works best when used to understand a topic in its full intuitive glory.

I wrote a shortform about my current learning system (and a short follow-up comment on making it useful for solving alignment), so I figured I'd share it here since I talk a bit about Anki and other similar things in your post.


TLDR of what I've been thinking about lately:

  • There are some great insights in this video called "How Top 0.1% Students Think." And in this video about how to learn hard concepts.
  • Learning is a set of skills. You need to practice each component of the learning process to get better. You can’t watch a video on a new technique and immedi
... (read more)

It so happens that every six or seven months an article like this pops up, and revives my interest in Anki for a short while. So far, it has never stuck with me, but maybe this time…
 

I’m preparing for a bunch of Maths (linear algebra, analysis, automata and grammars, and the like) and Biology (molecular bio, immunology, genetics) exams, all due in June. I haven’t seen/used any of the subjects in question for at least a year, although I have passed each of them with perfect marks back then.

  1. Is it too late to construct the cards now as part of the refres
... (read more)
4TurnTrout
Definitely not too late to make cards. I've learned a great deal of basic chemistry in the last month or so, just studying for random 30-minute chunks and binging interesting-looking wikipedia articles. A month+ is plenty of time for spaced repetition to work its magic. For math, I recommend Anki; for non-Latex-intensive subjects like biology, I recommend SuperMemo for fast card creation while you read and review material. Unfortunately, I have yet to write up my thoughts on the latter. I like to look at the "cheat sheets" for courses and ensure I know how to reason using the key concepts. Add cards for the key concepts you find in course review notes. For definitions and concepts, I like cloze deletions - they're easy to make and easy to learn. For exercises, add small practice problems for the key concepts and the items you know will be on the tests.  Don't learn every definition and theorem. 
1Eugleo
Thanks, that sounds encouraging. I have to admit, I despise Anki. I love well-designed tools, to the point it's probably detrimental to my life. But alas, it is like it is — and so every time I see Anki (and I've made around 300 cards for it past the last few years), I have an urge to throw the computer out of the window. But, that's just a detail. I plan to use Mochi, by the way. Ok, so the main advice is: don't make a card for everything, just the important concepts. And those concepts can be found in "cheatsheets" and "course review notes", it seems — unfortunately, I don't have any of those things. So, I'd be grateful if you could illustrate what do you think constitutes a major, Anki-worthy piece of information. Let's say you're trying to revise (in the context of my current exam prep) a base chapter on vector spaces from Linear algebra done right. You probably want a card with the VS axioms. What else do you view as important enough?
2TurnTrout
Why not use Google for notes from other schools?
1Eugleo
If the plan was to understand the given subject, then yes, that would work. And of course, that is the ultimate plan. However, the more pressing matter are the exams. I would be afraid that the intersection of the two programs won't contain all of the important concepts; that's my experience with "other" textbooks (meaning, other than the few recommend ones), at least.

Leverage the Pareto principle, get 80% of the benefit out of the key 20/30/40% of the concepts and exercises, and then move on.

This is hard to instrumentalize regarding difficulty. I find that the hardest exercises are likeliest to be skipped (after struggling with them for an hour or two), but it doesn't follow that I can expect the easier ones (which I happened to have completed) to lie in that key 20%.

2TurnTrout
I like to randomly sample X% of the exercises, and read the rest; this lets me know later whether or not I missed something important. Simple rules like "do every fifth exercise" should suffice, with the rule tailored to the number of exercises in the book.

Fantastic insights!  I especially like how you've articulated the value of using "easy" books to connect concept to the "real world."  I've certainly run into the problem of trying to Ankify a big dense book, and getting bogged down with it.

Incidentally I'm also working through the Book of Why and Causality by Pearl.  Great progression (starting with one, then returning to the other).

I've been using Anki for math and computing for several years now, and one word of warning is that your big, dense cards (i.e. screen-shotted definitions) are t... (read more)

I think you'd get a lot of value out of using incremental reading, it improves the learning to memorisation workflow tremendously. Currently SuperMemo is the best at IR. (Post I wrote about this: https://gingerjumble.wordpress.com/2020/08/28/the-main-reason-you-should-switch-from-anki-to-supermemo/)

I'm a bit skeptical of what you claim because it is so different from my approach to becoming proficient at pieces mathematics: usually I will work through progressively more-complicated problems in excruciating detail. I don't claim that this is the most efficient method, and it would be nice to find such an approach, just that memorization methods have usually lead to me agreeing with the mathematics, rather than really understanding it.

But maybe you are using Anki differently to how I expect. How exactly do you review cards? Do you look at a prompt until you can say verbatim what is on the card? Or if not verbatim what tolerance do you have for missing details/small mistakes?

3Eric 'Siggy' Scott
Anki's value depends a lot on how you use it. If you use it for rote memorization, without grokking concepts deeply---then what you get is rote memorization.  Think of memorizing capitals without learning anything else about states and nations. But if you work hard to identify conceptual landmarks and add good questions to tie it all together (more like a high quality PowerPoint presentation), then it can be amazing and facilitate retention of rich intuitions for years.  Think of memorizing major topographic landmarks on a map and a bit of history, so that you have something to relate capitals to in context. Incidentally, rote-memorization cards are actually harder to review with Anki: the "glue" holding them in memory starts to fade after a month or so, so eventually the drift toward "ease hell" and otherwise become unpleasant to review.
2TurnTrout
I went into detail on how I use Anki for math in this thread. Feel free to comment there with further questions.

In early 2020, I had an interview where I was asked to compute .

A trick I learned from Jaynes:

... or, if you want to be a smartass about it, write "" in place of "" and tell people you're "differentiating by 1": .

All you actually need to remember is .

Neat article. Few things I myself have been struggling with and would like some insight into are:

  1. Were you able to address the issue of ankifying the algorithms? If yes, how?
  2. How do you handle the Anki burnout(or for that matter Anki inertia)? I am guessing with multiple subjects at a time, you at least have to review 100-200 odd cards per day(maybe more), and based on the non-atomic nature of your example cards, it seems to me that getting yourself to review that many cards every day is not a sustainable activity?
  3. How do you preserve your sanity between
... (read more)
2Eric 'Siggy' Scott
Not the OP, but got a few opinions there: 1. Algorithms can be Ankified with the same strategies that work for other subjects: chunk the basic intuition into a short overview card, use diagrams, and hit the "basic idea" of how it works before adding more details cards for particular tricks & minutia.  As always, most cards should take <15 seconds to review (including time spent reading the card). 2. Personally, I avoid burnout a few ways:  A) pair Anki with activities like taking a daily 1-mile walk or doing dishes (iOS voice commands help with the latter, so you can review hands-free), B) don't be shy about reviewing my professional development cards during work hours, C) deleting social media from my phone, so Anki is the go-to gap-filler app, D) always be learning a few new cards in each deck (since decks with 100% old material are less fun to review), E) refactoring: I set a reminder to work on marked cards every few days, so I can break down difficult cards into higher-quality material.  A little maintenance goes a long way. F) card design, card design, card design.  Well-designed cards are far less fatiguing to review than hastily made ones. 3. Not sure I understand.  Anki kind of abolishes the line between reviewing and learning: you do both with the same system.  If you mean motivating yourself to study old material---my trick is to review from large, combined superdecks, so there is no pause between "Oh, I finished my learning decks, time to move on to old decks."  If I give myself that break, I'm more likely to procrastinate, but if I treat it all as one big deck there's motivation to finish it all in one go. 4. I actually find it motivating.  Learning complex subjects is usually hard, because when you hit something hard there's an urge to procrastinate rather than figure out what's blocking you.  Anki removes these energy barriers for me: I can always learn a little more about quantum mechanics or whatever by adding a card or

Perhaps I am overthinking this, but when it comes to applying your knowledge gained through spaced repetition, is there a difference in how effectively you can apply it in these two situations?

  1. You spend time memorizing what a function is. A function comes up in a problem in a very similar format to how it looked on your Anki card. You are able to understand it because the prompt of the function in the problem causes you to recall your understanding from studying and then apply it.
  2. You spend time memorizing what a function is. A problem comes up that can be
... (read more)
1TurnTrout
I don't think spaced repetition runs into the failure mode of #2 if you do it competently. Good spaced repetition prompts you to think about the concepts on a spaced, expanding basis, not to recite the magic words on the back of the card. When I study a card, I'm trying to train urge to understand concept in prompt -> concepts on back of card, not words in prompt -> words on back of card. For example, when I learned French with Anki, I translated entire sentences English-to-French (and vice versa). I learned vocab words by just finding sentences where I already knew how to say everything except the new word. By learning sentences containing the word for e.g. "swamp", I learned how to say "swamp" in normal, fluent conversation, without having to remember the exact sentence I used for Anki (as #2's model might suggest).  I don't see any real difference in quality between normal study and Anki study, if you make your cards well.

I have multiple times tried to get into an Anki habit and failed to keep it up. I think the main thing that makes me stop is that I try to make nearly every sentence of something that I'm studying into a card, because I have no idea what's worth remembering and what isn't. (As a general rule, throughout life, I suck at prioritizing.) The other thing, though, is that it feels like Work and things which feel like Work are Unpleasant and I procrastinate them. Do you have any advice for getting over that hump?

2Raj Thimmiah
In general, when I first started using SRS I had this issue too. Over time, I got a fair bit better of it with 1 or 2 rules: -I should be able to make at least 1 connection from the idea to something else (filter out only shallowly understood things) -I should be able to either connect it to a situation of real life applicability or a goal I didn't make it only: I need to be able to figure out applicability when making card, because a lot of useful knowledge might not that have it as obvious. But at the least, I should be learning it in a real context. E.g. some of my main goals now are: -learn UX design -understand how SRS/IR can be used to create genius -get better at teaching -become more resillient to disruption (emotionally) Those are specific-ish and learning in context of those instead of I can use this sometime in the next decade is a useful filter
2TurnTrout
One way to start is to put in things you find yourself looking up. If you're reading math and you have to keep looking up the quadratic formula, put it in. If you're reading the 'Nash equilibrium' Wikipedia page, put in a few examples (and maybe even the formal definition!).  Then, while you're reading a textbook, you might develop a taste for what things you'll probably look up later.  I like to think about Anki as giving me enduring experience points, which is Fun and Gratifying. Once you find it really work for you on something that matters, maybe you'll find it more exciting? I can only guess - I've naturally enjoyed Anki since nearly the beginning.
1MSRayne
Hmm... I could try actually counting experience points. Like, each flash card reviewed grants a point, and every time I reach, say, a new Fibonacci Number of points, I gain a level and... um... stuff! The idea of leveling up really isn't very motivating by itself, but it would help. Here's one idea: when reading a textbook or anything else I want to memorize, I might try to come up with just one question and answer pair about each page, and make a card out of that. Summarize the most important info on that page. Anything that's not too info-heavy, that should work. In things which are info-heavy, your method should work quite well.

Random Anki tips

When using Anki for a scattered assortment of topics, is it best to group topics into different decks or not worry about it and just have a single big deck.

If making multiple decks, would Anki still give you a single "do this today" set? 

I've only used it in the context of studying foreign language. 

3Jacob G-W
I prefer putting everything into one big deck for a few reasons. 1. It forces you to do all the cards every day. When I had different decks, I would do the "main" deck every day, but then not do the other decks. 2. I have not read any research on this, but it seems like interspersing the cards could make the cards less likely to anticipate and force you to learn better. 3. It also just mixes up the monotony and makes it more fun.

Why did you write autodidacting instead of selfteaching? It just put a google search between me and reading the article...

4TurnTrout
It was the first word to come to mind. I changed the title, thanks. 
1Tim Liptrot
Thanks! Looks good.

You mention that planting a seed of Anki usage in 6th grade is a good thing. Do you have any thoughts about how to help a child get started with an Anki habit?

3TurnTrout
Try checking out the external links in Gwern's post: https://www.lesswrong.com/posts/dtCfxYubZgRnEkGpQ/a-second-year-of-spaced-repetition-software-in-the-classroom  https://www.gwern.net/docs/www/bentilly.blogspot.com/f83ff5823759c2f47e889fa894273d84fa6551d0.html 
3TurnTrout
I don't, unfortunately. I don't have any special insight into children at this point in my life, and I can barely remember what my mental life was like back then. (Too bad I didn't use Anki to remember what it was like to be in 6th grade!) There are always instrumental things to appeal to (e.g. "by using this application, you can do your work faster and have more time to play with your friends"), but that might take away from their intrinsic motivation if they view Anki as just a means to more free time. I'd like to defer to LessWrong users who have actual experience with kids.
1Raj Thimmiah
The creator of spaced repetition has talked about how usage in children of SRS is not a great idea here   Roughly idea is that self-direct learn drive based learning is needed for coherent, well connected models. It's sort of like how I learned trig in high school and I can't apply it to a real world problem to save my life. If I'd learned trig for myself, when I thought it'd be useful, it'd be way more applicable and not be so siloed to 'this is how you use trig to solve math problems on a test'