Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

A vote against spaced repetition

45 Post author: ancientcampus 10 March 2014 07:27PM

LessWrong seems to be a big fan of spaced-repetition flashcard programs like Anki, Supermemo, or Mnemosyne. I used to be. After using them religiously for 3 years in medical school, I now categorically advise against using them for large volumes of memorization.

 

[A caveat before people get upset: I think they appropriate in certain situations, and I have not tried to use them to learn a language, which seems its most popular use. More at the bottom.]

 

A bit more history: I and 30 other students tried using Mnemosyne (and some used Anki) for multiple tests. At my school, we have a test approximately every 3 weeks, and each test covers about 75 pages of high-density outline-format notes. Many stopped after 5 or so such tests, citing that they simply did not get enough returns from their time. I stuck with it longer and used them more than anyone else, using them for 3 years.

 

Incidentally, I failed my first year and had to repeat.

 

By the end of that third year (and studying for my Step 1 boards, a several-month process), I lost faith in spaced-repetition cards as an effective tool for my memorization demands. I later met with a learning-skills specialist, who felt the same way, and had better reasons than my intuition/trial-and-error:

  • Flashcards are less useful to learning the “big picture”
  • Specifically, if you are memorizing a large amount of information, there is often a hierarchy, organization, etc that can make leaning the whole thing easier, and you loose the constant visual reminder of the larger context when using flashcards.
  • Flashcards do not take advantage of spatial, mapping, or visual memory, all of which the human mind is much better optimized for. It is not so well built to memorize pairs between seemingly arbitrary concepts with few to no intuitive links. My preferred methods are, in essence, hacks that use your visual and spatial memory rather than rote.

 

Here are examples of the typical kind of things I memorize every day and have found flashcards to be surprisingly worthless for:

  • The definition of Sjögren's syndrome
  • The contraindications of Metronidazole
  • The significance of a rise in serum αFP

 

Here is what I now use in place of flashcards:

  1. Ven diagrams/etc, to compare and contrast similar lists. (This is more specific to medical school, when you learn subtly different diseases.)
  2. Mnemonic pictures. I have used this myself for years to great effect, and later learned it was taught by my study-skills expert, though I'm surprised I haven't found them formally named and taught anywhere else. The basic concept is to make a large picture, where each detail on the picture corresponds to a detail you want to memorize.
  3. Memory palaces. I recently learned how to properly use these, and I'm a true believer. When I only had the general idea to “pair things you want to memorize with places in your room” I found it worthless, but after I was taught a lot of do's and don'ts, they're now my favorite way to memorize any list of 5+ items. If there's enough demand on LW I can write up a summary.

 

Spaced repetition is still good for knowledge you need to retrieve immediately, when a 2-second delay would make it useless. I would still consider spaced-repetition to memorize some of the more rarely-used notes on the treble and bass clef, if I ever decide to learn to sight-read music properly. I make no comment on it's usefulness to learn a foreign language, as I haven't tried it, but if I were to pick one up I personally would start with a rosetta-stone-esque program.

 

Your mileage may vary, but after seeing so many people try and reject them, I figured it was enough data to share. Mnemonic pictures and memory palaces are slightly time consuming when you're learning them. However, if someone has the motivation and discipline to make a stack of flashcards and study them every day indefinitely, then I believe learning and using those skills is a far better use of time.

Comments (59)

Comment author: Metus 10 March 2014 07:44:28PM 38 points [-]

If there's enough demand on LW I can write up a summary.

Please do.

Comment author: Gunnar_Zarncke 11 March 2014 11:47:56AM *  3 points [-]

Seconded. There are quite some comments referencing memory palaces here, but no post.

See also http://en.wikipedia.org/wiki/Method_of_loci

Note that Sherlock uses this method (with funny visualization in the series).

Comment author: Emile 11 March 2014 05:18:17PM 2 points [-]

(your formatting is screwed up)

Comment author: Gunnar_Zarncke 11 March 2014 08:29:53PM 1 point [-]

corrected

Comment author: ancientcampus 18 March 2014 03:28:35PM 1 point [-]

Now that this topic is buried on page 2, I don't know if anyone will see this post. However, I've begun work on my tutorial. I intend to do a "demo", constructing a memory palace. Is there a particular list (of about 5-9 items) that people might find universally useful? Memory palaces really need to be constructed by the individual, but for the demo, I'd prefer to to something at least mildly relevant.

Comment author: Metus 18 March 2014 03:47:06PM 0 points [-]

Great for starting to work on this.

Is there a particular list (of about 5-9 items) that people might find universally useful?

I don't understand what you want us to say here. What kind of items?

Comment author: ancientcampus 14 March 2014 02:44:06PM 0 points [-]

It's encouraging to see so much interest! I'll try to pull something together in the next few weeks.

Comment author: Transfuturist 13 March 2014 07:04:21PM 0 points [-]

Thirded. It's nearly impossible to find a definitive guide on how to do memory palaces.

Comment author: polymathwannabe 13 March 2014 07:11:51PM -1 points [-]

Possibly stupid question: didn't Giordano Bruno write the go-to guide for that?

Comment author: Transfuturist 13 March 2014 08:44:37PM *  0 points [-]

Yeah, in Latin. I don't think it's been translated to English, and it's somewhat suspect now, considering it's centuries old and was part of Bruno's obsession with the occult.

Comment author: polymathwannabe 13 March 2014 09:36:03PM -1 points [-]
Comment author: Transfuturist 13 March 2014 11:58:03PM 0 points [-]

I've seen it already. As far as I know it was only history, do you know otherwise?

Comment author: polymathwannabe 14 March 2014 02:14:16AM 0 points [-]
Comment author: Transfuturist 14 March 2014 04:01:23AM 0 points [-]

Again, not a guide.

Comment author: Cyan 15 March 2014 03:00:01AM 0 points [-]

No, it pretty much is. It's also got other stuff too is all.

Comment author: Jayson_Virissimo 10 March 2014 11:52:23PM *  14 points [-]

Here is what I now use in place of flashcards:...

How long have you been using these alternatives to flashcards? How likely is it that you are still in the honeymoon phase?

Comment author: ancientcampus 14 March 2014 02:07:38PM 0 points [-]

Great question. It's been a long transition from flashcards. I developed the picture technique myself over a year and a half ago. If I learned a fact a year ago, if I studied it using the picture technique, I have about 70% recall if I reviewed it once 6 months ago. If I used flashcards for a month, I have 5% recall now. If I used flashcards continuously (~1-2 hrs per day) for 6 months then stopped using the deck 6 months ago, I have about 10% recall.

(I did a very cursory self-test, then approximated these numbers. It's very far from perfect, but I didn't pull them from a hat. For the memory palace numbers, this is pulled from what I'm currently studying, so is reliable.)

I am still in the honeymoon phase for memory palaces. If I reviewed them once a week after creating them, I have about 90% recall one week following that. I currently have 100% recall for the 4 lists ~10 items long that I made last month and have reviewed 3 times each, but haven't reviewed in the last week.

Comment author: ChristianKl 11 March 2014 09:48:29AM 11 points [-]

The significance of a rise in serum αFP

I think that's mainly an example where it's not straightforward to make good cards.

Basically I would get a list that rise in serum αFP does X, Y and Z. Then I would make cards:

Does higher or lower serum αFP does X?
Does higher or lower serum αFP does Y?
Does higher or lower serum αFP does Z?

I personally formulate the cards a bit differently but that's the core.

Ven diagrams/etc, to compare and contrast similar lists. (This is more specific to medical school, when you learn subtly different diseases.)

I do have Anki cards that contain Ven diagrams. At the beginning the Ven diagram is shown empty and the user is asked where a given item belongs on the Ven diagram.

As answer card the whole Ven diagram is shown and the item that the user had to place is highlited with a special color. I haven't yet automated the production of such Ven diagrams but I think that's part of the future of Spaced Repetition Learning.

Your mileage may vary, but after seeing so many people try and reject them, I figured it was enough data to share. Mnemonic pictures and memory palaces are slightly time consuming when you're learning them.

If you want to learn information the day before a test and get a good score then Anki won't help you for that purpose. That's not what it's made for. Anki exists to prevent you from forgetting information. There a good chance that you will forget your memory palace information in a year. Especially because memory palaces have a reuse problem. If you use the same palace for multiple lists you get conflicts.

Your mileage may vary, but after seeing so many people try and reject them, I figured it was enough data to share.

I would guess most of those people violate various of Wozniaks rules. If I look at the most popular Anki deck about medicine that's on Anki shared deck list I fully understand that learning that way isn't optimal.

Comment author: Emile 11 March 2014 03:52:10PM *  3 points [-]

I would guess most of those people violate various of Wozniaks rules. If I look at the most popular Anki deck about medicine that's on Anki shared deck list I fully understand that learning that way isn't optimal.

That's a good point; I'm frequently frustrated by existing decks because they tend to include too much information in the answer, or not use as many cues as they could, or not repeat themselves as much, etc. A lot of people do seem to be using Anki sub-optimally, which may explain ancientcampus's observations.

Comment author: benkuhn 10 March 2014 10:26:53PM *  31 points [-]

Good information! This is really more "a vote against flashcards" than "a vote against spaced repetition", though, at least given your concrete issues with flashcards. Spaced repetition is an algorithm for figuring out when to review material that you want to memorize; flashcards are one thing that spaced repetition is applied to, because it's easy to stick flashcards in a computer. As far as I know, no matter what object-level mnemonic devices you're using, spaced repetition is still strictly better than "when I feel like I'm forgetting" or "right before a test" or any of the other obvious review strategies, if you can deal with the cognitive load of scheduling things, or get a computer to do it for you.

Is there space for some sort of SRS that allows for input of the more helpful types of memorizations that you listed (pictures, venn diagrams, etc.)?

Comment author: ancientcampus 14 March 2014 01:44:50PM *  3 points [-]

You are absolutely correct; this is a hair worth splitting. I meant "spaced repetition flashcards", and I have only seen formal spaced repetition algorithms applied to flashcards. In my particular case, I end up with 30 or so "pages" of related information, as opposed to 500 flashcards. I agree that using spaced repetition algorithms to tell me when to study which page is likely better than alternative methods, though I haven't found an algorithm optimized for that sort of thing, and at the moment my intuition of "when I'm forgetting" is sufficient for the low number of separate objects to study.

[For this comment, I will use the term "page" to mean any collection of related information, be it a list, table, memory palace, notes on a single topic, etc.]

To be explicit: I vote against using spaced repetition (of any sort) to identify specific facts within a "page" of information. When reviewing a page, of course you can go quickly over the parts you know well and dwell on the parts you don't, but I would encourage the student to not completely ignore the other details "until it's time."

As an example: I have a collection of facts that can be represented as a large table or as individual facts. If I study it in a table, then I get the advantage of keeping the "big picture in mind", plus I can activate spatial memory as well as rote. If I study it as separate facts: the Pro is I can use spaced repetition to greater effect, not reviewing the parts I know better, but the loss of the picture and the spatial memory makes it not worth the cost. (Note that the "big picture" isn't a single sentence I can write down; it's noticing trends in the data, how column A and B are similar except in in key areas, etc.)

As always, my experience is only in high volumes of information that can be organized vaguely hierarchically. (That said, I think if you look hard enough, you can find categories or hierarchies for any large volume of information, outside of truly random things like the sequence of a deck of cards.)

Caveat #2: "self testing" is really important! So if you are quitting flash cards, make sure you find some new way to quiz yourself, don't just read passively.

Conclusion: -Spaced repetition algorithms might be viable, though I don't know any suitable to my needs -I claim that spaced repetition flashcards are not useful for large volumes that have categories and/or relationships between facts. -benkuhn rightfully points out that my vendetta is mostly against flashcards, or any method focusing on "terminal facts" in random order without also studying closely related facts.

Life application: 1) If you see a table, don't vaporize it into your flashcards. Rather, study the table. 2) If you see a mass of new data: ask yourself if you can organize it in a way meaningful to you, then study it in the structure you built.

Comment author: bogus 14 March 2014 03:41:48PM 0 points [-]

If I study it as separate facts: the Pro is I can use spaced repetition to greater effect, not reviewing the parts I know better, but the loss of the picture and the spatial memory makes it not worth the cost. (Note that the "big picture" isn't a single sentence I can write down; it's noticing trends in the data, how column A and B are similar except in in key areas, etc.)

Why not do both? The thing about SRS flashcards is that facts that you have strong recall of are nearly costless to add, because the review schedules get spaced into very large intervals (weeks, then months or even years), and it only takes a split second to see a card and realize it's easy. So you could learn from tables if that really is more efficient for you, but having the data in SRS card form as well is a good insurance policy.

Comment author: ancientcampus 14 March 2014 05:27:43PM 0 points [-]

Frankly, because at the volume I was running, it was far too great an investment of time. When I stopped, I had about 75-100 scheduled (learned) flashcards per day if I added nothing the day before, though I usually added 60-some every day. The cards would take me 1-2 hours, and the amount kept building as I was adding to it faster than I was pushing them "out".

Additionally, here our mileage may vary, but even with easy flashcards I occasionally find myself staring dumbly at it for ten or more seconds before I realize what it's asking and smack my head. So I end up trimming out the stupid-easy ones, but that starts to defeat the purpose. Thus, for myself personally, I won't duplicate in flashcards what I'm already memorizing elsewhere.

I know that everyone is different, so this is just my experience and what I have observed in other people. If others continue to have success with SRS, then far be it from me to insist they fix what isn't broken.

Comment author: bogus 14 March 2014 06:52:04PM 0 points [-]

even with easy flashcards I occasionally find myself staring dumbly at it for ten or more seconds

To me, that suggests that the card is either too complex and should be split up further, or that you simply do not have solid recall of the relevant facts, so you should just flip to the answer and mark the card difficult. It's quite normal to forget even some basic info over time; the point of SRS is to refresh these memories at the lowest viable cost.

Comment author: ancientcampus 18 March 2014 03:23:04PM 0 points [-]

I appreciate the input, truly, but I can confidently state that's not the case in my situation. This happens even on the simplest questions that I know cold, and is a problem with mental fatigue, monotony, and reading. After the 100th card, I would expect similar results from "what color is the sky" occasionally. I highly doubt I am dyslexic, but I might be a little ADHD. Once again, I do not presume everyone has similar results, but when I did 150 cards per day (and lord help me if I missed a day), easy cards posed a significant drain on my time and mental energy.

Comment author: bogus 20 March 2014 12:04:48PM 0 points [-]

Interesting. If you get that kind of mental fatigue from reading, maybe flashcards really are relatively inefficient for you. If it turns out that dyslexia is the problem, there is an open source font that can help with that issue. Some people have set their SRS up to read questions aloud using computer-generated speech. But yes, most of the time it's a signal that you should take a break and perhaps switch to some other activty.

One thing that's relevant to this discussion is that the latest SRS versions can actually cope quite well with missed reviews. Yes, you'll still be presented with a backlog of suggestions, but the system gives you improved credit if you can recall a card easily even after the increased delay. Because that implies your memory of it was quite good in the first place, so it can be refreshed less often in the future.

Comment author: Slackson 11 March 2014 04:53:19AM 2 points [-]

Anki is very extensible. I think writing easy-to-use Anki plugins would be a great way to practice coding and get some useful stuff out there. In fact, I'm adding that to my list of things to look into.

Comment author: Mark_Neznansky 14 May 2014 03:25:36PM 1 point [-]

This is an idea I had only toyed with but have yet to try in practice, but one can create meta-cards for non-data learning. Instead of creating cards that demand an answer, create cards that demand a drill, or a drill with a specific success outcome. I find it a bit hard to find "the best example" for this, perhaps because the spectrum of learnable-skills is so broad, but just for the sake of illustration: if you're learning to paint, you can have "draw a still object", "draw a portrait", "practice color", "practice right composition", "practice perspective" &c, cards. After you finish your card-prompted drill, you move to the next card. Or if you're practicing going pro at a game (with existing computer program AIs), you can have "Play AI X in a game situation S and achieve A", "Practice game opening against AI until (able to reach a certain state)", "practice a disadvantaged end-game situation against AI and bring the game to a draw", and so on, cards. Of course reviewing the cards would take longer, but they are only meant to be used as scaffolding to harness the Anki spacing algorithm. The numeric parameters of the algorithm might need an adjustment (which is easy to do in Anki) for that, but I think that qualitatively it should work, at least for specific skills. Of course, this set-up, especially if it needs a major parametric-overhauling[1], is an investment, but every human breakthrough required its avant-gardians.

[1] Which is not granted: perhaps the algorithm is only problematic at the beginning of the "learning", being too frequent, in which case you can just "cheat" carefully and "pass" every other review for a while, which is not a major disturbance. Or, on the contrary, perhaps "well learned cards" (interval > 3 months, or even 1 month, for example) should be discarded for more challenging ones (i.e, "beat the expert AI" replacing "beat beginner AI", or "juggle 5 balls while riding a unicycle on a mid-air rope" replacing "juggle 4 balls"), which is even less of a problem, as you should immediately recognize well-learned skills (i.e. "practice counting up to 20").

Comment author: bogus 11 March 2014 01:21:53PM *  1 point [-]

Is there space for some sort of SRS that allows for input of the more helpful types of memorizations that you listed (pictures, venn diagrams, etc.)?

Unfortunately, Anki and other SRS software do not seem to support card dependencies (i.e. "only show card X if cards Y, Z... are firmly set in memory, as predicted by the spaced-repetition model"). If that was supported, using SRS to memorize heavily-structured data would simply be a matter of setting up appropriate dependencies. (A "memory palace" is really the same thing, except that the highest level of your hierarchy is a spatial model, i. e., your "palace" or "room", containing the information you want to memorize.)

(One could generalize this by also supporting the option: "prioritize card X when cards Y, Z... are more easily recalled." Then the deck could even include such things as mutual dependencies, loose associations etc. and the app could use them to "branch out" from what you know already, showing info that can most easily be committed to memory in a highly clustered way.)

Comment author: ChristianKl 13 March 2014 05:03:11PM 2 points [-]

Unfortunately, Anki and other SRS software do not seem to support card dependencies (i.e. "only show card X if cards Y, Z... are firmly set in memory, as predicted by the spaced-repetition model"). If that was supported, using SRS to memorize heavily-structured data would simply be a matter of setting up appropriate dependencies

The general idea of Anki is that you learn the knowledge first and then put it into Anki to avoid forgetting it.

Comment author: bogus 13 March 2014 10:04:56PM *  1 point [-]

The general idea of Anki is that you learn the knowledge first and then put it into Anki to avoid forgetting it.

Not necessarily. In some cases, the flashcard format is quite suited for learning new content as well - especially such things as vocabulary. Allowing inter-card dependencies could easily expand on these use cases.

It would also be directly useful in language learning: for instance, you could memorize some vocabulary words and then be prompted to learn related idioms, or collocations (i.e. words that are "often used together"). Despite its usefulness, this content is quite hard to memorize effectively in the absence of such specialized support.

Comment author: Mark_Neznansky 14 May 2014 02:48:21PM *  0 points [-]

This is not quite a "tech-tree" dependency structure, but you can use tags to stratify your cards and always review them in sequence from basic to dependent (i.e., first clear out the "basic" cards, then "intermediate", then "expert"). Even if the grouping is arbitrary, I think you can go a long way with it. If your data is expected to be very large and/or have a predictable structure, you can always go for a "multiple-pyramid" structure, i.e, have "fruits basic" < "fruits advanced" < "fruits expert", "veggies basics" < "veggies pro" tags &c, and perhaps even have an "edibles advanced" > veggies & fruits tag for very dependent cards.

On the assumption that the Anki algorithm works, just "reviewing down" to an empty deck every tag and proceeding thus sequentially from tag to tag, I think this would work too. Even if it so happened that by one Sunday you forgot "What is an American president" (basic) fact, it might still be profitable to rehearse that day the "Washington was the first president" card, despite the "20 rules" mentioned somewhere above. Presumably, if you had forgotten what a president is, the appropriate card is probably going to appear for review in the next few days, and so with a consistent (or even a semi-consistent) use of Anki, it would probably turn alright. This is more for the anecdotal sake, but this reminds me a time when I burst out laughing out loud while at the dictionary. I was reading at the time "Three Men in a Boat", and there was one sentence in which I didn't know 2-3 of the words; the punchline clicked as I read the definition of the last of them.

Either way, somewhere higher on this commenting thread, I have also thought about the possibility (or rather, lack of) of creating dependencies in Anki. I'm actually thinking of creating an add-on/plugin to enable that--- I'm learning Python these days (on which Anki runs), and I'm just about to start grad school (if I get admitted), so it seems like just the right time to make this (possibly major) meta-learning investment.*

* Not to mention that, since I'm learning Python, it's also a (non-meta) learning investment. Win-win.

Comment author: Mark_Neznansky 14 May 2014 04:59:08PM 0 points [-]

It was to be expected-- Someone had already created a "hierarchy Tags" addon: https://ankiweb.net/shared/info/1089921461

I haven't used it myself, but a comment there said "Simple, nice, and easy."

Comment author: Baughn 10 March 2014 08:53:56PM *  7 points [-]

What would be a good introduction to memory palaces? What did you use?

Comment author: Ben_LandauTaylor 10 March 2014 09:28:21PM 11 points [-]

The video of Brienne's presentation at the South Bay meetup is the most useful guide I've encountered.

Comment author: ancientcampus 14 March 2014 02:21:29PM 0 points [-]

Truthfully: I tried and failed miserably when I tried shorter articles (eHow, wikipedia, etc). My study skills coach taught me; his name is Ryan Orwig and teaches medical professionals around the country; he's talked with memory champions and has ~10 years refining the technique specifically for medicine (but it works with any large body of facts, I think it would help with Law too). So, unfortunately there's no resource I can point to. I can't share his powerpoint, but I can make and share my own, which I will do when I have time.

That said, I just skimmed Brienne's presentation in Ben_LandauTaylor's link, and it seems to hit many of the points I like. I'll listen to the whole thing later to see what I have to add.

Link to Ryan Orwig's class, his travel schedule is on the right: thestatprogram.com

Comment author: nbouscal 11 March 2014 01:24:52PM *  6 points [-]

Can we please taboo "memorize" here? It seems to me the problem is conflating two different mental activities: 1) developing the ability to recall specific discrete responses to stimuli, and 2) gaining cohesive understanding of the component pieces of a "big picture", and the connections between them. It seems not at all surprising that the best approach for each would differ, with spaced repetition being good for the former and things like venn diagrams, mnemonic pictures, and memory palaces being good for the latter.

Concrete examples for clarity: In the first category would be vocabulary learning, e.g. mapping the stimulus 国 to the response "country". In the second category would be, say, abstract algebra: learning the group axioms and how they relate to the semigroup axioms in one direction and the abelian group axioms in the other direction.

Comment author: bramflakes 10 March 2014 08:27:44PM 6 points [-]

Memory palaces. I recently learned how to properly use these, and I'm a true believer. When I only had the general idea to “pair things you want to memorize with places in your room” I found it worthless, but after I was taught a lot of do's and don'ts, they're now my favorite way to memorize any list of 5+ items. If there's enough demand on LW I can write up a summary.

How good is your ability to internally visualize? I've been putting off investigating memory palaces for my own use because I find it very difficult to use a mental paintbrush.

Comment author: MathiasZaman 11 March 2014 12:07:55AM 6 points [-]

My local LW group did some exercises with them and they worked rather well. I don't have a very good ability to internally visualize but was still able to remember significantly more with a memory palace.

Find a friend to try it a couple of times and see how it works for you.

Comment author: ancientcampus 14 March 2014 02:40:09PM 0 points [-]

A great point. I can confidently say mine is at least "average", likely above average. I consider myself a "visual learner," with good "story memory" and I agree that as such memory palaces are a particularly good for me. However, when I use the technique, I'd say it's mostly non-visual. I'd guess it's 20% me "seeing" the room, 10% "everything else" (texure, sound, smell, emotion, all of which I find much harder but make deliberate effort to employ), and 70% conceptual "The spaceship is crashing through the door, sending shards of wood scattered across the bedroom". That is one of the many "secrets" that make the technique so useful to me: most every object should perform an action that would in real life permanently damage or alter the room. With tricks like that, I think it is helpful for most all people, even those not visually inclined.

For what it's worth: I have no data on this myself, but my study coach posits that everyone can do it, some have more trouble than others, but when done well it's so effective that most everyone should benefit. He says most of his students are resistant, but almost all of them profess loving it once they develop the skillset to use it.

Comment author: Vladimir_Nesov 11 March 2014 12:01:03AM 3 points [-]

Here is what I now use in place of flashcards

This sounds like a false dilemma. Maybe on their own, flashcards are not very effective, and on their own, mnemonic devices work well enough (for you). But this is not enough to conclude that flashcards are not useful after you adopt mnemonic devices. Maybe the additional effort (of also using spaced repetition, in addition to mnemonic devices) doesn't significantly improve long term retention, or maybe it does.

Comment author: SoerenMind 11 March 2014 04:11:36PM *  2 points [-]

There's an Anki Deck on "The 20 rules of formulating knowledge [in SRS]". It's highly recommended for frequent Anki users. Here's some examples:

  • Start with the big picture
  • Refer to other memories
  • Use mnemonic techniques
  • Use imagery
  • Use graphic deletion (e.g. for diagrams, anatomy etc.)
  • Avoid sets ("contraindications of Metronidazole" would probably be a set) ...

So it seems that many of the points you mention are addressed if you use Anki effectively. Your post makes sense though: In my impression 1) most people are not using it as effectively as they could 2) it's not obvious how to use it effectively 3) effective use of SRS takes time and practice. There's certainly also cases where it's just not always the best technique.

Article: http://www.supermemo.com/articles/20rules.htm Deck: http://alexvermeer.com/download/How-to-Formulate-Knowledge.anki

PS: I second the post on memory palaces, sounds really interesting!

Comment author: ChristianKl 11 March 2014 04:46:08PM *  7 points [-]

While recommending the article I wouldn't recommend the deck because of cards like:

Front: redundancy can be performed by repeating information using...
Back: various methods

To me that card looks likes trouble.

Comment author: SoerenMind 16 March 2014 08:06:44PM 0 points [-]

True. In the deck they make a point about that it's okay to have really simple cards since those will take a very small share of your time. But some are just useless. I keep a finger on the delete-button while doing my flash cards.

Comment author: ChristianKl 17 March 2014 04:30:48PM 0 points [-]

The problem with the card is not that it's simple.

The problem is that even if I know that there method A, method B and method C for redundancy, my automatic answer to the question isn't various methods.

I might think of method A, sometimes I might think of method C. Because there's no clear answer the card doesn't lend itself to SRS. If I haven't seen the card in a year I won't think that the answer to it is "various methods" but I will think of more specific methods.

Comment author: ancientcampus 18 March 2014 04:22:42PM *  0 points [-]

For what it's worth: Though I do not claim to be a perfect user of SRS flashcards, I used them intensively for 3 years of medical school, constantly refining my technique. Many people here have suggested ways to improve my strategies. I have not yet seen an idea that I have not already tried extensively. Though I'm far from perfect, I think it's safe to say I have a better understanding than most beginners. There certainly is room for me to improve, but not much. If someone is considering using SRS long term for high volumes in medical school, here is my advice: it is possible a Perfect SRS User could use it more effectively than I did, but if you haven't already used SRS for years, you aren't such a person.

I never read that article, but I figured out many of those on my own. I agree with many of them, disagree with some. My input, for those that use it:

-Cloze deletion is simple, but to me, it is far too easy to "guess the teacher's password" using that technique, and is of limited use. It's great for high-school level fact regurgitation, but less useful for post-graduate stuff. You will quickly become good at the deck, but it does not strongly help your understanding of the material. That's an important point: your skill at answering questions in the deck does not necessarily translate to your skill at answering questions in real life.

-Graphic deletion - I used to do this all the time, but it is really time consuming to set up. I consider myself fast with an image-editor, but it's still a big drain. (Again, this is more of an issue in high volume) It also runs into the Cloze deletion problem.

-Use imagery: heck yes. I highly agree, in any situation (flashcards or no)

-Any technique splitting a larger whole into many smaller flashcards (the article lists several): This is possibly the WORST suggestion for high volume. While this is certainly very useful, again, when you use it in high volume I have found mental fatigue to become an issue. If you don't include the entire whole, you miss out on the big picture in a situation where the big picture truly is important. If you DO include the whole, you run into the cloze deletion "guessing the teacher's password" problem. That said, it has its uses in smaller volume, but I will never again use it in a high-volume deck.

To give an example: As the article suggests, I used to take a diagram, set up graphic deletion (make a series of images where a single element was blotted out), and run through the cards.

1) this takes a lot of startup time

2) Even ignoring the time to make the cards, I found reviewing the cards to be more time consuming than simply looking at the diagram, covering up the lables, and attempting to recall.

3) You get no practice recalling the diagram from memory

4) This technique is most effective if you will later see that exact diagram in real life/on the test, I argue it is a pitfall for guessing the teacher's password and provides less intuitive understanding of the diagram.

The strength in SRS comes from not wasting time on the easy parts and only spending time on the hard parts of the diagram. The theory is, after the first two cycles, you're only reviewing the "hard" parts of the diagram. On the other hand, you've spent more time making the cards, more time for the first and second card cycles, you're taking a big hit to the "big picture" style, and have no practice conjuring the diagram itself from memory. Ignoring the big picture and general understanding elements: if SRS provides any time benefit for the rote memorization vs going without SRS cards, i would only expect the benefit to "catch up" in 3 weeks BARE minimum; for me (for fatigue reasons in high volume) I pin the crossing point at 3 to 6 months, assuming it's an unintuitive diagram I use infrequently enough that I will forget it without review. I also argue that it provides a weaker general understanding of the diagram as a whole.

Comment author: Mark_Neznansky 14 May 2014 02:00:44PM 0 points [-]

Just to comment on the last bit: It seems odd to me that you stress the "3 weeks BARE minimum" and the "crossing point at 3 to 6 months" as a con, while you have used SRS for three years. Given that SRS is used for retention, and assuming that 6 months is the "crossing point", one would think that after three years of consistent SRS use you'd reap a very nice yield.

I know it's a metaphoric language, but it seems additionally ironic that the "BARE minimum" you stress equals to your frequency of exams, while you disfavor the cloze deletion's tendency to teach "guessing the teacher's password".

Is the advice perhaps against using SRS to learn/cram complex knowledge under a very limited time?

Comment author: Emile 11 March 2014 03:58:36PM 2 points [-]

From a Psychology textbook I read (and other sources, including here): "Elaborative Rehearsal" is a kind of reviewing that improves retention: instead of just rereading atomic "facts", it's more effective to look for meanings and connotations, to ask "why?", and to see how it fits in the bigger picture. Having a good understanding of principles and relationships makes transfer easier, i.e. it makes it more likely that you'll be able to use what you learnt in different contexts (i.e. in daily life and not just on LessWrong / when studying psychology).

Anki seems pretty bad at all of those, so something that puts more emphasis on the global structure of knowledge (like a Memory Palace / Method of Loci) may be better.

On the other hand, I mostly remember the above because I put it in Anki and review my decks every day :)

Comment author: ChristianKl 11 March 2014 04:22:57PM 1 point [-]

From a Psychology textbook I read (and other sources, including here): "Elaborative Rehearsal" is a kind of reviewing that improves retention: instead of just rereading atomic "facts"

I'm my experience psychology literature does not easily present atomic facts. Finding atomic facts in a article is a process that means that you have to understand what the article is about.

I took a book about learning theory and it told stories about how Aristotle did this and that and how some experiment turned out in a specific way. It names a bunch of facts but I didn't found anything that looked like an atomic fact in the first two chapters.

Comment author: Slackson 11 March 2014 04:35:01AM *  2 points [-]

Anki is good for trigger -> response sorts of memorization, but requires a bit of hacking for other things. Combining mnemonics with spaced repetition, I've heard, is ridiculously powerful. I've got a card with three sides, Trigger, Association, and Response, to try and strengthen the trigger -> response bond. I've set it up so I've got Trigger -> Response, Association -> Response, Trigger -> Association and Trigger -> Association and Response cards. If anyone wants me to share this format, I'm happy to do so.

ETA: Combining this with habit-training techniques is, I predict, potentially powerful.

Comment author: Emile 11 March 2014 09:02:22AM *  2 points [-]

habit-training techniques

What do you mean specifically here by "habit-training techniques"? What you describe above, or something else? (like, beeminder or ksotala's tips on habits etc.)

Comment author: Slackson 11 March 2014 09:46:11AM 6 points [-]

Stuff I learned at the Melbourne CFAR workshop. Class name was offline habit training, i.e. actually performing your habit multiple times in a row, in response to the trigger. Salient examples: Practicing getting out of bed in response to your alarm, practice walking in the door and putting your keys where they belong, practice putting your hands on your lap when about to bite nails, practice straightening your neck when you notice you're hunched. These are all examples I've implemented, and I have had good results.

Adding associations is a key part, too. For these examples, I imagine the alarm as an air raid siren and my house getting bombed if I don't get out of bed on time. I imagine Butch being shot by Vincent in an alternate version of Pulp Fiction if his father's watch wasn't on the little kangaroo and he had to hunt around for it. For biting my nails, I imagine Mia Wallace being stabbed in the heart . The connection here is biting nails can make you sick. The vividness and intensity makes up for how tenuous that is. For posture, I imagine Gandalf the Grey compared to Gandalf the White (plus triumphant LoTR music).

Since I made that comment, I got about a third of the way through Moonwalking With Einstein, and practiced the Memory Palace/method of loci a couple of times. I've lived in a bunch of different houses, so that works pretty well for me. Some of the stuff that was mentioned sounds a lot like spacing techniques. ""[...] if you revisit the journey through your memory palace later this evening, and again tomorrow afternoon, and perhaps again a week from now, this list will leave a truly lasting impression."

This is another bit of evidence suggesting that spaced repetition would be powerful in combination with mnemonics. What Anki provides, which is far more important than the flashcard thing, is testing. I've been thinking about applying some of the ideas from test-driven development to self-programming, and Anki cards would be a core part of that.

Sorry, I realize most of that isn't relevant, but I hope the parts that were are useful.

Comment author: Emile 11 March 2014 10:51:02AM 3 points [-]

That's exactly the kind of info I was looking for, thanks :)

Comment author: Mark_Neznansky 14 May 2014 06:12:35PM 0 points [-]

Funny. I've used triumphant LoTR music once to overcome my terrible fear of heights. I was climbing mount Kathadin with friends (including passing along "Knife Edge "), and the humming/singing out loud this music (+imagining a chopper-camera shooting from above) has completely effaced my fear. Possibly being called "Legolas" during middle-school and high-school helped, too.

Comment author: WingedViper 11 March 2014 10:18:20PM 1 point [-]

As spaced repetition and flashcards are a technique and tool respectively it is (to me) obvious that they are useful for certain kinds of circumstances. Flashcards really are useful only when you want to associate 2 things to each other (for example a word and its translation) and might not be the best way to build an organized knowledge of a subject. Because of that I wouldn't use them for that purpose in any case.

Thank you for pointing out an area where they fail, that was useful information.

A question to the community: Do you really believe as much in spaced repetition/Anki as the post suggests?

Comment author: hamnox 13 March 2014 03:49:54AM 1 point [-]

I believe it's really effective when used effectively.

I also think it's really damned hard to use effectively. There's a lot of cards I have queued to fix or delete because I didn't formulate them well enough.

Comment author: ancientcampus 18 March 2014 03:25:57PM *  0 points [-]

A question to the community: Do you really believe as much in spaced repetition/Anki as the post suggests?

Excellent question; I'd like to know too.

Comment author: ChristianKl 13 March 2014 09:48:18PM 0 points [-]

Flashcards really are useful only when you want to associate 2 things to each other (for example a word and its translation) and might not be the best way to build an organized knowledge of a subject.

That like saying that computer are only useful when you want to add 0's and 1's. There's Turing completion for computers. In a similar way you can break most knowledge down to atomic units. Often that means that you have to think about the knowledge in a more detailed way.

A question to the community: Do you really believe as much in spaced repetition/Anki as the post suggests?

If I remember right something like 12% use it. On the other hand more than 12% did use it but don't anymore. On the other hand those people who speak about Anki, are the people who believe in it.

However while I do believe in it I think there's a huge room for improvement through things like automatic creation of Ven diagrams based cards.

Comment author: gjm 12 March 2014 01:00:20PM -1 points [-]

Dunno.

Love,

The Community.