Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

A Second Year of Spaced Repetition Software in the Classroom

29 Post author: tanagrabeast 01 May 2016 10:14PM

This is a follow-up to last year's report. Here, I will talk about my successes and failures using Spaced Repetition Software (SRS) in the classroom for a second year. The year's not over yet, but I have reasons for reporting early that should become clear in a subsequent post. A third post will then follow, and together these will constitute a small sequence exploring classroom SRS and the adjacent ideas that bubble up when I think deeply about teaching.

Summary

I experienced net negative progress this year in my efforts to improve classroom instruction via spaced repetition software. While this is mostly attributable to shifts in my personal priorities, I have also identified a number of additional failure modes for classroom SRS, as well as additional shortcomings of Anki for this use case. My experiences also showcase some fundamental challenges to teaching-in-general that SRS depressingly spotlights without being any less susceptible to. Regardless, I am more bullish than ever about the potential for classroom SRS, and will lay out a detailed vision for what it can be in the next post.

Prologue

You might want to at least skim last year's report, even if you've read it before. My job description is unchanged, and my core approach to SRS stayed the same. I will not be going into those details today.

Also, you'll probably find much of this report disappointing for reasons that have nothing to do with SRS itself and everything to do with me—reasons I will now invite you to understand with a philosophical introspective. If gazing into my navel really isn't your thing, just skip the next section.

Autopilot years

The commonly appreciated upside of being a public schoolteacher in the United States is having more days off and greater job security than most workers. Less commonly appreciated is that, once you've got things figured out, you probably don't need to put in tons of extra hours. You can mostly leave your work at work.

So what do you do with the time?

Good teachers usually direct a lot of that back to the students. They coach a sport, or sponsor a club. They leave detailed feedback on essays. They tutor.

I've done most of those things, but my personal preference is to reinvest in my core classroom offerings by improving my content and tooling. On top of some smaller tweaks, I usually pick one big project every year that will add a new layer of awesome to what I do. One year, it was better writing mini-lessons and example content to go with them. Another, it was visual vocab slides with real-world example sentences. Twice, it was improved systems for grading and classroom management, facilitated by my hobbyist programming skills.

Not all such projects are successful. My first management system, for example, was an unworkable mess, necessitating the System 2.0 I brought on line the year after and have used ever since. But lest I drift uselessly on waves self-doubt and shifting interests, I pre-commit to seeing a project through for the year, aborting only if I have strongly compelling evidence that I'm wasting my time.1

Last year, my pet project was the SRS experiment. The daily commitment to card creation, on top of everything else I do, was substantial. For this year, it would have been natural for my project to be a similar commitment to improving those cards and the the lessons that go with them.

It didn't happen.

The commonly-appreciated downsides of being a public schoolteacher in the US are low income and emotional weathering. The intensities fluctuate, but the effects are cumulative.

Which brings us to those years where a teacher pours little or nothing back into their job, pressing “replay” on whatever they did last year and staying relatively detached. I've never heard a teacher use the term “autopilot year”, but every teacher who's been around long enough knows exactly what I'm talking about and has probably had at least two.

I discourage you from judging them for it. A good teacher on autopilot is better for your kid than the flailing rookie who takes their place when they switch careers. Sure, there are bad teachers on autopilot, but you should prefer they stay that way; the only thing worse than a bad teacher on autopilot is a bad teacher who is not detached, and actively imposes on the self-motivated students who might have otherwise learned something on their own.2

Life happens, and even this job is still just a job. Maybe your kid's teacher seems a bit checked out because they're taking care of an ailing family member. Maybe they're running a home business to fill the income gap left when you voted down the tax override. Maybe they're still reeling from the emotional damage of the student who viciously jabbed at their worst insecurities for most of last year.

Maybe they're like me, not doing any of those things at the moment but certainly feeling pinched by expenses that climb faster than his salary; feeling like he doesn't quite need to switch careers but needs to feel like he has the option; feeling like it's time to reinvent his approach to teaching but unable to execute on his ideas; feeling like he should pour every spare minute into aggressively levelling up his programming skills, and then doing so, on the theory that “If you don't know what you need, take power.”

So no, I haven't done much of what I set out to do in my SRS goals list from last year. I didn't rework very many of my cards. I didn't take a less linear and more opportunistic approach to introducing new content. I didn't even reduce the number of cards!

Because, in the end, these tweaks began to feel far too small, like cobblestones on a path terminating in a local maxima where I'm doing “pretty cool” things in my classroom that no one else will replicate. I've been down that road before, and I'm not in the mood.3

It's time to go big or go elsewhere, and upgrading myself has felt like the only way to make that happen. Course improvements would have to wait.

Standardized test results

State test results for last year's students came back eventually, and were disappointingly inconclusive. Being a brand new test, the reports were sparser than I was used to. There was no “value-added” analysis to show me whether expectations were beat for my particular students (an area I have historically done well in), only a broad comparison to the students of other teachers in my department. My classes' average was only marginally higher than that of our most similar peer classes—a difference I would put handily within the margin of error, as we often seen very uneven distributions of talent between classes. The most noticeable signal was of my classes producing more outlying high scores than would otherwise be expected, but that's a small n.

I'm not going to make up stories explaining why the scores might be disappointing because I could just as easily tell stories that would have explained a stronger signal of success.4

Still, I think it says something that my classes continue to perform with increasingly minimal note-taking and homework. Certainly, the student experience is improved without harming test outcomes. I'm the English teacher students beg to have again (and occasionally manipulate the system into getting). A teacher can be effective without being liked, but positivity has real advantages. Certainly, I enjoy teaching more when my students are having a good time. Creativity and connection-making are also known to be enhanced by a positive mood (at the expense of vigilance and accuracy).5

I can't also help being cynical here, though. It may be that what we do in class simply doesn't matter on the test, because the state is evaluating softer skills that derive more from raw intelligence and pleasure-reading habits than from anything I can teach them.

Just before classes started up for the new school year, I was given an impromptu opportunity to present SRS and what I'd done with it to most of the other teachers in my department. They seemed interested and supportive, asking some good questions, including about implementation details for starting up classroom decks of their own.

Not one of them tried it.

I wasn't surprised, and I certainly don't fault them for it. I'll get more into why in my next post, but over the weeks that followed the main impression I got was that they saw classroom SRS as being at the teacherly equivalent of a low technology readiness level; they were glad someone like me was experimenting with it, and hoped it would become a more obvious win down the road.

I did come close to getting a couple of teachers to at least try SRS for their personal use when I talked about how I was using it to learn my students' names before I had met them—as they had seen me doing between meetings.

Names and faces

I'm not someone who organically picks up people's names very well; without some kind of a system in place I can go embarrassingly far into the year relying on my seating chart. Pre-learning names with Anki is something I had seen talked about on this forum before, so I tried it.

I had five calendar days between the time I could access the rosters for my classes and the first day of those classes. Our district's roster management system has images on file for students that have been in the district for at least a year or so.

After subtracting the photoless newcomers and the students I had last year, I had about 120 names I could learn. It was straightforward (if grindy) to use the standard Windows image capture “snipping” tool to paste these images into Anki cards.

I was striving for a high level of automaticity, as well as both first and last name recognition. So I did multiple daily sessions. In total, I spent about 3.5 hours, spread out over that 5 day period. This doesn't include the time I spent making the decks (40 min maybe?) or the much shorter review sessions I continued to do after school started.

For students who resembled their cards, I could (and occasionally did) greet them by name as they walked in my door that first day, intentionally adding uncertainty to my voice to cut down on the creepiness quotient.

There are some obvious inefficiencies to pre-learning. Not everyone looks like their file photo, or goes by a name that resembles their legal name. And rosters tend to fluctuate that first week.

Will I do it again? I might. It helped us hit the ground running. Historically, I would pad the first couple days with some independent work just to give me a chance to stare at them, which is awkward for everyone. Name learning also fits with that impending start-of-year mood where I often feel less prepared than I am, and seek out compensating tasks. But 4+ hours is not an insignificant investment, and there were still a lot of names I couldn't know for lack of photos.

I think my preference would be to wait for the first day of classes and use an app that has you take a picture of, say, five students at a time holding up name cards, automagically turning this into SRS cards.

Someone go make that.

The sapping seduction of legacy cards

Having an autopilot year was disturbingly easy as far as our class Anki decks went. Last year I had made sure to tag everything, and to organize the archives a bit before summer. Pulling out the right cards for a day's instruction this year was thus the work of a few clicks.

But a wealth of existing cards makes new card creation feel onerous by comparison. Worse, you're reluctant to modify a lesson if it might “break” some existing cards. This could definitely have pulled me towards stagnation even if I hadn't made a conscious decision to coast this year.

There's also a related problem here, which is that it's harder to stay excited about cards you've seen dozens of times. Cracks began to show in the affable MC personality I wear for review sessions. That “apathetic third” of students I talked about last year? It's more like the apathetic supermajority when I haven't brought my 'A' game. The lesson here is a general one, I'm sure: charisma is a poor foundation on which to build a lasting system.

Missing out: creation

The aversion I've felt to making new cards for my students this year is actually kind of funny in light of the fact that I've made some 3,000 cards for my personal use over the same period, and that these, being more technical (programming), were mostly harder to write. I think some of that hesitancy stems from a growing apprehension that I'm not doing students any great favor by writing all of their cards for them.

There is much to be said for figuring out how to take an idea that is new to you and put it into concise form. As I continue to think of ways to reduce the note-taking I make my students do, I'm also cognizant of the fact that I rarely take notes for myself anymore—I just go straight to making Anki cards.

But I've also gained a deeper appreciation for the difficulty of creating a truly good card, so much so that I'm hesitant to trust teenagers to do it even remotely well.

So the question I keep asking myself is how to give students most of the benefits of participating in the card creation process without sacrificing the time-efficiency and card quality that come from a professional writing the cards. It's thorny. But having turned it over in my head enough times now, I think I have some answers (which will have to wait until the next post.)

Missing out: free association

Another benefit they're missing out in is the free association time I get on the exercise walks where I do much of my personal study. Minute-for-minute, I only get about half as many cards done this way, but as I pause reviews to run for a stretch, negotiate challenging terrain, or appreciate my surroundings, I'm also mulling over the cards I've last seen and letting connections form between them—and between anything else I've been thinking about.

This is a strange habit that works largely unconsciously, and one I'm not sure I would have thought to intentionally cultivate but for an older Kahneman quote in my personal deck:

An idea that has been activated does not merely evoke one other idea. It evokes many ideas, which in turn activate others. Furthermore, only a few of the activated ideas will register in consciousness. Most of the work of associative thinking is silent, hidden from our conscious selves.

This year, I've grown to appreciate that the subconscious is a black-box back-office you can deliberately prime with study cards; it will reliably return assorted insights when given enough raw materials and workspace.

I actually added a card I couldn't answer the other day, on a hunch that I might figure it out over the course of a walk once it had popped up a few times.

Success.

How to give students this benefit? That's a tough one. Worth thinking about, though. Worth thinking about… [lowers phone, takes in sunset].

Tiers of availability

It's also clearer than ever to me that being able to remember something when prompted is not the only worthy end-goal for a card.

Sure, for some types of information, it is indeed enough that, given a very specific prompt, you can return a corresponding fact. World capitals, for example. Vocab definitions (while reading). Special-purpose algorithms. Usernames and passwords for important-but-infrequently-used services.6

But for most of the information actually worth committing to memory instead of Google, we want it to spontaneously fly out of us without any specific prompting whenever we're in a context where it might be useful. Wise quotations. Multi-purpose algorithms. Vocab words (while writing).

For some information, the most valuable thing it can be doing is bouncing around in your near-subconscious, making itself a target for collision and fusion with other ideas. If you're looking for an analogy to nuclear physics (and why wouldn't you be?) think of this as increasing the neutron cross section of an idea.

For still other information, you want it to retrieve itself instantly as a matter of reflex before you even become consciously aware of it. This is true where speed of recall is critical, and in situations where conscious recognition is unhelpful or a point of failure. Think grammar error recognition. Word fragment meanings. Muscle memory tasks. Implementation intentions. (aka Trigger Action Plans).

The thing is, different levels of availability require different rehearsal commitments. I've not seen any explicit support for varied automaticity goals in Anki or the other spaced repetition programs I've played with. The best I can do is try to decide on a review-by-review basis whether I should set the next interval of a given card more conservatively than suggested.

I find that cards needing rapid reflexive recall are best kept very short. This lets you review them aggressively without a huge time commitment, and also makes them more likely to bubble out of your head in response to relevant stimuli.

My existing classroom decks predate these insights, and could definitely use some optimization.

Crickets and chihuahuas

I've seen that the particular mix of students you get in a given class makes an enormous difference in student buy-in.

I have one morning class where eyelids hang low, indifference runs high, and we can go several Anki cards in a row without anybody raising a hand. Crickets herald a death spiral where we can't review as many cards in a given period of time, and where don't spend as much time reviewing because the energy falls off more rapidly.

I have another class for whom Anki is the highlight of their day, and they beg to spend a greater portion of class doing it. I can go through 15 cards without needing to call on the same volunteer twice. These students are my little Anki chihuahuas, and to see what would happen I decided early on to let that class stay caught up with with the deck more often, even if it meant cutting into other planned activities.

A natural experiment, you ask? Nah. They self-selected, and are my only section of that grade level. Has it done them extra good? I'm not so sure. But I've learned a few things.

Here comes a new failure mode!

Newly identified, at least. I've had a hunch, but now I know for sure that many students see Anki only as a way to get validation for ideas they remember, rather than as a way to re-learn information they had forgotten. This is most visible among the chihuahuas, where a number of students have a reputation for “owning” particular cards, answering them with cultivated panache and expressing indignation if someone else “steals” them.

This is entertaining to preside over, but isn't exactly the point. I'm glad Anki locks in the cards they know, but it seems that if they didn't know the card the first time it came up, they'll probably never know it.

I could draw a couple of lessons from this. 1) That the “vivid memory, card ready” mantra I devised last year is even more important than I thought. 2) Nothing short of pressing the red button will do for a student who has forgotten a fact.

Horizontal integration

From that same class of enthusiasts I've seen another pattern emerge. Students strong in cards with a common theme tend to take more interest in other similar cards. They become Grammar Dude. Little Miss Word Fragment. Chief Petty Officer of Words-That-Sound-Vaguely-Like-Genitals.

Alas, this does not seem to help their interest in other categories, and actually seems to work against it. Turns out many of my chihuahuas are actually hedgehogs.

Absolute zero

A few of my students have an impressive, empirically verified zero absorption rate. I am not exaggerating. I can compare pre-test performance to post-test performance and find absolutely no improvement, even with easy objective content that we covered extensively.

SRS has nothing in particular to do with this, but it does shine a soul-crushing spotlight on it. A student can show up every day, take every note you give them, look like they're paying attention over weeks of Anki review… and test exactly as well as if they'd spent every minute in the bathroom doing their hair instead.

Stuck in neutral

Stand back as I invoke the name of the adversary, She-Who-Will-Not-Be-Inspired, the Angel of Trite, the Rock of our Stagnation, the Blahgiver, the Alpha and the Neutrogena. Her name is Apathy, and she doesn't give a shit.

You know how you can zone out and review something three or four times without processing it, but then you realize, kick yourself in gear, and engage your faculties, pushing it into your mind?

You have to care, just a little bit. To keep it up, you have to care a little bit more. It's not always easy to find and sustain that spark of caring, because the learning target might be only indirectly instrumental to your goals. But you can do it.

But what if the vending machine stopped carrying your Juice of Sapho? What if you lost your Care Bear Stare?

A large fraction of teenagers lack the ability to direct caring at will. They can be made to understand why they should care about something in school, but there are too many layers of indirection between it and the impulses that can actually move them. You can chalk this deficiency up to short time preference and trouble delaying gratification, if you want. You can even blame the more obvious fact that, at any given moment, a teenager is probably sleep-deprived, hungry, emotionally preoccupied, or some combination of the three. But whatever their reasons, they Just. Don't. Care.

Some, I fear, are so in thrall to Our Lady of the Vacuous Conception that they don't even know where to find the part of their brain that tries. Maybe they never did. Is it possible? Could some people be born without the ability to mentally engage? I almost feel silly asking, like I'm ranting about a pandemic of philosophical zombies. But I've seen things, all right?

Where does classroom SRS fit in a dystopian world where the High and Flighty One walks abroad, claiming our youth as her disciples of indifference?

It doesn't fit well when the cards will go on without them, as happens when we do Anki together as a class. This doesn't make SRS any worse than all of the other things they won't do (like the things it replaced), but it doesn't make it better, either. I had very good reasons for starting out with a whole-class model rather than a 1:1 model, and those reasons haven't disappeared. But I don't think I can expect SRS to significantly improve the outcomes of apathetic learners without 1:1 learner-app interaction. And I don't want to keep writing those kids off.

Because brains in neutral can still learn. They just can't do it on purpose.

I credit my relative success with lower-tier teens over the years to my pragmatism about this. I try to keep things breezy, sweeping the students along in a gentle flow that washes them through the Zone of Proximal Development and out the other side with a few bits of knowledge surreptitiously clinging to them.

I said last year that easy cards are important. I also talked about the value of a vivid narrative or experience as a hook for new content. Let me go a step farther now and assert that a good card for an apathetic learner will feel unneeded on creation, because it's only a baby step away from what they already knew and because you pounded it home by way of a narrative so gripping that even the laziest brain couldn't help following it to its conclusion. A card for this will smell like overkill. Hold your nose and make it anyway.

Everyone will get such a card right on the very first try, but that doesn't make it unnecessary. A vivid implanted memory will replay and refresh itself readily, but only if something prompts it. So it still needs a rehearsal schedule.

Neglected, it is the fate of all things to be claimed by the Whore of Babble-On and cast into Outer Wherever with a flash of her indolent smize.

Abort, retry, fail?

One of my goals from last year was to be more conservative when setting the next reviews of cards. This worked well enough at the start of the year when the decks were relatively small. By second semester, however, we had outrun our tail again, just like last year. Considering that I didn't reduce the number of cards or increase the length of our study sessions, this was inevitable. For every card I kept in higher circulation there was another card languishing unstudied. If you don't keep up with your reviews, the effective size of your deck will always be smaller than its actual size.

Where I never thinned my deck, the thermodynamics of Anki did it for me. Cool? To the degree that I chose higher-priority cards wisely, this worked out fine. But it added an additional wrinkle to cards on the margins.

See, due to the way Anki works, when you review a card that is long overdue, if you don't press the red button to re-learn it completely, your only other feedback options set the next review far in the future. I understand the logic: If you didn't know the answer, then why wouldn't you reset it? And if you did know the answer, then it clearly didn't matter that the interval was so long, so why not set the next one even longer?

However, these are not great options when sharing a single instance of a deck with a roomful of students. These are also poor options if you're trying to sustain a higher level of automaticity with that card (see the Tiers of Availability section above). Should I press the red button and make it one of the Chosen again—dooming some other card to oblivion if our reviews don't get any longer—or should I press the next button over and kick this card so far back out that it might as well be dead?

Even in the case of my personal learning, I think there is room in an SRS for buttons that reduce the next interval on a card without resetting it all the way back to zero.

Red button repulsion

I had made it an explicit goal to reset a card when it was clear we had lost our grip on it. But even when I wasn't worried about time constraints, I still found it very difficult to press that red button. I think this was mostly self-consciousness. Pressing that button means putting the card back in an initial learning mode where, unless you cut the session short, it will come up multiple times again that day. This is by design, of course. But at the front of a classroom, it feels wrong. I blame two factors:

1) “Vivid memory, card ready” means we get most new cards right on the first try, so a repeated card feels broken by comparison.

2) I feel like I'm playing the part of an 18th century schoolmaster leading a class in chants at ruler-point. “Drill and kill” is a catchy phrase today's teachers hear in their heads whenever they use rote learning, because we've been heavily conditioned via our training and Hollywood that this is what Bad Teaching looks like.

This might be one of the greatest overcorrections in the history of education. I'm not saying we should use unadorned repetition as our tool of first resort, but learning and remembering do not happen without some sort of reiteration going on, and we've made it taboo to use the most direct approach to it. We're like soldiers in a gunfight who have decided we should only kill via ricochets.

It looks deathly boring to an observer, but, in moderation, most students actually enjoy traditional rote learning. They enjoy the confidence that they will for sure get the information into their head before moving on. They enjoy the validation they get with each chance to confirm that they remember something. They enjoy going with the flow of a whole class doing the same thing.7 They enjoy the respite of learning on rails for a change, without any expectation that they take initiative or parse instructions.

So, when I do work up the courage to press that red button, it's not uncommon for me to see students perk up and show additional interest even as I'm feeling embarrassed and fighting off an icky sensation in my gut.

Fight it I do, though. For one content type—Greek/Latin word fragments—I've actually switched to using Anki as our first exposure learning tool. It's a natural fit, and far more time efficient than the softer introductions I've used in the past. Adding 6-10 new word fragments a day builds a solid repertoire in just a couple weeks, and each day's learning builds to a rewarding climax as they go from timid to confident with the newest cards.

I propose a new catch-phrase: Drill and instill.

So say we all

The stats say I went faster with the cards this year. This was deliberate. Our total number of reviews did not rise, however, as our study sessions tended to be a bit shorter. This was not deliberate, but more a reflection of engagement tending to peter out sooner. It remains my policy to stop reviews before they feel too much like drudgery.

Description here.

[Two years of data for one class period, accidentally merged together (I hadn't realized Anki would remember the history of deleted decks).]

Most of the time savings came from stopping the use of our colored feedback cards. By December, nearly every student had stopped holding up any color other than green, perhaps tiring of the small inconvenience of considering their personal relationship with a card and rotating it to the appropriate color. Once I felt like the quality of information I was getting from them was too low to justify those few seconds I spent soliciting the feedback on each card, I stopped asking for it.

This may have been a mistake.

I felt like engagement levels dropped off a bit after the switch and never recovered. I don't think it had to do with feeling de-voiced, since hardly anyone was still giving actual feedback via the cards. But I think the cards might have served as a participation priming device, keeping students in the Anki mindset by giving them a way to say “Amen”.

The calendar is a harsh(er) mistress

Last year, I bemoaned the difficulty of keeping up with an SRS regimen when not every day is a school day. Well, our calendar changed this year. We now take a shorter summer in exchange for longer breaks in the fall and spring. Pause and predict the consequences! You will very likely be correct.

Ready?

We were able to recover from two weeks off in the fall readily enough, but just as we were about to catch up from the winter holidays were were beset upon by an equally lengthy spring break. Right on its heels was a short holiday week that was itself hounded by three full weeks of a special block-period testing schedule that saw each class meeting only three times a week.

And so it was we went into the state test with our Anki freshness at an all-time low.

I'm still pretty down on the odds of getting anyone to study an SRS on weekends and days off, but the state of our calendar insists that I revisit the notion.

Nature, nurture

A modest number of students from last year had me again this year. It was my goal to keep an eye on them. Would they be supercharged? Would they regress to the mean?

A bit of both. Some of my mediocre repeats seemed much more confident, but I'm used to seeing that with second-years even without SRS. Anki gave them a way to demonstrate their retention, though, as some of my boilerplate cards are the same for both of my grade levels. They enjoyed having a strong grip on these from the get-go, and occasionally surprised me by recalling something obscure that was unique to last year's deck.

Low performers were still low performers.

But among my high-achieving returners there were some surprising contrasts. Most of them started strong and have generally just coasted along on a plateau of awesome. Two of my all-stars, however, started the year weirdly inept, bombing pretests as though last year had never happened.

These cases can be terribly disheartening. If I didn't already believe the research about the mostly fixed-at-birth nature of intelligence, these seemingly leaky brains would certainly have pushed me in that direction. But this is actually a strong argument for SRS. Regular participation in class Anki seems to have steadily pumped at least some students with lower-than-I-had-appreciated innate ability to a much higher functioning level of performance.

The two students in question eventually resumed outperforming most of their peers, but it took a while. And here we see a strong argument for the narrower idea of trying to sell students with any motivation on the life-changing potential of an independent, year-round SRS habit.

A life changed

It can happen. My wife had spent some time using Anki with her high school Spanish students last year, in tandem with my own experiments. (She has since switched to Duolingo for very good reasons, but with all of the difficulties inherent to getting students to use apps independently).

She describes one of her students from last year as a slow-but-motivated learner who really struggled in all of her classes. But she found Anki powerful enough that she started making cards on her own for her other classes. It was the lifeline she hadn't known she needed. Her confidence and performance climbed steadily, and she is now said to be in the running for valedictorian.

The moral of the story is that the low hanging fruit of SRS is awareness, because there is a tiny fraction of the student population who will latch onto it and reap epic benefits—students who have this enormous pent-up charge potential and are grasping blindly for a conduit. So if you do nothing else, at least get the word out.

But that's just one student—not even mine—out of hundreds now. And she might have found some other way to succeed. So how big of a deal is this?

That's hard to say, mostly because we rarely find out what students do with their lives after they've had us. For all I know, there could be a bunch of sleeper agents among my alumni—students who will remember SRS when they reach a point in their life where they finally have the level of focus and motivation needed to make use of it. This doesn't seem too far-fetched, judging from the often dramatic change in students I see from one year to the next if they have me again, and from memories of my own low motivation at that age.

Conclusion

While I've continued to learn from my experiences with classroom SRS, I think we actually got less value from it than we did last year. I'm not content with this. While I could simply blame this on the way I've prioritized my personal growth over course improvement, this would be missing the point, because a classroom SRS system that only a driven tech-savvy veteran in a good mood can make work is not nearly good enough.

I persist because I know spaced repetition, at its fiery molten core, works, and I want to find an approach that will work for other teachers, other classrooms, other students. I don't think that approach will look very much like what I'm doing now.

The good news is that months of hammock-driven development and tech skill-ups have not been wasted. I don't just have goal to make classroom SRS work now.

I have a vision.

[To be continued...]


Notes

1. John Cleese, of Monty Python fame, has some interesting advice about what he calls "open mode" and "closed mode", and how to use them, though the actual video address linked in this article is no longer posted.

2. Is a student able to read for fun during your class, dear teacher? If not, you'd better be have some damned good content. Pleasure reading is strongly correlated with improved learning and life outcomes—much more strongly than you are, I'll wager. 

3.That “2nd system” I built for collecting, computing, and communicating a huge range of classroom metrics has become my periphery brain. It’s the thing teachers and administrators actually come to me about. It’s also crafted entirely out of noodly Excel VBA by someone who was thinking only of himself; someone who hard-coded almost everything at the function level—whose idea of input validation was “I’ll just never send it that”; and whose idea of accessibility was undocumented hotkeys that favor Dvorak on the left hand. In other words, it's utterly unsharable. I could write a friendlier version, but it would be a pretty epic project that hardly anyone would actually use, simply because of different teaching styles. If I'm going to build a system to share, it's going to be System 3... the future system now haunting my dreams.

4. I actually told myself both sets of stories ahead of time to keep me honest when results day came around. "If you are equally good at explaining any outcome, you have zero knowledge."

5. Daniel Kahneman, in Thinking Fast and Slow, summarizing conclusions from a study about the effect of mood on the ability to make connections between ideas: "When in a good mood, people become more intuitive and more creative but also less vigilant and more prone to logical errors."

6. I don't actually write login credentials on my cards. I leave the answers blank and just rate them so conservatively with every study that I never have a chance to forget them.

7. I once had the privilege of observing part of a lesson in a traditional Mennonite one-room schoolhouse. I don't speak a word of Low German, but it was clear the kids knew whatever it was they were drilling as they stood up and recited together. Most striking was the fact that they were all on the same page. There were no stragglers spacing out, slumped over, dozing off. The teacher could confidently build up to whatever came next without fear of leaving anyone behind.

Comments (29)

Comment author: toomanymetas 31 May 2016 11:32:40AM 2 points [-]

I have been using anki to install something like Trigger Action Plans for more than half a year and it's been working great. Wrote a blog post about it: http://guzey.com/blog/thought-patterns-marginal

tldr: create a deck with max interval of 1 day.

Comment author: NancyLebovitz 04 May 2016 08:13:20AM 2 points [-]

Have you considered sharing some version of this essay with your students? I think one of the bad things about conventional schooling (generalizing from myself) is getting the impression that the whole thing just happens, rather than that there's adult thought going into how teaching is done.

In re reading for pleasure: even if it isn't something you're teaching, at least you aren't spoiling it for your students.

I'm looking forward to your next installment.

Comment author: tanagrabeast 05 May 2016 02:01:58AM 2 points [-]

Have you considered sharing some version of this essay with your students?

This question makes me squirm a bit, which makes me think it might be important.

I do discuss the rationale behind my course design choices with students, in some limited domains. I should have mentioned in this report that I've tweaked my intro-to-SRS presentation I gave at the start of last year; I now bill it as a kind of superpower, and we have some cards in our deck about the principles of it -- cards that still get some play even this late in the year. I hope this may create more of those "sleeper agents" I speculated about, who may bloom into power-learners down the road.

I also make sure my students understand how valuable I think pleasure reading is, with a different presentation that spruces up the more interesting findings from that report I linked to. And I put my money where my mouth is by making sure they understand how very unlikely I am to give them a hard time for reading during my class, even if it's not exactly what they're supposed to be doing.

I even try to let them know why each unit is in our curriculum, whether it's "because the boss/district/state says so, but we'll try to make it fun" or "because I want to help you get into college and I know this will help".

But a lot of my thinking I don't share. I understand some of my reticence: there are things I do that wouldn't work as well if they knew I was doing them, and there are other things that would be exploitable if I laid out the strategies behind them. I'm struggling to articulate the rest of my hesitation, though.

Like the stuff about apathy and caring. I had some experimental lessons dealing with this sort of thing about 7 years ago when I taught a course with a broader curriculum mandate. I don't feel like these lessons got a lot of traction, though, in the same way that other "life skills" lessons tend to fall flat with typical teens. This age group is so slippery... so reluctant to accept advice where others would see it, so wary of anything that smells of paternalism.

My instincts now tell me to approach these things obliquely, as though I'm accidentally letting out the secrets I know they're too immature to make use of. I'm not telling them what they should do. I'm talking about how the rash actions of young characters in our stories make sense because said characters don't understand how adolescent brains are wired for overconfidence and short tempers. I'm making a seemingly off-hand comment about the rare superpower of "taking advice". I'm giving an off-script response to a question about my past with an answer about that time I totally kicked butt by putting in extra hours of effort, as though this were a cheat giving me a secret edge.

I remember being a teen and thinking much more deeply about the things adults seemed to let slip than about their prepared remarks. It hadn't occurred to me until now that some of those slips might have been carefully scripted.

Comment author: Lumifer 05 May 2016 02:37:47AM 0 points [-]

How old are your students?

Comment author: tanagrabeast 05 May 2016 02:41:31AM 0 points [-]

14-16, usually. These are 9th and 10th graders, with a few repeating upperclassmen.

Comment author: Elo 02 May 2016 02:55:28AM *  2 points [-]

This is an excellent review. Thank you; please keep writing!

A few aspects of behaviours you have described seem to be the behaviour of students trying to guess the teachers passwords, and shortcut the learning process.

Things like:

  • chihuahuas, where a number of students have a reputation for “owning” particular cards
  • many students see Anki only as a way to get validation for ideas they remember
  • Students strong in cards with a common theme tend to take more interest in other similar cards. They become Grammar Dude. Little Miss Word Fragment. Chief Petty Officer of Words-That-Sound-Vaguely-Like-Genitals.

Does this seem like an accurate insight into their behaviour to you?

Comment author: tanagrabeast 02 May 2016 03:44:48AM 2 points [-]

Good question, but no, I wouldn't say these students are trying to guess the password. The cards they're remembering aren't complex enough to qualify. The answer is the answer, not a surface representation of some deeper knowledge they're skipping.

This feels more like a case of selective attention, of perking up and caring more about cards they see as "in their wheelhouse". It's an easy way of being better than everyone else at something, even if that something is pretty narrow. If you've ever done any cooperative social gaming, you can probably recall analogous situations where new players spontaneously start seeing themselves as specialists with some power-up, weapon, player class, etc. It's a land grab for the ego, and mostly just harmless fun.

Remembering passwords takes effort; a mystical incantation is harder to memorize than an answer you can logically derive from deeper knowledge. Hence, password guessing is something I mostly see in students who are grade obsessed, and I don't get too many of those.

Comment author: Elo 02 May 2016 07:00:01AM 0 points [-]

What I am suggesting is that in "knowing one card", they have thoroughly decided to commit to knowing the password in this one field. They are essentially checking out from doing anything other than reciting the surface concept back at you. potentially finding a way to cheat their way to the applause lights.

I wonder if it would be possible to trick them into changing the goal from "correctly recite this password" to, "get every password right".

The exact link escapes me but someone suggested a concept that is useful to think about is, "The Desire To Pass Tests", which if your student has, can be all it takes to succeed. What do you think of TDTPT?

Comment author: tanagrabeast 02 May 2016 11:47:14PM *  2 points [-]

I think all of these strategies give the type of student I'm talking about too much credit, as they are mostly emotional creatures not prone to strategic planning. I guess TDTPT comes closest, but I would change it to a phrase I use with my students: "It's fun to be right." IFTBR.

Easy trivia apps were all the rage among my students a couple years ago. Nobody was trying to get a high score or trying to advance to the next level, but if you put a question in front of someone that they think they know the answer to, the urge to get validation for knowing it is irresistible. You've probably seen ads on the internet that work on this principle.

It's why Who Wants to Be a Millionaire always started off with insultingly easy questions, and why easy cards in the class Anki deck are so important for raising participation and morale.

Comment author: ChristianKl 02 May 2016 03:43:36PM 1 point [-]

The thing is, different levels of availability require different rehearsal commitments. I've not seen any explicit support for varied automaticity goals in Anki or the other spaced repetition programs I've played with. The best I can do is try to decide on a review-by-review basis whether I should set the next interval of a given card more conservatively than suggested.

For normal Anki learning I think the solution is having redundancy in cards. I'm at the moment learning anatomy and I have lots of cards for every muscle and bone. I have graphics from different angles. I have cards asking for holonyms.

Comment author: moridinamael 02 May 2016 04:53:25PM *  3 points [-]

Along these lines, I have embraced the power of Cloze deletion. I have no problem with keeping all of the following cards in rotation:

The [...] is a cognitive bias in which relatively unskilled persons suffer illusory superiority, mistakenly assessing their ability to be much higher.

The Dunning–Kruger effect is a [...] in which relatively unskilled persons suffer illusory superiority, mistakenly assessing their ability to be much higher.

The Dunning–Kruger effect is a cognitive bias in which [...] suffer illusory superiority, mistakenly assessing their ability to be much higher.

The Dunning–Kruger effect is a cognitive bias in which relatively unskilled persons [...].

Even if I don't actually care about memorizing the wording verbatim, breaking the information up this way forces me to learn the information in a sort of "anisotropic" fashion.

edit: Also, yes, at least two of these cards would be dead-easy, practically already known before I saw them even once, but seeing the information "too much" at the start can help push you over the initial hump.

Comment author: tanagrabeast 03 May 2016 12:31:22AM 0 points [-]

It should be noted that how the cloze cards play out changes greatly depending on whether you allow different cards of the same note to show up on the same day. One version gives you that early overload effect, while the other gives a kind of extended familiarity effect where for months you'll probably have at least one variation of that cloze come up every day or two. The more variations on a note, the longer this stretches out.

The problem in Anki, at least, is that this is a global deck setting ("Bury related reviews until the next day") and not one you can customize for individual notes. Maybe I should start organizing decks by desired automaticity levels rather than by content.

Comment author: Arshuni 02 May 2016 04:43:26PM *  2 points [-]

Tangentially related: I have found the ease of creating cards one of the most important factors determining the speed at which I learn. For example in anatomy, I started with making cards from a photographic atlas, but this took way too long. (I still desire to make cards from them, since they use actual photos, not simple schematics). For the later, I had to manually cut out the images, and extract the labeling. In contrast, what I ended up using was Gray's Anatomy for Students Flash Cards. It's a 817 page book, with most of it in being alternating pages of images and corresponding names.

This was much easier to make (digital) flash cards from. With pdftk, I could separate the pages into a file of all images and a file of all assorted labels. With pdftotext I could easiliy convert the pdf of labels into text (which I could simply, ~automatically form into flash cards) while I used imagemagick to extract the images themselves. (with all the borders, etc, cut off.) (while I have done a part of the conversion process manually (some names were split into multiple lines, and there were minor irregularities in the text format of the file, so I manually made each section into a separate flashcard (Inserting headers of a card, deleting newline characters if splitting an item into two, and marking the beginnings and endings of clozes), but I am certain that that could've been easily automated, with a proper understanding of regexps. (learning which would've been more economical on my time)). With all this, it took me 13 days until I had first exposure to all 3630 cards, probably significantly less, than if I had to do everything manually.

So, whenever you can, automatize.

(Before this, I used to extract glossaries of books manually. Obviously, I use the same tools for that too, now.)

I learned to love books which had great glossaries, and great summaries. Some are so great, that by reading the summaries, you don't even have to read the actual chapters. (Which may be artificially inflated in length because of a length goal the publishers set, or because it is more targeted towards entertainment value, than for quick conveying of the ideas behind them. A quick skimming may still be worth it, though, even if in preceding chapters you established that their compression is pretty lossless.) Only tables are better.

One thing I want to add to my current toolset is a way to automatically extract wiktionary definitions: sometimes the whole idea is in there, but either way, speaking the language of your desired subject by the time you encounter the more in depth books is handy. (this, coupled with a trickle system, AKA (~20) word(s) of the day)

I used to try to avoid duplicate cards, but I learned to love redundancy.

Also, mind sharing your cards for the holonyms?

Comment author: tanagrabeast 03 May 2016 12:18:44AM *  2 points [-]

You know, I had a start-up idea along these lines recently: something that would combine SRS with social bookmarking.

Example: I'm slowly-but-steadily working my way through Learn You a Haskell for Great Good. I have it on good authority that few people make it as far as I have. I feel like the only reason I can do it is because I stop to make cards for terms, concepts, and many of the examples. I take days or weeks away from the book between sessions while I let those facts firm up in my head, and then I resume.

While I hold that there is real value to making cards yourself when this involves putting things into your own words, making a high-quality card is also a time-consuming chore that is just as much about formatting. I've often wished, as I read, that I had a browser extension that would let me pluck pre-made cards out of a side-bar that went with the passage I was reading -- cards by one of the thousands of people that have no doubt come before me in that chapter.

You can see how this might work. People could build karma when others copy their cards. Site creators might create their own cards as a way to help readers and boost traffic, or pay bounties of some kind to others who make them.

You could browse other cards by the writers of cards you've cloned, and all cards would have automatic links to the sites they go with -- getting around a big problem with imported cards, which is that they are shorn from their creation context.

Monetization? Maybe ads in the corner of the side-bar or something. Maybe partnerships with popular for-pay learning sites.

There are no doubt some thorny copyright issues at play though, and the overall potential market is probably pretty small.

Comment author: ChristianKl 04 May 2016 10:05:32AM 0 points [-]

It seems like your approach would create a lot of cards. How much time to you spend per day reviewing your cards?

Comment author: Arshuni 04 May 2016 06:36:12PM *  0 points [-]

I honestly don't know. I would say quite much, but it does not feel like that: I do not review all my cards at one time in the day (I have notifications periodically nagging me if there are still due cards, so I don't forget, and they aren't too much bother) Another nice trick is to make more, smaller decks. When I see that there are 120 cards in one deck for review, I am not that ecstatic about that. If those same cards are split into 4 decks with 30-30 cars, I don't even think about it. Generally, 20 cards are play, they don't even register, and 80 seems to be the other end, that starts to feel a bit too much. (And the actual number of cards never changed)

If I somehow miss a day, though, that can make things indeed messy.

Comment author: eeuuah 04 June 2016 03:44:42AM 0 points [-]

How do you get notifications only if there are still due cards? I would like this

Comment author: ChristianKl 05 May 2016 05:29:31PM *  0 points [-]

Anki has good statistics so it shouldn't be hard to get the number of daily time spent on reviewing cards. The fact that you don't know is suprising to me as I frequently check the Anki stats. Care to elaborate?

Comment author: Arshuni 05 May 2016 11:55:46PM 0 points [-]

I use org-drill, which, AFAIK, does not collect such data.

Comment author: ChristianKl 03 May 2016 02:07:46PM *  0 points [-]

Also, mind sharing your cards for the holonyms?

I'm still at organizing my way to deal with the information. One example:


Card 1: Front: [anatomy] cubiti/joint.latin(Between Humeris and Radius)

Back (typing): Articulatio Humeroradialis

+Image


Card2: Front: [anatomy] holonym.latin(Articulatio Humeroradialis)

Back (typing): Articulatio cubiti +Image


Card 3/4/5: Front: [anatomy] cubiti/joint.latin(Image1/2/3)

Back (typing): Articulatio Humeroradialis


As far as images go I think 3D programs are the way to go. BodyParts3D is a promising project as it comes with an open license but unfortunately it's not complete and it's UI isn't user-friendly. BioDigital is my other source but unfortunately it has a closed license that prevents sharing of the finished deck.

While we are at the topic of the ontology of anatomy, what's wrong with the English language to have polysemy in "arm" and have it mean both the whole arm and the upper arm?

Comment author: Arshuni 03 May 2016 05:28:37PM 0 points [-]

For BodyParts3D, there is a wikimedia category for a good few animations (it's the place I actually first met it). (https://commons.wikimedia.org/wiki/Category:Animations_using_BodyParts3D_polygon_data) You can download whole categories with (https://commons.wikimedia.org/wiki/Commons:Imker_%28batch_download%29).. For how well does that category cover the desired items, I don't know.

Comment author: root 04 May 2016 02:55:18PM 0 points [-]

I'm aware I'm being off-topic, but have you ever thought about alternative methods of memorization?

Here's two examples, and they have an important thing in common: the answer is visible.

  1. Non-trivia questions: Just like trivia shows, except that they're focused on a narrow subject.

Practical example: Which of the following is [xyz]? [A] [B] [C] [D]

  1. Fill in the blank and a 'answers paper'. You need to fill in the correct answer from one of the answers provided in a separate paper.

Question XYZ: ____ Answer1 Answer2 Answer3 .... AnswerN

I've designed those on the basis of me having a strong nonconsious memory, but haveing difficulty with active recall. But I feel much more confident in my answers when I can remember them like that. Your own milleage may vary.

I'm also interested in some criticism of SRS, because every time I see something 'good' I also want to see how many holes can be poked in it. The wiki gave me some sort of 'this is amazingly awesome' and I'm just curious, how true is that? For example, if we have x number of cards in a typical deck, can we grade the usefulness of each card? It can get rather personal here but sometimes I have a conflict between perfectionism and practicalism, in which perfectionism says 'You could completely screw up by missing those details' and practicalism says 'How important is it that you know?', and I'm curious if I'm the only one who feels this way.

Comment author: ChristianKl 04 May 2016 03:17:37PM 1 point [-]

Practical example: Which of the following is [xyz]? [A] [B] [C] [D]

I did made hundreds of Anki cards on that basis with 2 to 3 answers and my conclusion is that it's a bad idea. Given "what fires together wires together" cards like that seem to create links between the question and the wrong answers.

For example, if we have x number of cards in a typical deck, can we grade the usefulness of each card?

The typical deck is going to be different for different people.

Comment author: tanagrabeast 05 May 2016 02:05:30AM 1 point [-]

I did made hundreds of Anki cards on that basis with 2 to 3 answers and my conclusion is that it's a bad idea. Given "what fires together wires together" cards like that seem to create links between the question and the wrong answers.

There's also a risk that you become dependent on being able to look for the answer visually rather than being able to fish it out of year head; in most real-world cases, it's the latter skill you need.

Comment author: ChristianKl 05 May 2016 05:19:58PM 0 points [-]

Yes, depending on how you need the knowledge that's an issue. But it's an issue that I would expect most smart people to be conscious of when they make the decision to make cards like this.

The effect I mentioned isn't easily anticipated and it took me a lot of empirical study to find that it's there.

Comment author: richard_reitz 02 May 2016 05:38:30AM 0 points [-]

my classes continue to perform with increasingly minimal note-taking and homework.

Which homework hasn't been assigned because of Anki? Remembering back to my high school English classes, the only homework I can remember doing was reading readings and writing essays. I can't see how either could be displaced by Anki.

Comment author: tanagrabeast 02 May 2016 05:51:51AM *  1 point [-]

I've cut back on note-taking quite a bit thanks to Anki. They weren't looking up those notes anyway. If they want them bad enough they can look them up on my web page or go straight to the Anki cards.

Anki hasn't displaced much homework, though, as there wasn't much left to displace. I don't give it mostly because few of my students would do it; they are not strongly motivated by grades. This is especially true of reading homework; I gave up on that a year after I stopped teaching honors after getting about 10% compliance. Reading happens in class or not at all, and yes, it is a big challenge to squeeze this in and still do all of the other things we need to do. It's important, though. For most of my students, the reading we do together is the only reading-at-length they do all year. They admit this readily -- even proudly.

Essays are more mixed. We don't do too many full ones, and the ones we do mostly get done in class. The "homework" is there just as safety valve for those who care enough to make their essay great.

Comment author: Vaniver 02 May 2016 05:56:43PM 0 points [-]

I don't give it mostly because few of my students would do it; they are not strongly motivated by grades.

Hm. I was about to suggest that the natural way to get daily Anki compliance is to have that be homework--it should be easy for students to send you some record / to find or build a webapp where you can see whether or not students are reviewing their cards.

(This runs into trouble with digital access; students may have a hard time getting on the Internet on Saturdays or Sundays. But for many schools this isn't a problem.)

Comment author: tanagrabeast 02 May 2016 11:32:43PM *  1 point [-]

Monitoring features are definitely a part of the vision I'll be laying out in the next post, but more as a way to make classroom time more productive than as a homework enforcement aid. To get them to use something on their own time I'm going to have to be more clever, and make them feel like it was their idea.