Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Spaced Repetition literature review prize: And the winner is...

26 Post author: jsalvatier 19 August 2011 08:35PM

The Spaced Repetition literature review prize for the best new review of the evidence on Spaced Repetition has ended and the judging panel has made its decision. The prize attracted entries from Duke (entry) and Gwern (entry). After reviewing the submissions separately and then discussing them together, the judging panel unanimously judged Gwern's entry to be the best.

With great pleasure, we now award Gwern the prize of $385. 

Anki cards for Gwern's review are available as the shared deck "Gwern Spaced Repetition Lit Review". If you have improvements or alternate decks, post them in the comments.

The three judges, BenLowell, Guy Srinivasan and John Salvatier (me), are Seattle LessWrongers who volunteered to judge the contest. 

We thank Duke and Gwern for their submissions, as well as users randomwalkerAntisuji, Dr_Manhattan, Benquo, Nick_Roy for contributing to the prize fund.

Comments (25)

Comment author: gwern 19 August 2011 11:50:17PM *  16 points [-]

I'd like to thank John for putting up the prize money and just starting the prize - such things can be thankless, and as public goods one might expect them to be under-supplied. I would have worked in some of that research, but not this soon nor in as much depth.

I also appreciate the feedback he and the judges left on the SR page. Feedback is a pretty rare thing for things I write.

I'm afraid I don't have grandiose plans for the money - $272 is earmarked for repairs to my car (in part thanks to my sister), and I already spent $70 on fish oil and vitamin D. The $40 left may enable me to finally experiment with nicotine, which ought to be interesting.

EDIT: I ordered the nicotine-water last night. Preliminary plans: using the habit-formation for exercise, spaced repetition, and n-backing.

Comment author: Metus 20 August 2011 06:41:12PM 3 points [-]

Upvoted for putting your money to good use. I enjoy your work on nootropics and am looking forward to reading more from you.

Comment author: jwhendy 31 August 2011 10:40:43PM 1 point [-]

Your website might have stripped out the division sign. I see:

The concentration is 100 mg of nicotine per ml, there are 1000 ml per liter, one wants 2 mg per dose, so 100×10002=50000 doses...

Took me a sec to figure out it should be 100 x 1000 / 2.

Comment author: gwern 31 August 2011 11:14:23PM 1 point [-]

That's just you. Your web browser is not running Javascript, you don't have the fonts, or something. I see the division as I wrote it in Firefox: http://i.imgur.com/c2U2R.png

Comment author: arundelo 31 August 2011 11:34:06PM 2 points [-]

I see the HTML source uses MathML:

<math display="inline" xmlns="<http://www.w3.org/1998/Math/MathML>"><mrow><mfrac><mrow><mn>100</mn><mo>&#215;</mo><mn>1000</mn></mrow><mn>2</mn></mfrac><mo>=</mo><mn>50000</mn></mrow></math>

I couldn't tell from the screenshot what browser and OS you're using but for me (on Linux), it shows up as "100×10002" in Chrome but as a (correct) fraction in Firefox (3.5). In both it acts like "100×10002" in the contexts of searching and copying/pasting.

Comment author: jwhendy 01 September 2011 02:42:03AM 1 point [-]

Ditto -- on Arch linux, chrome with javascript enabled does not work; firefox 3.6 does.

Comment author: gwern 01 September 2011 12:56:08AM 0 points [-]

I see the HTML source uses MathML:

Correct. The Javascript is because apparently browsers have to be forced to render the MathML; I don't understand the details.

I couldn't tell from the screenshot what browser and OS you're using

Debian unstable's Iceweasel.

Comment author: Douglas_Knight 01 September 2011 02:03:38AM 2 points [-]

I tried three browsers and in none of them did javascript make a difference. (I had both success and failure.)

Comment author: arundelo 01 September 2011 01:16:08AM 1 point [-]

(Just in case it wasn't clear, I do have JavaScript enabled in Chrome.)

Comment author: gwern 19 August 2011 11:50:24PM *  19 points [-]

With all that said, I do have some feedback for jsalvatier! (Please don't take this as defecting criticism; also, in the interests of honesty - I noticed all of #2 shortly after the contest began, and kept quiet out of plain self-interest, waiting until the judging happened).

  1. First, it caused me a little stress that judging took so long. The deadline was 1 August, it didn't have to take 18 days to judge, even if events intervened (as they apparently did) to block the earlier gathering which would judge the contest - it could have been done online by so few judges. This is especially true for a contest with only two entries.
  2. The contest was mishandled from the standpoint of publicity. I should not have been so confident I would win; there should have been more than 2 entries with $400 at stake for a task intrinsically worthwhile. There are a number of ways this could have been advertised better:

    1. This could have been announced on the Mnemosyne and Anki mailing lists and forums. It is quite relevant to their interests and they are the obvious places to announce such a contest, even if they would not link it on their front pages.
    2. It could have been advertised better on LW. This was really generous and interesting and potentially a key community tool! (lukeprog won't be this active forever.) Why wasn't the contest an article on the Front Page? (AFAIK, it never appeared there.)
    3. Piotr Wozniak, the expert of experts on spaced repetition in practice, could have been contacted to participate or possibly advertise it on Supermemo.com; he may sound intimidating from the Wired article, but my email correspondence with him was perfectly normal and fluid, with the ordinary delays.
    4. The contest, per #2, could have been submitted to Reddit (the psychology subreddit being only the most obvious of multiple potential targets) or Hacker News; both communities are interested in spaced repetition. This is also true of Imminst.
    5. Google AdWords could have helped; $10 buys a lot of clicks when it comes to spaced-repetition ads, and could well be worth it. ($350 is about as impressive as $360 as seed for a prize.)
    6. Interested LessWrongers could have been personally contacted. Anyone who has created a deck of flashcards is obvious; and why not contact lukeprog? Reading and summarizing papers and reviews is practically his specialty. He is busy, of course, but it's not like spaced repetition is that worthless compared to his current topics, and the prize may make up for lost time.
Comment author: jsalvatier 20 August 2011 12:37:46AM 9 points [-]

1 I did actually announce this on the Mnemnosyne and Anki mailing lists, but I received no responses. :(

2 I felt timid about advertising this on the mainpage, though I am not sure why in retrospect.

3-5 are good suggestions I hadn't thought of. The question is how to generate similar suggestions for prizes in the future. I suppose 5 is obviously applicable, 4 probably applies to many contests, I'm not sure about 3. Perhaps ask if there are any famous figures in the field I am trying to summarize.

6 feels like it wouldn't be effective to me, but I'm not sure why. I suppose the costs to trying this in the future are low.

Thanks again :)

Comment author: gwern 20 August 2011 12:59:50AM *  5 points [-]

1 I did actually announce this on the Mnemnosyne and Anki mailing lists, but I received no responses. :(

Whups! So you're right. I replied with news about the verdict.

#2 is definitely something you should try. If there were no real prize, I think people might laugh at an attempt for the main page, but with that much skin in the game... at least, I would expect LessWrongers to respect someone putting a good chunk of money where their mouth is. They seem to've in the past, with reports about bets, prediction market trades, and SIAI donations all getting upvoted.

I suspect you feel about #6 the same way you feel about #2 - that it's overreaching and arrogant. That is something you may want to look into if you feel the same way in a second contest of equivalent quality. (I understand 'rejection therapy' may help with that.)

EDIT: I've spent a fair bit of time messing with AdWords for my own purposes, and I think I have a handle on the basics, so I'd be willing to help out with future contests as far as that goes.

Comment author: jsalvatier 20 August 2011 02:34:10AM 1 point [-]

I think I would feel much more confident about a second prize. In the beginning I didn't have other people helping me do this kind of work, but now I do.

Comment author: GuySrinivasan 20 August 2011 12:38:39AM 0 points [-]

I agree with all of this, and thanks for the feedback. For my part in causing the judging to take long I apologize; I didn't try to delay it, but I definitely didn't put any explicit effort into speeding it up either. Publicity is one of the things we discussed in our brief retrospective. It's likely another contest will take place, and if it does, I will make an effort to spread the word, but am not committing to make such an effort that no more will be needed. ;)

Comment author: jsalvatier 19 August 2011 11:58:28PM 0 points [-]

Thanks gwern!

Comment author: JenniferRM 22 August 2011 05:25:05AM 5 points [-]

I had a six month long "4-6 days a week" Anki habit under development before I started my most recent gig. The new work routines disrupted my Anki habit and several days ago, actually using some of the content, I realized I'd forgotten how to fingerspell a certain letter that was in one of my decks. Reading these essays reminded me of all the good reasons I had for doing Anki and I started working through my backlog of cards.

I just wanted to say thanks for the contest and the essays and everything. It is more likely than not that I'll reform my habit in my new environment. You caused the splash and I benefited from the positive ripples. Thank you to everyone involved :-)

Comment author: JoshuaZ 20 August 2011 01:32:32AM 3 points [-]

Ok. I had earlier promised to donate $20 to the charity of the winner's choice. So, Gwern please choose.

Comment author: gwern 20 August 2011 03:01:09AM 7 points [-]

Oh! Oh, OK then - donate to the Internet Archive, please.

(No doubt people would disagree because it comes out very badly on a utilitarian calculation, but this is more about discharging my personal emotion of obligation - I use the Archive on a literally daily basis.)

Comment author: JoshuaZ 22 August 2011 12:28:01AM 2 points [-]

Done. Receipt number is 4518-9128-6646-8864.

Comment author: gwern 22 August 2011 12:39:37AM 0 points [-]


Comment author: Dr_Manhattan 21 August 2011 01:59:46PM 1 point [-]

Curious what do you find so useful other than looking up ol' webpages

Comment author: gwern 21 August 2011 02:15:12PM 2 points [-]


There's also a lot of media on IA; I got a good chunk of Turkish and instrumental music there, as well as a number of movies.

Comment author: [deleted] 21 August 2011 02:17:26PM 0 points [-]

No doubt people would disagree because it comes out very badly on a utilitarian calculation

People have weird utility functions.

Comment author: GuySrinivasan 20 August 2011 12:41:35AM 2 points [-]

My biggest takeaway from the research is that it's worth thinking briefly about every new Anki card whether I can make it an effective short-answer question rather than a cloze-deletion.

Er... that's my biggest action item. My biggest takeaway is that research definitely supports spaced repetition + testing far over studying.

Comment author: jsalvatier 20 August 2011 02:37:32AM 2 points [-]

I actually hadn't noticed that Spaced Repetition is testing till this research. I didn't know about the testing effect. I think knowing about the testing effect will make me more effective learning things more generally. I'd actually like to see an stand alone article on the testing effect.