Comment Permalink

gwern13y50

I'd like to work on this (be paid to substantially improve my http://www.gwern.net/Mnemosyne ? Where's the downside?), but you ask hard questions which would require a lot of research to be done to my satisfaction and during which I could be scooped. I'm not sure a race is the best way to run this.

Duke13y40

Why do you care if it is done to your satisfaction when the prize is awarded based on other's satisfaction with it?

This is an expected value problem. Decide how much a unit of your time is worth, how much time you are willing to invest and then (the hard part) estimate your likelihood for success.

So, if you value your time at $10/hr, are willing to invest 10 hours and estimate you will win the $155 X% of the time then we get this equation:

P(winning)$amount won - P(losing)$amount not won = initial investment

x$155 - (1-x)$0 = $10/hr * 10 hours

solve for x

x ... (read more)

0jsalvatier13y

Could you specify the drawbacks you see to a race? Does something else seem like it would work better? Does a dated prize (where the best entry is selected on a specific date) or a work contract seem like it would work significantly better? I'm Just Trying Things right now, and I'm not especially attached to this format. Which are the especially hard questions? What is the primary source of difficulty?

See in context

26 [prize] Spaced Repetition literature review

by jsalvatier

7th Jun 2011

2 min read

26

EDIT: I am canceling this contest because I feel that the structure of the incentives were poorly thought out (see gwern's comments). I will be posting a new better structured contest in the near future (now posted). If you feel this is unfair or otherwise feel slighted, please contact me at my username at gmail.com.

I'm interested in making projects happen on Less Wrong. In order to find out what works and to inspire others to try things too, I'm sponsoring the following small project:

Spaced Repetition is often mentioned on on Less Wrong as a technique for remembering things. I've started using Anki and it certainly seems to be useful. However, I haven't seen a good summary of evidence on Spaced Repetition.

I hereby offer a prize to the first person to submit a good summary of the evidence on Spaced Repetition to the main page. The winner will get the prize, currently: $265 + 40 to charity (see comments)

The summary should address at least the following questions:

What spacing is best?
How much does spaced repetition actually help memory?
Does spaced repetition have hidden benefits or costs?
Does the effectiveness vary across domains? How much?
Is there research on the kinds of questions that work best?
What questions do researchers think are most important?
Is there any interesting ongoing research? If so, what is it on?
What, if any, questions do researchers think it is important to answer? Are there other unanswered questions that would jump out at a smart person?
What does spaced repetition not do that people might expect it to?

The post should summarize the state of current evidence and provide citations to back up the claims in the article.

If you think you would benefit from the result of this project, please add to the prize! You can contribute to the prize on the ChipIn page.

Whether the summary is 'good' will be judged by me. If there is a serious dispute, I'll agree to dispute resolution by any uninvolved party with more than 5k karma.

If you have suggestions, questions or comments, please leave them in the comments.

If you would like to work on this project, please say so in the comments below. Collaboration is encouraged.

This project is tagged with the 'project' tag and listed on the Projects wiki page.

Spaced Repetition

Personal Blog

26

Mentioned in

23[prize] new contest for Spaced Repetition literature review ($365+)

New Comment

35 comments, sorted by

top scoring

Click to highlight new comments since: Today at 5:55 PM

[-]Dr_Manhattan13y60

summary of the evidence on Spaced Repetition

Would probably help to quantify that somewhat - as in "evidence that spaced repetition does X" (X being what you want to get from it).

With this framework evidence that SRS does not do X should also get the prize.

Putting it as "claims are made that SRS does X, here is the pro/con evidence" would also be a good format.

As I also consider this topic as very valuable I'll follow Joshua's suggestion and offer $20 to SIAI for this post.

[-]gwern13y50

[-]Duke13y40

Why do you care if it is done to your satisfaction when the prize is awarded based on other's satisfaction with it?

This is an expected value problem. Decide how much a unit of your time is worth, how much time you are willing to invest and then (the hard part) estimate your likelihood for success.

So, if you value your time at $10/hr, are willing to invest 10 hours and estimate you will win the $155 X% of the time then we get this equation:

P(winning)$amount won - P(losing)$amount not won = initial investment

x$155 - (1-x)$0 = $10/hr * 10 hours

solve for x

x = 64.5%, in other words, to "break even" you need to be sure that if you invest 10 hours of your time in this project that you will win it at least 64.5% of the time.

A race may not be the best way to run this for you, since I suspect that you value your time a high rate relative to the potential payoff. But someone who values their time less (or is more productive than you per unit time) may think a race is a wonderful idea.

[-]gwern13y-10

Why do you care if it is done to your satisfaction when the prize is awarded based on other's satisfaction with it?

Why do I care... what I care? Seriously?

But I do agree with your expected utility calculations. The problem is, I can't motivate myself to write under conditions of uncertainty like that. I understand that shouldn't matter, especially since I consider such a project worthwhile even in the absence of $155, but it does.

[-]Duke13y20

Getting it done to other's satisfaction and getting it done to your own are not mutually exclusive. You can work quickly to win the prize and then go back and expand.

[-]gwern13y-10

You can work quickly to win the prize and then go back and expand.

No, I can't. That's my problem, as I've been explaining in my comments.

[-]jsalvatier13y10

Perhaps you two should collaborate and split the prize in some way.

[-]gwern13y-10

I think it's a bit late for that. But I've PMed Duke.

[-]jsalvatier13y00

Which are the especially hard questions? What is the primary source of difficulty?

[-]steven046113y60

Could you specify the drawbacks you see to a race?

Duplicated effort, some of which is presumably invisible.

[-]gwern13y60

From what I remember of the prize literature, prizes work best for well-delineated, objectively judgeable accomplishments. Like putting a rocket into orbit - you can plan out what is necessary, and it's clear when you succeeded. A literature review isn't really either. So I'm not sure a prize in general is succeeded for the task.

My personal problems are that:

I can't bring myself to start a new project like intensively researching spaced repetition to establish exactly how good the evidence is, how effective it is, and what domains it does and does not work in, without having a good idea of what I might receive in turn. Uncertainty is paralyzing for me.
This is especially true because your prize is large - I felt guilty taking a bitcoin from Kiba for Bitcoin is Worse is Better because I knew it was incomplete in several respects, although it's one of my best essays and gave him a lot of traffic; and a bitcoin is only worth $20, so how would I feel delivering something bad for $155? I'd feel really terrible!

The difficult questions are:

How much does spaced repetition actually help memory?

Quantifying the advantage in any more than an ad hoc way like http://www.gwern.net/Mnemosyne#fn3 would be difficult since it requires delving into the specifics of the data, and forgetting curves are non-linear so it's not clear what is even meant by 'how much' - memories per review at 1 year? At 10 years? At 50 years? Integrated over an actuarially projected lifespan?
Does spaced repetition have hidden benefits or costs?

There's probably no direct research on this, so now one has to delve into related research fields and look for very weak evidence or correlations, like maybe high memory correlated with longevity in old people.
Is there any interesting ongoing research? If so, what is it on?

New research tends to be hard to get (= wasted time going around paywalls) and requires following a lot of citations in bibliographies and checking out blogs and emails to really get a sense of where a field is now.

The best way is probably to informally or formally contract with someone like lukeprog (who does this sort of literature review routinely) to conduct the review.

[-]jsalvatier13y20

Great comment. Perhaps I should be reading the prize literature! I'll think about how to structure this better.

On your personal problems: consider finding someone who has more time and acting as a consultant for them. For example, Duke, who comments elsewhere in this thread.

Just to make sure I'm clear, I'm not trying to gain a detailed or very specific understanding of the literature on SR, but to figure out how confident I should be in recommending SR to others and to myself and what I should recommend others and I do and what I should expect it to help and how much.

I actually hadn't read your page before, so thanks. I actually think your page is a fairly good start.

[-]jsalvatier13y10

Do you have a good starting point for reading on the prize literature?

[-]gwern13y-10

Hm, not really. What I would do is google Robin Hanson, Anders Broberg, and LW for mentions of 'prize' and use them as entrance into the academic literature, and spider authors & citations & bibliographies from there.

[-]jsalvatier13y10

ok, thanks.

[-]gwern13y30

It's not a literature review, but http://33bits.org/2011/06/06/the-surprising-effectiveness-of-prizes-as-catalysts-of-innovation/ does link to a McKinsey report on prizes, which ought to be good reading if their reputation can be relied upon.

[-]randomwalker13y40

It's funny you linked to my post on prizes, which brought me here, because I'm actually very interested in spaced repetition. In fact just a couple of weeks ago I mentioned it's potentially worth millions of dollars to me: http://arvindn.livejournal.com/132233.html

You guys need to get more rational about the value and importance of what you're doing in this community. You're arguing over trivial amounts of money here. I'd say that any post that's good enough to get voted up to the main page is worth tens of thousands of dollars to mankind. Easily.

I'm putting my money where my mouth is chipping in $100 for the new prize. [Edit. Hmm, I got a paypal receipt but the total on chipin hasn't changed. Maybe just need to wait for a bit.]

Finally, I'm only a sporadic reader, but allow me to express my deep gratitude and admiration for the community.

[-]gwern13y20

! You're Arvind? I had no idea. Gosh, the past few days have been interesting for me - first Piotr Wozniak tells me he's incrementally reading 8 articles mentioning or by me, then Danny Hillis replies to a comment of mine over at the Technium, and now here's Arvind Narayanan.

I'd say that any post that's good enough to get voted up to the main page is worth tens of thousands of dollars to mankind. Easily.

On one hand, I agree - if a main page post improves one high-value person's time, that alone is probably a few tens of thousands' worth. The other part of me is cynically remarking that Wozniak and psychologists in general have been advocating spaced repetition for the last >2 decades / centuries (respectively), the research is unassailable, and yet... So what's the marginal return on another literature survey?

A few comments:

You mention creatine on your blog. Creatine only works in specific demographics; I don't know whether they include you.
You mention dual n-back; the results seem to keep getting weaker since Jaeggi 2008, and Jaeggi 2011 seems extremely weak. I used to be extremely enthusiastic, but as the studies keep coming in... At this point, I'm not sure how much time is worth putting into DNB for just the WM benefits. Probably not much past D5B though.

[-]randomwalker13y00

Thanks for the information! That does look pretty damning for dual n-back. As for creatine, I'm not vegetarian but my meat consumption is lower than average, especially when I'm not weight training. At any rate, I stopped taking it after a couple of months because I decided the unknown long-term health risk from increased renal stress wasn't worth it.

[-]Bongo13y00

voted up to the main page

Note that posts aren't voted up to anywhere; they're promoted by editors.

[-]randomwalker13y00

Ah, I see, thanks.

[-]lukeprog13y30

For relevant research, see all the sources I listed here.

[-]JoshuaZ13y30

I'm not quite willing to chipin for the individual in question. But I'm happy to donate $20 to the charity of their choice.

[-]jsalvatier13y00

Can you explain your reasons?

[-]JoshuaZ13y30

It is a compromise between fuzzies v. utilons because I am unable to purchase them separately along with some other constraints. If the money goes to a charity I get more fuzzies, and given that whoever would do this would likely pick a charity that I consider to be a worthwhile charity (given the general framework of LW) there's a high chance that it will produce heavy utilons. I also don't have that much money so I can convince myself to part with more if I know that it is going to a charity rather than going to someone's personal use.

[-]jsalvatier13y00

Thanks :)

[-]Duke13y20

My entry has been submitted: http://lesswrong.com/lw/64k/memory_spaced_repetition_and_life/

[-]Nick_Roy13y20

I chipped in $10 for the winner of the prize.

My sincere thanks to anyone who participates!

[-]Duke13y20

I am an Anki user and I am interested in working on this project.

[-]Duke13y10

I am working on this vigorously and expect to produce a submitted product soon. Proceed at your own risk if you are considering competing for this prize.

[-]jsalvatier13y20

Can you describe what does and does not increase your motivation? Does more money increase/decrease? The race aspect? The competitive aspect?

[-]nazgulnarsil13y10

the answers to your first few questions seem to unfortunately be "it depends". the point of SRS is that you review when you are on the cusp of forgetting a piece of knowledge, which varies drastically depending on individual, conditions, and topic.

[-]Duke13y00

Cross-commented from submission thread:

I think it is unfair of you to post a public critique of my submission since this is a contest. I have effectively been penalized for being first. Every submission that follows will have the benefit of seeing this critique.

I am also concerned that you have decided to change the contest format immediately following my submission. In my estimation, you had either already decided to change the format prior to my submission (clearly a major disadvantage to me), or you decided to change the format based on my submission, which, again, effectively penalizes me for being first.

Additionally, I request arbitration. I think this contest has been mismanaged to my detriment; as such, I am entitled to financial compensation.

[-]jsalvatier13y00

Link

[-]Duke13y10

After some deliberation, I've decided to withdraw this request. I am content with my submission. I am also content not to receive the prize or any portion thereof.

Moderation Log