Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

sinak comments on Who Wants To Start An Important Startup? - Less Wrong

42 Post author: ShannonFriedman 16 August 2012 08:02PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (407)

You are viewing a single comment's thread.

Comment author: sinak 14 August 2012 09:21:35PM 15 points [-]

In addition to my last idea, here's another thing I've been kicking around:

* Anki-as-a-Service *

Problem: Anki is great, but the user interface is mediocre and it acts as a standalone application on the platform of your choice (desktop, mobile, etc).

Solution: A hosted version of Anki accompanied by a mobile application that allows users to enter items manually, capture items from the browser via a javascript bookmarklet, or allows third parties to submit information for users via an API. In essence this would amount to "learning as a service" and could eventually be extended beyond the feature set currently offered by Anki by including customized tests for different content types.

Current development: Very much in the idea stage. Interested in hearing what ideas people have around this.

Comment author: Micaiah_Chang 17 August 2012 12:56:05AM 6 points [-]

I imagine this would be very hard to monetize and get customers as-is. The below is merely a brief list of problems that I've thought about

The average user needs to be sold on the effectiveness of a product very fast, on the first usage (or perhaps even sooner!) in order for them to continue using. However, SRS software in general are almost by definition antithetical to that goal: Their benefits do not come until far into the future, worse still it's an undefined time in the future. Sure you can use arguments about the benefits of SRS and the psychology of memory and <insert gwern.net FAQ here>, but it would appear to be an uphill battle to make the benefits immediately relevant and immediately relevant to the people who wouldn't already be using Anki and other free equivalents.

In addition, before you can even start using the product as advertised, you have to learn how to make cards that are easy to memorize or download a deck which is already well made. The first is "Wait so you want me to learn all these tiny rules before I can even start learning? <browser back button>". The second presents a chicken-and-egg problem. How are you going to have high quality decks that teach things? By having users! How are you going to get users? By having high qual- oh, darn.

It would appear that your general idea is going in the right direction; to make the best SRS program as painless as possible and to extend it to be more powerful. Your emphasis though, would appear to be more oriented toward existing power users of SRS. So there's the matter of getting them to switch which... I have no idea how hard that would be. (Sample size of two; you'd obviously build something you'd want to use; I'd jump on board instantly if I could transfer my existing Anki decks).

One possible solution to the adoption is to piggyback it on an existing service; if users get to use it as an additional option on something they already use habitually then getting consistent usage wouldn't be as much of a problem. I believe Khan Academy has expressed interest in including SRS in there. Another is to try and "gamify" it (argh I hate that word) by either making the entire application a sort of game or incorporating cow clicker like features in there to get the user hooked (IT'S NOT EVIL IF THE ADDICTION IS GOOD).

The making your own decks feature can be mostly hidden from the normal user, with a gradual introduction to it as they use the product more (paid feature?). As for having high quality decks; you can try porting the entire Anki library of downloadable decks, filter them in some way and use that to bootstrap up to a much higher standard of quality.

Of course, any and all advice here means absolutely jack compared to the behavior of actual users; release a minimal version, see who bites and check to see what the users complain about before even thinking about what I said here. Making money is and should probably be a distant 4th or 5th consideration behind making a product that you would use and making it easily extensible.

Comment author: Persol 17 August 2012 01:29:20PM *  7 points [-]

I think Micaiah_Chang mostly nailed this. I actually wrote a site that did this a few months ago. I had about 4000 users who had actually gone through a complete session.

it would appear to be an uphill battle to make the benefits immediately relevant

As guessed, the problem is that I couldn't get people to start forming it as a habit. There is no immediate payback. Less than 20 people out of 4000 did more than one session.

you have to learn how to make cards that are easy to memorize or download a deck which is already well made

This one is easily solved. The Anki decks have a (weak) rating system, and allowing people to import anki decks was easy.

Additionally, there are at least 18 competitors. Here's the list I made at the time. Very few seem to be successful.

I shut the site down about a month ago. There are numerous free competitors which don't have any great annoyances. I wouldn't suggest starting another of these sites unless you figured out an effective way to "gamify" it.

Comment author: gwern 18 August 2012 01:33:21AM 3 points [-]

As guessed, the problem is that I couldn't get people to start forming it as a habit. There is no immediate payback. Less than 20 people out of 2000 did more than one session.

Wow. You did a spaced repetition site which had 4000 people try, 2000 finish a session, and <20 return for a second review session?

Comment author: Persol 18 August 2012 02:43:42AM *  3 points [-]

Sorry, typo. ~4000 people finished a session. Many more 'tried' than 4000... I just couldn't determine which users were bots that registered randomly vs users that didn't finish the first session.

  • Tried: lots (but unknown)
  • Finished 1 session: ~4000
  • Finished >1 session: ~20
Comment author: Shak 18 August 2012 05:35:48AM 1 point [-]

Persol, what traffic generating methods did you use to get those kind of figures?

Comment author: Persol 18 August 2012 12:13:26PM *  2 points [-]

About 75% of the hits came from Google adWords, which was on for about 8 months. Maybe about 10% from search results. I also had a few links from subject specific websites. Average CTR was about 0.25%. Best CTR were ads that mentioned 'flashcards' and 'online'. The best conversion rate (answered a study session question) was 17% with the ad below:

  • Remember Facts
  • Spend less time studying.
  • Remember more material.
  • www.superbrain.me

Graphic/animated ads were a waste of money, but at least I learned how to make animated GIFs.

Comment author: supermemofan 21 August 2012 01:10:47AM 6 points [-]

Supermemo has been working on this problem since 1982, and they struggle to make Supermemo popular because it is not "sticky" enough. Basically it involves hard work for future benefit. This is the mental equivalent of "earning an honest living," or "getting rich slowly." We do not live in a world that portrays honest, slow but meaningful progress. We live in a world that is obsessed with instant gratification, and SRS methods go contrary to the river-like "current" of this system. See: http://wiki.supermemo.org/index.php?title=Why_isn'tSuperMemomore_popular%3F Also, the only way I could see SRS hitting massive appeal is if it were designed from the ground up to be a game of some sort where doing flashcard repetitions resulted in progress. I have an idea of how it would work, but I doubt we will see anything like that for the time being.

Comment author: shminux 21 August 2012 07:08:57PM 3 points [-]

Also, the only way I could see SRS hitting massive appeal is if it were designed from the ground up to be a game of some sort

Judging by the user-hostile interface of the linked site, there is probably a much lower hanging fruit in the app's UI redesign.

Comment author: supermemofan 22 August 2012 01:59:17AM 0 points [-]

Supermemo was designed by a scientist, and a simple user interface is not as high on the priority list as optimized algorithms, incremental reading, etc. I highly doubt that outside of Supermemo a more effective flashcard software exists. Thus, to make flashcard-based learning hit critical mass, it must be packaged in a way that is extremely appealing and fun (Aside from the joy of learning) so that people can survive the two-week "hump" that most people stop using Supermemo. Other than a game or a more aesthetically pleasing UI, I think we're kind of stuck with our present situation, unless a higher authority (Education system, government) makes such learning techniques mandatory, which will not happen due to the inefficiency of modern institutions to fulfill their explicit purpose. At least they exist for the self-motivated to use, and it gives us a mental edge on those that do not use (Or care to use) SRS.

Comment author: Persol 21 August 2012 06:54:46PM *  2 points [-]

That link (fixed version ) is very accurate. I wish I'd considered the first few points BEFORE programming/advertising the site.

Comment author: sinak 18 August 2012 02:11:56AM 1 point [-]

Persol, that list of competitors is massively helpful, thank you.

I'd love to hear more about your experiences, and to get a better of idea of exactly what you had built. I think what I have in mind is a more mass-audience version of SRS (see my response to Micaiah above) rather than a more traditional Anki-type system.

I'd love to know how you were monetizing the service, and if there are any screenshots of what the site looked like. Did you offer a mobile application? Did you try to push people to engage via push notifications at all? I think this is definitely a core part of a strategy that I'd push for. I think "gamifying" is also an interesting route, need to think about this angle a little more carefully though.

Comment author: Persol 18 August 2012 03:22:22AM *  3 points [-]

Here's the actual PHP code, weighing in at 18Mb. It's probably the best way to get a feel for what it was, and it might help you decide what to do.

It includes:

  • most of the site code - This code is from about a month before I moved onto a more rewarding project, but it's the last full set I have.
  • automatic stylesheets/icons for iPhone and Android (not an application, but did create an icon on the home screen)
  • a bunch of draft banner ads - the animated GIFs summarize how the site worked
  • a research folder with information on SRS publications
  • screenshots of other SRS engines

* This version may not have the correct repetition calculation. Due to the inherent time factor, it was a hassle to test, so I didn't fix that part of the code until later. *

It was admittedly an ugly (but fast loading) site. After a few weeks of cheap banner ads and seeing the minimal reuse, I just set it to coast until the year ran out.

I did do some A/B testing with email notifications about a month in. It didn't have a measurable effect of return use.

Monetization was via banner ads. Via A/B testing, the best location for the ad was under the card's question. Once flipping the card, the ad was hidden. I also deactivated the ads for awhile too see if they were too intrusive; return visitors didn't improve.

I also incorporated graphics and audio, since the most successful SRS systems seem to revolve around vocabulary. I personally used it to help learn basic Mandarin for use with my in-laws... but it is a boring way of learning a language. While it is much more effective than Rosetta Stone, it is very difficult to stay engaged.

"Remember what you read" If you take a look at a newer version of SuperMemo, it has this feature.

Comment author: sinak 21 August 2012 12:49:30AM 1 point [-]

This is great - I'm going to take a look through the code and see if I can get it running on a test server.

Comment author: arundelo 17 August 2012 02:29:09PM 6 points [-]

incorporating cow clicker like features in there to get the user hooked

For anyone who doesn't know it, here's the (really interesting) story of Ian Bogost's "Cow Clicker".

Comment author: sinak 18 August 2012 02:02:44AM *  2 points [-]

MIcaiah, thanks for the detailed and well thought-out response. I'll try and respond to some of your thoughts:

I imagine this would be very hard to monetize and get customers as-is.

As far as monetization goes, I think the best route would be to charge online education providers on per-API-call basis. The end goal would be to become something akin to the "Twilio of online learning." With a sufficiently developed system, I think it'll be possible to convince companies in the online learning space that this is a worthwhile value proposition for their users. End users who have committed to a particular online learning program are much more likely to be willing to use a spaced-repetition learning system to aid in their progress in a particular course.

Your emphasis though, would appear to be more oriented toward existing power users of SRS.

I think I gave the wrong impression here, I think I'd much rather target non-users of SRS. Building something simpler but more accessible seems like a more viable alternative. Gaining traction with average, non-SRSing users, and then later adding best-of-breed features to tempt online learning providers seems like a more reasonable approach.

The average user needs to be sold on the effectiveness of a product very fast, on the first usage (or perhaps even sooner!) in order for them to continue using. However, SRS software in general are almost by definition antithetical to that goal: Their benefits do not come until far into the future, worse still it's an undefined time in the future.

I've thought of a couple of simple use cases for this sort of platform that I think seem easy to build and quite compelling for an average non-SRS user:

  • Vocabulary expansion - For People looking to expand their vocabulary - a simple javascript bookmarklet that would allow users to learn the definitions of new words that they come across.
  • "Remember what you read" - It seems that given the number of things that an average person might hope to learn in a particular day, but which are instead soon forgotten, having a simple way to record those items would be quite immediately valuable. For example, as soon as I found Instapaper, I began saving documents that I wanted to read later. I could see a simple javascript bookmarklet for "things I've read online and want to commit to memory" being used in a similar fashion. This implementation would be a very, very crude version of SRS, but I think it could help get users on board.
  • Name-Face Identification - A tool that helps users learn the names of all the contacts in their LinkedIn or Facebook friends lists. Forgetting the names of acquaintances is a common problem, and an SRS program is an ideal solution.

Beginning with a simple, self-curated deck like the ones described above would also help to avoid the problem of not having good content for first-time users.

Very interested to hear feedback on the above.

Comment author: Kindly 19 August 2012 08:52:21PM 3 points [-]

Congratulations; you've motivated me to decide I'll use some sort of SRS to remember the names of my students, the next time I have to TA a class (most likely in January).

I'll find something to use one way or the other, but if you manage to provide a working alternative by that time, I promise to use it.

Comment author: Vaniver 19 August 2012 11:28:38PM 7 points [-]

I used SRS to remember my student's names. They thought I had superpowers.

(I didn't have the TA facebook until after the first session, and so I introduced myself and shook the hand of every student, and then got the facebook and memorized the names with SRS before the second session. It might be similarly impressive to know their names before the first session without them telling you, but that runs the risk of seeming creepy.)

Comment author: gwern 19 August 2012 11:39:51PM 2 points [-]

I always thought that memorizing faces and personal details would be a good use of SRS, but I never really had any opportunity where it was worthwhile. Glad to hear it works!

Comment author: orthonormal 09 September 2012 04:16:28AM 4 points [-]

The July minicampers made an Anki deck for learning faces/names/bios before camp. We didn't get quite enough time to train properly, but it was still a massive help compared to learning 30 names in real time.

Comment author: gwern 08 February 2013 10:32:07PM 1 point [-]

How has it gone so far?

Comment author: Kindly 09 February 2013 01:30:44AM *  1 point [-]

The class I'm TAing has about 60 students in it; I see 40 or so regularly because one of the recitations is early in the morning and fewer people show up to it. Despite being rather lazy about reviewing, I can match around 90% of the photos to names, and make almost no mistakes going in the other direction (trying to imagine the photo given the name).

However, I think I only have around 50% success rate when faced with actual students. The photos I have are likely outdated (they're the photos taken for university IDs freshman year, and I suspect most of my students aren't freshmen) so that might be the problem. Another problem is that when learning the cards, I tend to remember facial expressions and incidental details like background color, even though I try not to; obviously this knowledge doesn't help in actual interaction with the students.

(I used Anki instead of MemStash.co because I couldn't get the latter to do what I wanted.)

Comment author: Micaiah_Chang 19 August 2012 08:23:38PM 2 points [-]

All three look promising. However, you might be pidgeonholing yourself by trying to go back to the "SRS as generalized learning tool". For most people, it would appear as if that's too abstract. You may be much better off focusing on the most generally appealing use case (The name-face ID one sounds the most promising; I can't imagine any people who weren't already self-motivated autodidacts using the first two). In fact, it might turn out to be much better than than the original Anki-as-service app; it appears to me that many more people view "oh god what's her name I just met her a week ago THINK" as a problem than "Oh, hm, it appears I've forgotten how to say 'praying mantis' in Japanese".

To extend the Name-Face identification concept, you could also add things such as people's birthdays, dates of important events such as anniversaries into it; although I'm not sure how many of those things aren't problems anymore because of calenders etc.

I can't comment too much on the "Twilio of online learning" idea; I don't know the interest level of online courses such as Udacity, Coursera and Codeacademy on something like that. Although I will warn that there's a real risk that it'll be treated as "just another complicated feature that I don't need to use" by the average student. But if you get a hardcore userbase who are happy with the product and willing to give feedback then you're in much better shape than trying to arbitrarily design for the "average" user.

Comment author: Dr_Manhattan 21 August 2012 02:18:52AM *  5 points [-]

Just a word of caution: this seems to be the idea almost every geek I know thought of doing (including myself, had a prototype going 4 years ago). You can draw some conclusions from this.

Comment author: wedrifid 21 August 2012 07:37:02AM 3 points [-]

Just a word of caution: this seems to be the idea almost every geek I know thought of doing (including myself, had a prototype going 4 years ago). You can draw some conclusions from this.

Another here. A group of four of us got as far as releasing a beta version of such a product.

Comment author: sinak 21 August 2012 04:38:30PM 2 points [-]

Interesting. Thanks for pointing this out. I'm not sure whether this is necessarily a bad thing, but it's good to know that it is a known geek archetype.

Comment author: ShannonFriedman 21 August 2012 04:46:02PM 5 points [-]

It implies that its a problem a lot of geeks have and that there's a market. Sounds like a problem that is easy to solve moderately well but has not yet been solved with excellence.

Comment author: sinak 27 August 2012 06:02:20AM 1 point [-]

Great feedback - thanks Shannon.

Comment author: sinak 03 October 2012 09:08:23PM 2 points [-]

Hey everyone - I made an initial version of this at the TC Disrupt Hackathon, see here: Memstash. It was build in 20 hours so it's obviously very very MVP, but we ended up winning trips to Paris as well as getting over 50k visitors and 5k signups (mostly via a Hacker News post).

I think that the response generally proves that there are definitely people interested in this sort of thing - but return traffic hasn't been great, which is understandable given how basic the functionality is. Anyway, would love to get your feedback on what I've built thus far.