An upvote doesn't quite seem adequate here, so: I thank you & Jonah for embarking on this project in the first place, and discussing your findings along the way; for keeping the project going in maintenance mode rather than pulling the plug on it entirely; and for laying out the rationale behind your decision to scale it down.
Good question!
A number of these were lurking in the back of our minds as potential issues (and in particular, they were raised by people who we talked to), but we didn't see them as decisive, and were still thinking in terms of there being a reasonable probability that we should continue until a few weeks ago.
I think that the main indicator ahead of time was that we strained to think of ways to disseminate our content. We were hoping that we would come up with ideas over time, or that opportunities would emerge organically as we explored, but it was concerning that we didn't have a clear plan of action.
I think that all of the reasons listed above are largely related to our not having gained more traction. If we had gotten more traction, we would have been more optimistic about getting funding, less inclined to think that developing technologies might be more promising, and wouldn't have needed to let the project brew in the background to get a better idea of its potential.
The process of our coming to our conclusion was gradual, based partially on indicators such as web traffic and the flow of incoming advisees, and partially based on conversations with people in related areas who generally expressed the view that what we were trying to do is very hard.
Should this cause me to update on the value provided by other "human capital development" EA orgs like 80K and CFAR?
Note: I haven't run this response by my collaborator Jonah, so he may not agree with some aspects of my answer.
I guess you're concentrating on "Reason #1: Downward update on social value"
As we mentioned, one reason for the downward update on social value for us was that we though we're already close to having plucked the low-hanging fruit with respect to receptive advisees. In other words, it was an update in the direction of "the target market is smaller than we thought." We haven't updated much in terms of our view of per capita impact.
I think that in the case of both CFAR and 80K, they are okay with a relatively small target market because they have a mechanism for generating huge returns per audience member: CFAR can charge people for workshops(something we can't do -- our advisees are young and cash-constrained students who would generally have been unwilling to contact us if they had to pay for it). 80K hasn't charged advisees (as far as I'm aware) but they get huge altruistic returns per capita by affecting people's career decisions in the immediate aftermath of the advising process.
We were aiming to affect people's general way of thinking and learning while they were students, perhaps several years before they join the workforce or make other direct contributions. Since there's a greater time lag between people being exposed to our ideas or advising and actually getting out into the workforce, there's a greater level of fade-out -- of the people whom we reach out to, only a small fraction will apply it to their lives a few years down the line. Given this damping factor, reaching out to a larger audience was more critical for Cognito compared to 80K and CFAR, and our pessimism about whether we can do so was one of the reasons we decided to switch to maintenance mode.
Another point is that both 80K and CFAR started out as parts of more established organizations/centers, therefore they have had more time to develop and grow before they needed to start making a case for funding them in isolation. (CFAR was a spinoff from MIRI and 80K is part of the Centre for Effective Altruism that also houses Giving What We Can). Cognito is a standalone operation, so we have less of a time runway to experiment and demonstrate feasibility. However, as I wrote in the post, if Cognito is indeed promising and valuable, we hope to see our traffic increase while we're in maintenance mode, and we hope that people we've advised so far continue to benefit from what we've done. So we can revisit the case for Cognito a few years down the line with more evidence to decide if it's worthwhile.
Not sure how much relevant overlap there is. CM seems focused primarily on education, and on spreading relatively available but difficult to compile info to people who can derive value from it.
CFAR is largely focused on much less available content, and on the developing of new content, bith focused at more general needs.
Not so sure about 80K, although it is much less education focused. It also seems to have an additional purpose - providing support for people to do good, as well as do be more competent.
Neither appear to have a funding gap
As far as Cognito being an income-generating proposition, you should approach it from the perspective not of CFAR complement, but instead look to the tutoring/test-prep/college-app-prep industry.
I know, I know, that's not the business you're looking to get into and it's not one you admire the impact of, but they have found ways to sell education (or something related to it) to families at all income levels. Thinking directly in terms of the near-term value proposition for the families, and how your services could be made to appeal to a family dropping $5K on test-prep, are the most reliable ways to get to sustainable profits.
If this turns your stomach, think of it in terms of how much better a value you can provide than that $5K of test prep!
This perspective also indicates how you should start thinking of marketing efforts.
This was Lukeprog's suggestion in the linked post, but they seem to have rejected it based on the difficulty of picking up motivated clients that way. Jonah phrased it as "teenagers and young adults are often rebellious and don't want to do what their parents tell them to". I think that this is something of an exaggeration - kids take test prep seriously after all - but it's true that the value of the service is more difficult for teenagers to see than something like exam tutoring.
Is it worth spending time mentoring kids that aren't interested and so won't get much out of the service, in order to get to the x% of students that will take it seriously? I would have thought so, but from this post it seems they've concluded that the answer is no, at least after you take into account the switch in focus (more personal advantage, less effective altruism) needed to get parents to pay for the service.
I'd think it wouldn't be too hard to have a selective set of clients. A single screening interview makes sense here, and might even help appeal to parents who want to think that their child is being treated as special -- which wouldn't be a bad thing, if the child actually was special.
As an SAT tutor, I've tried to impart life lessons along with bubble-filling lessons (on how to look at tests in general, how to hack studying, etc.), but the scope of those has necessarily been limited, both by the demands of the SAT and by the types of students I work with (I do more 1100-to-1500 transitions than 2000-to-2300).
Still, I feel that the "life lessons + advice for incoming college students" part of my work is much more valuable than the basic subject tutoring. And parents don't seem to object to my sharing their "turf" as far as lessons go. But this may be because I'm still young enough (20) to seem more like a high school student than a surrogate parent. And the life lessons were always a bonus in addition to SAT work; as a primary business, perhaps not so good.
Anyway, I'm sure the Cognito guys have considered all this -- I just hope that someone gives you the chance to pick up the work again in the future (and maybe hire me to help). Thanks for the Quora work, and good luck with your future endeavors!
I wanted to write something to you for a while and seeing that Cognito is scaling down means this might be the last opportune moment, so...
What do you think about situations where kids must take a single test to qualify for local colleges? My understanding is that in India, IIT admissions are basically based on a single exam. In China there is the gaokao. Are there any unique study strategies you would recommend there?
What about kids who aren't career-oriented? This is different from being smart or dumb. It's possible to be very smart and have getting into a good college or making money only at roughly the same priority as other goals like enjoying time with family and friends, exercise, and travel. How would a person with these more laid-back goals "optimize their life trajectory"?
What do you think about trading off how challenging a course is versus the grade you get? My guess is that you don't want to take courses you can't get A's in. Period. Your goal should be to get all or almost all A's in your subjects and only after that do you worry about taking AP courses, college courses, or whatever.
Finally, is there any advice you would give specifically to kids in developing countries who want to attend US colleges?
Thanks for these questions. We'll look into them over the next few months, and I'll update this comment to include updated thoughts or links to our content discussing these issues after we've written about these.
So far, we've thought of ourselves as content-oriented people, so we've used standard off-the-shelf software such as WordPress (for our main website and blog) and MediaWiki (for our information wiki). Part of the reason is that we wanted to focus on content creation rather than interface design, but part of the reason we've stuck to these is that we didn't think we could design interfaces. Once we've acquired more programming and design experience, we might be more open to the idea of designing interfaces and software that can meet particular needs of our target audience.
I think using the standard software was the correct choice, and IMHO it would be better to forget completely about making your own code. Even if you acquire more programming and design experience.
To explain why, just look at how much time it takes to add new features to LessWrong website, even the seemingly trivial ones like "display meetup announcements and dicsussion articles separately". Months and months.
It's because the web technologies are getting more and more complex over years. More and more features are invented, more and more functionality is considered "standard", because everyone else has it. But it all still needs a lot of work and maintenance! Unless you want to become a full-time software development company, that means unless you have a group of coders that only do the web interface and nothing else, just don't do this. Otherwise you risk spending hundreds of hours on mediocre product, where you could have better functionality by e.g. using some more advanced MediaWiki features.
If information is your product, I would suggest making the wiki page easier to navigate. For example the starting page seems like... well, honestly if someone would send me a hyperlink to that, or if google would show me that, I would just quickly close the browser tab.
For example:
College portal links to pages about college admissions, college selection, college academics, majors, online presence, and issues related to emotional and social well-being.
Uhm, why are the words "college admissions", "college selection" et cetera not hyperlinks? Is that some anti-SEO signalling? Which one of your customers would search for "portal"? Does anyone even use that word in this century? Instead of "a link to portal which hopefully contains the links to X, Y, Z", just provide the links to X, Y, Z directly. Pur a simple but colored picture representing "college", and clickable "college admissions", "college selection" etc below it.
Okay, maybe when you started doing this, you didn't see the wiki as your primary tool, so you gave it less attention. But maybe now it seems different. Make it easy to navigate, if that's your main product. You made a lot of good research -- great! But it also needs to be accessible, otherwise people will not read it.
Thanks for taking the time to comment and share your thoughts :).
I think using the standard software was the correct choice, and IMHO it would be better to forget completely about making your own code. Even if you acquire more programming and design experience.
I agree that if we are trying to do something with the same approximate functionality as MediaWiki or WordPress, it's best to use these standard softwares, even if we think our needs are somewhat different. So I am satisfied with the interface choices we made given our content focus.
What I had in mind was more that we might want to create some sort of customized, interactive website, software, or app, such as the ones we listed (Anki, Beeminder). Here, it's harder to use an existing, off-the-shelf solution. We'd have to code it ourselves. Now, at present, we don't have specific ideas for such interactive websites, but there may be scope for such websites in the future. If we have more programming experience, we'll be able to come up with and evaluate such ideas.
If information is your product, I would suggest making the wiki page easier to navigate. For example the starting page seems like... well, honestly if someone would send me a hyperlink to that, or if google would show me that, I would just quickly close the browser tab.
We will be looking into improving our main page. Thanks for the pointer there. In general, there's a conflict between keeping the front page sufficiently simple and including enough information on it so that people can easily find what they want. We've been looking regularly at our site's usage analytics to refine our decisions as to how to restructure the site, and we intend to continue doing so.
Sorry for the tone in previous message; I was tired, and wanted to deliver the information.
I'd ask it this way: What is your core mission? Seems to me it is "providing advice for students". So I'd recommend to focus 100% on this, and treat everything else as a shiny distraction. (Unless you want to change your mission to "develop an improved version of Anki or Beeminder". But you don't have the resources to do both, even if you had experience and ideas.)
I say this because it feels to me like you are not fully commited to the tools you have, and therefore you don't use them fully. For example, if you knew for sure that you are going to stay with MediaWiki, you would probably put more care into improving the wiki navigation. (But maybe you feel like "meh, this is just some temporary tool, we will use something better later", and then improving the details feels like a waste of time, if you expect it to change later.)
Seems to me that your current wiki structure uses too much introductions, too much redirection. For example: On the main page, you have a link to "What we offer and why". Why not put that into the main page? The page "What we offer and why" contains a link to "Learning portal"; the page "Learning portal" contains a link to "Our category on the benefits of learning particular subjects" (and also directly to math, algebra and economics; this part well-done!), and the page "Category:Subject learning benefits" contains a link to e.g. "Chemistry learning benefits". And the "Chemistry learning benefits" contains a text "We're still working on this page, so check back later!" (and a hyperlink to a useful topic). -- Uhm, seriously? It takes four clicks to discover the page about to chemistry, only to learn that you have almost nothing about chemistry?
On the other hand, I guess I understand how this all happened. This is a "top down" approach: you create a huge abstract structure where you want to fill the details later. This feels very high-status. But it's optimized for how you feel about it, not for the convenience of the reader. -- I suggest the opposite, "bottom up" approach. You have the valuable pieces of information (for example the external link to quora article about learning chemistry). That's the value you provide, and you want to navigate the reader there as easily as possible. So you build the navigation pages around the content you already have, not around the content you wish you had.
For example, one external hyperlink is not enough to make a separate page about chemistry. You probably have more such links for other subjects, so it makes sense to create a page "Advice about specific subjects", divide it to subjects by headers and subheaders, and put the links there. (In future, when you have more than 10 chemistry-related links, or perhaps if you have 5 chemistry-related links but you also provide a short summary below the link, then it's the right time to create a separate "Advice about Chemistry" page.) And your main page should link directly to the "Advice about specific subjects" page, because that's one of the main things you provide. There: Just two clicks, and the reader is reading the Quora article. There is the same information there as before, it's just easier to find.
Maybe it would be good to have one person researching the information you want to put on the web page, and another person who would maintain the structure of the web. One person to focus on "what" and other person to focus on "how".
Hi Villiam,
Thanks again. We really appreciate the detailed comment.
I discussed your comment with Vipul, but didn't have a chance to run the final version of my response by him.
(Unless you want to change your mission to "develop an improved version of Anki or Beeminder". But you don't have the resources to do both, even if you had experience and ideas.)
Vipul was making the general point that to date we've been focused on content, partially out of virtue of lacking technical skills, and that it could be more socially valuable create a platform. We don't have concrete ideas for what sort of platform we would make (in particular, we're not thinking of trying to make an improved version of Anki or Beeminder specifically). If we were to do this, it would constitute a significant shift in mission.
I say this because it feels to me like you are not fully commited to the tools you have, and therefore you don't use them fully. For example, if you knew for sure that you are going to stay with MediaWiki, you would probably put more care into improving the wiki navigation. (But maybe you feel like "meh, this is just some temporary tool, we will use something better later", and then improving the details feels like a waste of time, if you expect it to change later.)
Until very recently, we were thinking in terms of the wiki (once fleshed out) being our core offering. I agree that our wiki navigation has much room for improvement – we were more focused on content creation, outreach and fundraising. Now that we're shifting to maintenance mode, improving the wiki's navigation is higher priority. However, we expect that people will find our pages through searching for a particular topic via Google more than through the portal.
Seems to me that your current wiki structure uses too much introductions, too much redirection.
Until a month ago or so, we had most of our pages linked to from the front page. It was starting to seem too cluttered, so we switched to a more modular design. Now that we've decided not to produce a lot of pages in the near future, it may be better to switch back, or adopt some compromise.
Back in December 2013, Jonah Sinick and I launched Cognito Mentoring, an advising service for intellectually curious students. Our goal was to improve the quality of learning, productivity, and life choices of the student population at large, and we chose to focus on intellectually curious students because of their greater potential as well as our greater ability to relate with that population. We began by offering free personalized advising. Jonah announced the launch in a LessWrong post, hoping to attract the attention of LessWrong's intellectually curious readership.
Since then, we feel we've done a fair amount, with a lot of help from LessWrong. We've published a few dozen blog posts and have an information wiki. Slightly under a hundred people contacted us asking us for advice (many from LessWrong), and we had substantive interactions with over 50 of them. As our reviews from students and parents suggest, we've made a good impression and have had a positive impact on many of the people we've advised. We're proud of what we've accomplished and grateful for the support and constructive criticism we've received on LessWrong.
However, what we've learned in the last few months has led us to the conclusion that Cognito Mentoring is not ripe for being a full-time work opportunity for the two of us.
For the last few months, we've eschewed regular jobs and instead done contract work that provides us the flexibility to work on Cognito Mentoring, eating into our savings somewhat to cover the cost of living differences. This is a temporary arrangement and is not sustainable. We therefore intend to scale back our work on Cognito Mentoring to "maintenance mode" so that people can continue to benefit from the resources we've already collected, with minimal additional effort on our part, freeing us up to take regular jobs with more demanding time requirements.
We might revive Cognito Mentoring as a part-time or full-time endeavor in the future if there are significant changes to our beliefs about the traction, impact, and long-run financial viability of Cognito Mentoring. Part of the purpose of "maintenance mode" will be to leave open the possibility of such a revival if the idea does indeed have potential.
In this post, I discuss some of the factors that led us to change our view, the conditions under which we might revive Cognito Mentoring, and more details about how "maintenance mode" for Cognito Mentoring will look.
Reason #1: Downward update on social value
We do think that the work we've done on Cognito Mentoring so far has generated social value, and the continued presence of the website will add more value over time. However, our view has shifted in the direction of lower marginal social value from working on Cognito Mentoring full-time, relative to simply keeping the website live and doing occasional work to improve it. Specifically:
What might lead us to change our minds: If we continue to be contacted by large numbers of potentially high-impact people, or we get evidence that the advising we've already done has had significantly greater impact than we think it did, we'll update our social value upward.
Reason #2: Downward update on long-run financial viability
We have enough cash to go on for a few more months. But for Cognito Mentoring to be something that we work full time on, we need an eventual steady source of income from it. Around mid-March 2014, we came to the realization that charging advisees is not a viable revenue source, as Jonah described at the end of his post about how Cognito Mentoring can do the most good (see also this comment by Luke Muehlhauser and Jonah's response to it below the comment). At that point, we decided to focus more on our informational content and on looking for philanthropic funding.
Our effort at looking into philanthropic funding did give us a few leads, and some of them could plausibly result in us getting small grants. However, none of the leads we got pointed to potential steady long-term income sources. In other words, we don't think philanthropic funding is a viable long-term revenue model for Cognito Mentoring.
Our (anticipated) difficulty in getting philanthropic funding arises from two somewhat different reasons.
What might lead us to change our minds: If we are contacted by a funder who is willing to bankroll us for over a year and also offer a convincing reason for why he/she thinks bankrolling us is a good idea (so that we're convinced that our funding can be sustained beyond a year) we'll change our minds.
Reason #3: Acquisition of knowledge and skills
One of the reasons we've been able to have an impact through Cognito Mentoring so far is that both Jonah and I have knowledge of many diverse topics related to the questions that our advisees have posed to us. But our knowledge is still woefully inadequate in a number of areas. In particular, many advisees have asked us questions in the realms of technology, entrepreneurship, and the job environment, and while we have pointed them to resources on these, firsthand experience, or close secondhand experience, would help us more effectively guide advisees. We intend to take jobs related to computer technology (in fields such as programming or data science), and these jobs might be at startups or put us in close contact with startups. This will better position us to return to mentoring later if we choose to resume it part-time or full-time.
Knowledge and skills we acquire working in the technology sector could also help us design better interfaces or websites that can more directly address the needs of our audience. So far, we've thought of ourselves as content-oriented people, so we've used standard off-the-shelf software such as WordPress (for our main website and blog) and MediaWiki (for our information wiki). Part of the reason is that we wanted to focus on content creation rather than interface design, but part of the reason we've stuck to these is that we didn't think we could design interfaces. Once we've acquired more programming and design experience, we might be more open to the idea of designing interfaces and software that can meet particular needs of our target audience.We might design an interface that helps people study more effectively, make better life decisions, or share reviews of courses and colleges, in a manner similar to softwares or websites such as Anki or Beeminder or Goodreads. There might also be potential for a more effective online resource that teaches programming than those in existence (e.g. Codecademy). It's not clear right now whether there exists a useful opportunity of this sort that we are particularly well-suited to, but with more coding experience, we'll at least be able to implement an idea of this sort if we decide it has promise.
Reason #4: Letting it brew in the background can give us a better idea of the potential
If we continue to gradually add content to the wiki, and continue to get links and traffic to it from other sources, it's likely that the traffic will grow slowly and steadily. The extent of organic growth will help us figure out how much promise Cognito Mentoring has. If our wiki gets to the point of steadily receiving thousands of pageviews a day, we will reconsider reviving Cognito Mentoring as a part-time or full-time endeavor. If, on the other hand, traffic remains at approximately the current level (about a hundred pageviews a day, once we exclude spikes arising from links from LessWrong and Marginal Revolution) then the idea is probably not worth revisiting, and we'll leave it in maintenance mode.
In addition, by maintaining contact with the people we've advised, we can get more insight into the sort of impact we've had, whether it is significant over the long term, and how it can be improved. This again can tell us whether our impact is sufficiently large as to make Cognito Mentoring worth reviving.
What "maintenance mode" entails
Conclusion and summary (TL;DR)
We (qua Cognito Mentoring) are grateful to LessWrong for being welcoming of our posts, offering constructive criticism, and providing us with some advisees we've enjoyed working with. We think that the work we've done has value, but don't think that there's enough marginal value from full-time work on Cognito Mentoring. We think we can do more good for ourselves and the world by switching Cognito Mentoring to maintenance mode and freeing our time currently spent on Cognito Mentoring for other pursuits. The material that we have already produced will continue to remain in the public domain and we hope that people will benefit from it. We may revisit our "maintenance mode" decision if new evidence changes our view regarding traction, impact, and long-run financial viability.