SpaceX’s goal is “settle Mars”, its MVP was “build a low-earth orbit launch system”, and sub-MVPs of that (experiments to validate the highest-uncertainty thing they needed to validate) consisted of things like engine test firings.
Using SpaceX as an example, there are a lot of sub-MVPs that they need to build before they actually have an MVP that they can bring to market and see if they get traction. And so it would take them a long time before they actually even have an MVP. It seems to me that this conflicts with conventional wisdom, because conventional wisdom says that if you haven't brought your MVP to market and tested it in, say, six months, that you aren't being lean enough. But with SpaceX it seems that it may not really be possible to do. What do you think?
The Lean Startup by Eric Ries explains “MVP” as a broader concept of a tool to obtain validated learning. Getting traction is a special case of learning for companies whose most pressing question is whether people really want what they’re building. This special case encompasses most startups. But SpaceX had less doubt about demand for wildly cheaper launches and more doubt about technical feasibility, so an engine test firing was a kind of MVP for them.
I see. That makes sense.
I'm still not sure we addressed the question of whether there are times when MVPs inherently take a long time to build though. What do you think about that question? Sorry if you addressed it and I'm not understanding.
You can always chart a path that maximizes your rate of validated learning / de-risking your biggest risks. Like if you’re building something huge, build the most uncertain parts first. Yes sometimes a project is inherently slow and capital-intensive but that’s rare and most people who think their startup is that exception are actually just being sucked into the most common fatal attractor.
Whenever I ask any employee or contractor to do a project, I always say show me the first half-baked sign of progress after 1 day of work. I never want someone to do a bunch of work on a project that has any kind of risk without a quick feedback loop. I try my best to avoid trusting that it’ll all work out sometime later.
I notice you're talking about market size, but not about marginal value to the median member of that market. I no longer play poker seriously, but I still know a number of pros and some authors/publishers/coaches that were reasonably successful. When you say:
if you're a poker player and are actually trying to get better, that you should have some sort of poker software to study with
You're making a few major errors. Assuming that very much of your potential audience is all that dedicated to getting better is one - poker survives on optimistic idiots, not on smart, patient, long-term thinkers. There are very few players who win consistently, and most of those don't spend that much effort studying.
Assuming that studying is the main part of poker success is also a mistake. Most profit comes from identifying and exploiting opponent mistakes, not in optimizing your own play beyond a certain competence. It's very hard, even for very good players, to formally model the kinds of mistakes they profit from, so very hard to imagine that software calculations help them improve their exploitative play.
Assuming that software is a critical part of studying is another, and that your paid software is the one people would choose even worse. There's plenty of pretty decent free software and many reams of printed study material. And once you reach a certain level of competence, you get paid to study by actually playing hands and then reviewing hand histories (often with the aid of free software) afterward. The exception is live helpers (aka "cheats") for online games - those that scrape the screen and sometimes even click buttons for you, based on shared player databases. I'm out of the current market, but these were big business a decade ago.
The nice thing is you have a built-in revenue method: if the software is really all that great, you should use it to make money at poker, rather than selling it to your opponents.
I agree that poker survives on optimistic idiots. My thoughts are that we can divide players up into three buckets: pros, somewhat serious players, and fish ("optimistic idiots"). Fish haven't really spent any time ever trying to learn how to play the game. Or if they have, it's only a book or two. Somewhat serious players have read a bunch of books and study/review hands maybe once a week or so. And pros are pros. I figured that the number of fish are in the millions, somewhat serious maybe 100k or so, and pros, maybe a few thousand. I say this because you seem to be dividing the population into pros and fish, whereas I add in the somewhat serious players.
I think the big mistake I made was that I assumed that these somewhat serious players all somewhat consistently use poker software. I know that the "optimistic idiots" clearly don't, but I figured that the somewhat serious players do.
Assuming that studying is the main part of poker success is also a mistake. Most profit comes from identifying and exploiting opponent mistakes, not in optimizing your own play beyond a certain competence. It's very hard, even for very good players, to formally model the kinds of mistakes they profit from, so very hard to imagine that software calculations help them improve their exploitative play.
I would disagree with that. I think software can really help you improve your exploitative play. This video is an example. Another example is looking at different flop textures and seeing which ones you'd generate enough folds on to profitably cbet.
There's plenty of pretty decent free software
There are some free equity calculators, but they are limited. And from what I understand, there are no free range analysis tools like my Hit Calculator or Flopzilla, and range analysis tools are important for a lot of things, like getting a sense of how much fold equity you have.
I've also heard a lot of coaches complain about students who pay them $100+/hr not do the homework that they are assigned. The coaches beg and plead, but the students just don't want to do the work.
Indeed this is a common experience among music teachers. A singing teacher I know, who teaches middle-aged amateurs, says they never practice what they're told to, and pretty much come along to (a) do a little expensive supervised practice and (b) have a nice chat about anything.
In contrast I started singing lessons a few years ago, and did what a pro singer friend advised me: recorded the lesson. Then I listened back to the whole thing afterward, and did several hours practice before the next lesson, doing everything the teacher had told me.
At my second lesson, the teacher said I had got much better in just a week - like, as if I'd had months of lessons in between - and he couldn't quite believe it. Presumably almost no-one remembers most of what the teacher told them, let alone practices it for hours between each lesson, like you're meant to.
The same is true of physiotherapy. I once had some treatment for a sprained ankle. The physiotherapist gave me some daily exercises to do at home, and explained that if I did them then they would work, but in practice almost no-one ever does them. (Or they do them once or twice then give up.)
This may suggest a point even more basic than More Dakka (which IIRC said, if something doesn't seem to work, or only works a bit, try doing it more): if you don't do something at all, then it definitely won't work.
How about this for affiliate outreach:
I’m not sure your software is used during an online game like that, but my point is you want to really spoon feed them into realizing the value of your software to get them to actually care about it and use it.
I spent a good amount time trying to do this sort of stuff. I found that people weren't too responsive. But perhaps with even more persistence they would be. And it sounds like a good thing to do in general.
They [Y Combinator] talk a lot about focusing heavily on the product, as opposed to marketing it. I at least get the impression from them that making something people want is what it's all about, and that if you manage to do so, you'll have success.
Well, it’s an important fact that most startups fail by not making something people want, so “make something people want” is the best 4-word advice, but you also need a customer acquisition mechanism and it can be fatal if you can’t find one.
In fact I think difficulty with customer acquisition, i.e. customer acquisition costs tanking the business’s unit economics, is the second biggest startup killer after not making something people want.
Paul Graham made the point that these days when you want to judge a startup idea, you’re actually judging the idea pair (what you’re doing that people want, how you acquire customers).
Paul Graham made the point that these days when you want to judge a startup idea, you’re actually judging the idea pair (what you’re doing that people want, how you acquire customers).
Huh, I didn't realize that. Thanks for pointing it out. I read most of his essays, and a lot of YCs stuff more generally, but I don't recall much talk about customer acquisition. Anything come to mind that you can link me to?
http://www.paulgraham.com/ds.html
Tons of amazingly insightful points packed into a small essay. I was paraphrasing the “Vector” section.
Another good concept from that post is the “Collison Installation”. I also like to call it manhandling your traction.
The Poker subreddit only has 100k subscribers, and it’s a much bigger commitment for people to buy poker software than read a subreddit. So it seems plausible that the total market is only 1k users, but you never know.
Founders shouldn’t worry too much about market size, just focus on getting early users and growing the user base at some rate like 10%/month. For example, Uber has grown beyond the total market size of taxis. Peloton has grown beyond what many people might have guessed was the market size of people who want a home exercise bike. Microsoft’s original market was BASIC programmers.
To be able to direct your effort toward where it will make a difference, you need to get a feedback loop going where you give people value and observe the consequences. You made a quality piece of software that could give people value in theory, but in practice value usually can’t be transmitted without an intense manual effort at sales/delivery/onboarding. The way to give people value is to grab specific people and manually stuff your value into them.
Most startups have to manually hunt down their initial traction one customer at a time. It almost always takes a long time to get momentum going where you don’t have to spend manual effort and resources to acquire the next incremental customer.
The Poker subreddit only has 100k subscribers, and it’s a much bigger commitment for people to buy poker software than read a subreddit. So it seems plausible that the total market is only 1k users, but you never know.
Well, my thinking was that only a fraction of the somewhat serious players are on the subreddit, so if only 10k of the 100k on the subreddit are somewhat serious, maybe we can multiply by 5-10 (or hopefully more) to account for the fact that not all of the somewhat serious players are on the subreddit.
Now my thinking is to assume that only like 100 of the 100k are somewhat serious, not 10k.
Founders shouldn’t worry too much about market size, just focus on getting early users and growing the user base at some rate like 10%/month. For example, Uber has grown beyond the total market size of taxis.
That makes sense, but only if you have a plausible path to growing the size of the market.
The way to give people value is to grab specific people and manually stuff your value into them.
This belief has been growing stronger and stronger in my mind over time. And it seems like a really important thing for founders to understand.
It makes me feel really uneasy though, because I don't want to be spammy. What are your thoughts on that?
BTW, one of my unfleshed ideas is to have a platform that connects self-proclaimed early adopters with early stage founders. Maybe the early adopters would get access to products discounts or something for being on the platform, and early stage founders would have to pay to be on the platform. I plan on fleshing this out and writing about it some time in the future. It easily could turn out to be a bad idea; just thought I'd mention it.
That makes sense, but only if you have a plausible path to growing the size of the market.
I can imagine that someone could make poker software that makes poker more fun and addictive, pulling more people into learning to play well who otherwise wouldn’t have bothered. Peloton got regular people interested in indoor cycling. Codecademy got regular people learning to code. It’s possible to demonstrate signs of this potential just by having passionate early users and a sturdy growth rate starting from a small baseline.
And [grabbing specific people and manually stuffing value into them] seems like a really important thing for founders to understand.
Ya my whole blog bloatedmvp.com is about founders not understanding this.
It makes me feel really uneasy though, because I don't want to be spammy. What are your thoughts on that?
Also, when you narrow down what your MVP is to the point where it’s almost by definition an incremental improvement in one specific person’s life, you should feel more confident to attempt to do a Collison Installation on that person. If it doesn’t work out, they should be able to understand why you were convinced this would be worth their time, even if you were wrong.
BTW, one of my unfleshed ideas is to have a platform that connects self-proclaimed early adopters with early stage founders. Maybe the early adopters would get access to products discounts or something for being on the platform, and early stage founders would have to pay to be on the platform. I plan on fleshing this out and writing about it some time in the future. It easily could turn out to be a bad idea; just thought I'd mention it.
I dunno, try giving a specific value prop story (https://www.lesswrong.com/posts/iMQmfgieRKY62MC3z/the-power-to-judge-startup-ideas) illustrating why it’s much better than Product Hunt.
I can imagine that someone could make poker software that makes poker more fun and addictive, pulling more people into learning to play well who otherwise wouldn’t have bothered.
Yeah, that's a good point, but I'm not sure that it applies to every product. So then, it still seems to me that you should try to think about whether it is plausible to "grow the pie".
What do you think? I lean slightly towards thinking that you disagree, because you initially said that founders shouldn't worry too much about market size. If so, what is the thought process behind that? Because you think there's a route to "grow the pie" for any product? Because at an early stage your goal is to be nimble and figure out what direction to run in?
Ya my whole blog bloatedmvp.com is about founders not understanding this.
Awesome, I'll check it out!
Your own ego will already bias you toward not putting yourself out there enough.
Huh. I thought it would be the opposite. That people have egos that are too big and they think they're more important than they actually are. Or are you saying that I personally don't have a big ego, because I'm having this conversation?
Also, when you narrow down what your MVP is to the point where it’s almost by definition an incremental improvement in one specific person’s life, you should feel more confident to attempt to do a Collison Installation on that person. If it doesn’t work out, they should be able to understand why you were convinced this would be worth their time, even if you were wrong.
That registered really well with me, thank you! I agree! I do find myself thinking this a lot, that I'm doing them a favor by reaching out.
I dunno, try giving a specific value prop story (https://www.lesswrong.com/posts/iMQmfgieRKY62MC3z/the-power-to-judge-startup-ideas) illustrating why it’s much better than Product Hunt.
If I'm an early stage founder and I launch on Product Hunt, I may not get any attention. Whereas with my idea, the founder would be able to browse the email list of early adopters and reach out to the ones who meet the target demographic.
With Product Hunt, you have to actually launch something to be seen. What if you want to chat with users before you launch? What if you launched already and want to do user research? What if you releases a new feature, or changed pricing, and want to send a sales email?
With Kickstarter, you can put yourself out there before you launch, but you can't actually reach out to users, from what I understand. I think there's value in having a list of user email addresses that you can browse through. Basically, warm leads, because they signed up, and because you can see demographic info and other user profile stuff.
Note: I think it'd be pretty necessary to limit the amount of emails that founders can send out. Otherwise early adopters would get totally blasted and not want to sign up. So maybe the email addresses wouldn't actually be visible, and you'd have to message them through the platform, and the platform would limit how many messages you can send.
What do you think? I lean slightly towards thinking that you disagree, because you initially said that founders shouldn't worry too much about market size. If so, what is the thought process behind that? Because you think there's a route to "grow the pie" for any product? Because at an early stage your goal is to be nimble and figure out what direction to run in?
It's good to think about the market and where you can grow it. Paul Graham asks "What Microsoft is this the Altair Basic of?" If you can tap into early passionate users and an exponential growth rate of revenue (or at least usage), and the business is inherently scalable (i.e. not a local diner), then it's hard to be confident that the growth rate will slam into a wall in the near future, but it's also then quick to test whether you can keep the exponential curve going or not. So the question of whether the idea is good approximately reduces to the question of whether you can get a traction and growth curve going, and that's usually the highest priority thing to focus on proving. Sometimes you might convince yourself that the market has very little potential even if you do get early passionate users with a good early growth rate, but it's a rare outcome and often worth giving it a shot and following where it goes.
Huh. I thought it would be the opposite. That people have egos that are too big and they think they're more important than they actually are. Or are you saying that I personally don't have a big ego, because I'm having this conversation?
Salespeople typically deal with flakiness and rejection all day. A big part of their job is to relentlessly hound their prospects. I did some sales outreach the other day trying to get affiliate partnerships and it doesn't feel good to be reaching out to a bunch of people and getting ignored and turned down. But on the other side of the table, when salespeople pester me, I may be slightly annoyed at the spam but I understand they're playing a game and I don't hate them for it.
Re the "early adopters email list" idea:
Try making the Value Prop Story more specific by identifying an actual specific person on one or both sides of the 2-sided business, and explain why specifically they need a new email list beyond what exists now.
Are you specifically an example of the "early adopter" you have in mind? Am I an early adopter? I don't know because that's a vague term. I'm curious to skim descriptions of e.g. a cool new database, but not happy to adopt it. I'm unlikely to early-adopt a new beauty product.
I'm guessing that you launching your poker software is an example of a target user on the looking-for-early-adopter-feedback side of the equation. Is "email list for early adopters" really that big of a win for what you needed? It seems like the Poker subreddit and other poker communities should already have had enough leads you could email about testing your product.
Your original Value Prop Story should probably already about the kind of person you have access to in your life, depending on what field you're working on, so I'm skeptical that what founders really need is an email list of early adopters. And if they do just need that, there's already Product Hunt, Hacker News, startups reddit, startupschool.org, nugget.one, tons of incubators/accelerators who offer advising, usertesting.com, etc.
Re the "early adopters email list" idea:
My thinking is that by signing up as an early adopter, it wouldn't be industry specific, you'd just be saying that you're interested in being contacted about new products and stuff.
Yeah, I myself am an example of someone who would sign up as a founder on this platform. With my poker app, and with future projects. With the poker app, I found that the poker subreddit, other poker forums, in person networking, and cold emails didn't amount to enough leads.
I know that sounds implausible. Maybe it's an uncommon experience. I think the success of this platform idea we're talking about depends a good amount on the answer to that question, because if founders are currently doing fine with customer outreach, then it wouldn't really be solving a problem for them. But my impression is that a) many founders struggle to get in touch with these sorts of early adopters, and b) even if they are successfully getting in touch with some, they'd still want more. Of course, this is a hypothesis that could be validated without actually building anything, eg. by talking to founders.
Your original Value Prop Story should probably already about the kind of person you have access to in your life, depending on what field you're working on, so I'm skeptical that what founders really need is an email list of early adopters.
That's understandable. I don't feel particularly confident in the idea.
With the poker app, I found that the poker subreddit, other poker forums, in person networking, and cold emails didn't amount to enough leads.
[...] I think the success of this platform idea we're talking about depends a good amount on the answer to that question, because if founders are currently doing fine with customer outreach, then it wouldn't really be solving a problem for them.
No question that acquiring leads/customers is a huge problem for a huge number of businesses. Google and Facebook are worth a combined $trillion because they sell a solution to that problem.
I just don't think an "email list of early adopters" would have helped you acquire leads more than you can already help yourself by using existing forums that aggregate poker-lovers plus existing forums that aggregate people who are open to testing startups.
You probably think whoever currently follows Product Hunt is a good candidate to be on your email list, if only Product Hunt had a section for pre-launch companies. Well, it's not hard to look up 100 PH users' emails and blast them an email to take a look at your poker bot :) you can target PH people who have liked poker-related products. This would constitute an MVP of testing your idea, even though on the surface you might not imagine it as such.
But my impression is that a) many founders struggle to get in touch with these sorts of early adopters, and b) even if they are successfully getting in touch with some, they'd still want more. Of course, this is a hypothesis that could be validated without actually building anything, eg. by talking to founders.
Before talking to other founders, a good first step is to convince yourself you would be a passionate user of this, which I'm skeptical about. E.g. would you pay $50/month for access? (This $50 threshold is a generally-applicable stronger version of the Value Prop Story test.)
I definitely agree about talking to founders and prospective early adopters as an MVP. I think it's a great example of what you talk about on your blog, about how and MVP doesn't actually have to be a product.
I myself would definitely have paid for it. Somewhere in the ballpark of a couple hundred dollars. Possibly more if I actually was making money and growing. I spent a lot of time struggling to get in touch with users, at various points, for various reasons, and if the email list solved that problem for me, that'd be hugely valuable.
Again, in theory I should be able to easily get in touch with people by messing around in different poker communities, but that just doesn't seem to actually work for me in practice. I have a great example. I posted about my app on Reddit at various points. Lots of people commented saying nice things. Some even commented to say that they would pay for it when it's on the market, and to let them know. Yesterday I took some time and DMed everyone. I sent out 24 messages, but only got one reply. And that is for people who have already said they love my app.
Well it's not surprising that someone on a forum who said they love your app weeks or months ago is probably not super likely to respond to a DM now. There's just a general level of flakiness that suits all of us to partake in to some degree.
How about DMing 10 users who have commented in the poker reddit in the last 48 hours and asking if they'd do you a favor and answer questions about their poker interest on a video call? I bet you'd get a couple takers. Until I understand why your specific situation is best helped by an email list of early adopters, I'm skeptical that your value prop generalizes the way you think it does.
That makes sense about them not being too responsive to this particular DM. I predict that I won't get many more responses over the next few weeks though.
I have actually spent a good amount of time DMing people asking for that sort of stuff, and also just posting saying that I'm looking for people to video chat with. It just hasn't really worked for me. That's why I feel like the email list idea is interesting, because I myself have had the problem of not being able to get in touch with users. I do place more weight in your intuition though, because I'm just one data point and you seem to have more experience with startups, so if you feel skeptical then I shift that way a lot and don't expect it to be successful. Although I'm not so skeptical that it doesn't seem worth talking to some founders though.
Can you link to specific posts you've made in relevant communities for poker and/or startups asking for feedback or discussion about your ideas? If they get 1k+ views then I'd expect at least one or two leads to come out of such a post.
Here are the four that actually got some attention:
I was thinking about it some more last night. It feels like I DMed a lot of people, but the number isn't quite in the hundreds, I don't think. Maybe it's just a numbers game and you really actually have to reach out to hundreds and hundreds and hundreds of people. But at least on forums, that could be tough, because you could get flagged/banned for spamming.
I think many of the people who were commenting in your threads would have been up to let you interview them in a video chat if you’d DM’d them at the time. I don’t think it should take more than 10 DMs to get a video interview with this kind of approach.
Perhaps. At the time what held me back was not wanting to be spammy, but as we talked about, I think I was being too shy. Next time I'm going to jump at those sorts of opportunities, so thanks for helping me realize that, amongst other things.
More evidence that it takes a lot to actually get people to look in your direction, and that customer acquisition is hard.
People are lazy and irrational
Yes, we all are to some degree. For example, I was a self taught pianist for 15 years before I finally got rational enough to hire a piano teacher because I predicted it would vastly increase my learning rate, and sure enough there was a ton of low hanging fruit I learned. It’s just that seeking and hiring the teacher had friction/hassle and risk of not working out, so didn’t feel appealing.
If the top bestselling poker book sold 100k copies total, that might be 10k copies/yr, meaning the book’s profits are under $100k/yr and the author’s cut is less.
These sorts of books sell for around $50. $100k/yr profit at 10k copies/yr would imply they only make $10/book. I would have thought profit margins would be better. But that's a really good point about publisher/marketing costs.
Anyway, your overall point that the top poker book isn't making millions for the author is well taken. It seems like a pretty strong signal that I shouldn't really be expecting to make millions either. I wish I had thought about this earlier, thanks.
Previous posts:
I finally made the decision to call it quits. Now I think it would be a good time to reflect on my experiences and see if I could learn something from them.
Market size
As I talked about in the previous post, initially, I thought that the market size was hundreds of thousands of users, and that I could make something like $100/user. After talking to people in the industry, I now believe that the market is more like 5-10k users, and not all of them are willing to pay $100.
This is incredibly important! Going after a $20M market is very different from going after a $200k one. If I knew it was the latter in the beginning, I wouldn't have pursued this as a business. Spending 2+ years for the chance at making maybe $200k just isn't worth it, given the inherent uncertainty with startups, and the alternatives of pursuing a startup with a higher upside, or getting a job that pays approximately that much but does so with 100% certainty.
So what happened? Where did I go wrong? Let's see. This was (roughly) my initial logic:
Maybe I really underestimated the divide between passive + free and active + paid. Watching a YouTube video is something that is passive. You sit there and consume information. Using poker software is something that is active. You have to sit and think and mess around with numbers. Watching a YouTube video is free. Poker software costs money.
But then what about the existence of poker books? There around 500 poker books on the market. The top ones get up to 100k sales, and many others get in the tens of thousands, I think. Maybe it's that with books, someone is telling you what the right answers are, but with software, you have to figure out the right answers yourself.
Anyway, I think the bigger point is that I should have found people in the industry and asked them about the market size. I started doing that towards the end of my journey, but I should have done so from the beginning. People in the poker world all seem to have a pretty solid idea of what the market is really like. Why screw around trying to figure it out myself with these questionable proxies when I could just ask the people who actually know?
I really can't emphasize enough how huge this is. It would have only taken a few hours, and it would have saved me so much time.
So why didn't I go out and talk to people in the industry? I'm not quite sure. I think a part of it was that I didn't actually feel like I had to. It seemed pretty clear to me that the market was big, so I was more concerned with making the product awesome.
Another part of it is that I didn't see it as an option to talk to people in the industry. Because why would they want to talk to little ol' me? They're basically B-list celebrities, in some sense. They've written books. Tons and tons of people know them. Don't these people have hundreds of fan emails every day that they never respond to? That was my thinking in the beginning. Now, I've come to realize that they're just people, and that they're often happy to chat and provide advice.
It also would have been good if I had access to the actual financial data of my competitors. But none of them are public companies. Does that mean this isn't an option, or are there still ways?
I came across one cool approach last night. If you don't have access to their financial data, you could look at how many employees they have, and multiply by something like $125k or $200k. Something in that ballpark should give you an idea of their revenue. My competitors, from what I can tell, are all working solo. So that is a sign that they aren't making millions and millions of dollars. Not definitive, but definitely points in that direction.
Another interesting option is to actually call your competitors and talk to them! Eg. you could pretend to be a prospective employee, and in that conversation start asking about revenue and stuff. I'm not sure how I feel about that sort of deception though.
Here's a closing thought for this section: the world isn't that big. For the majority of my journey, my thoughts on the market size for poker has been, "I'm not sure exactly how big it is, but it's big enough." The world is just such a big place, and so many people play poker. The market just has to be huge. Now I realize that this isn't true.
"Just another month or two"
As I explain in the first post, for a very long time, I kept thinking to myself:
This just kept happening over, and over, and over again. It's crazy.
And each time it happened, it felt like this was the time it really would only be another month or two. I was wrong last time, and the time before, and the time before, and the time before... but this time... this time I'll be right.
Wow. Articulating it like that really helps put things in perspective. I need to diagnose myself with a chronic case of the planning fallacy. I need to do a better job of adjusting in the opposite direction. I have a tendency to be overconfident, so I need to adjust in the other direction, and be less confident. That's what you have to do with known biases: try to adjust in the other direction.
And with the planning fallacy in particular, there's a known cure: the outside view. Don't try to reason from the ground up. Look at how long similar things have taken in the past. Maybe use the reference class of "times I've thought it'd only be another month or two".
Man, that makes me laugh. "Times I've only thought it'd be another month or two." Ha! That reference class is full of miscalculations, so it's pretty clear that I need to adjust in the other direction pretty hard.
I say all of this stuff, but I still worry that I'm going to make the same mistakes again.
Agility
Check out this excerpt from the first post in this series:
And I have this little voice in my head saying:
And then I say in response:
I'm going to resist the temptation to respond to this right now. Right now I just want to tell the story. The story as it actually happened. But I do want to say that there were a lot of voices swimming around in my head questioning what I was doing.
I said in that post that I'm going to resist the temptation to respond to it. Here's where I do get to respond to it.
Here are my general thoughts about the whole lean startup thing:
1. I think the essence of it is to ruthlessly avoid spending time on things unnecessarily. Eg. you don't need to spend a week building a fancy navigation menu before you even know whether people want your product. I heartily, heartily support that message.
2. It can be tempting to think that there are no quick experiments you can run. This is usually wrong. Think more carefully, try to isolate the component assumptions, and get creative.
I definitely could have done a better job with (2).
3. But sometimes there truly aren't any quick experiments you can run. Eg. SpaceX. How is SpaceX supposed to build a quick MVP? (Well, there are some things they could do, but I don't want to get distracted from the central point that some hypotheses inherently just take a long time to validate.)
Now that I'm finished with Premium Poker Tools and am reflecting on what I've learned, it's time to add a fourth general thought.
4. If you are in a situation where (3) applies and are going to spend... I don't know... two years and three months testing a hypothesis... then the upside damn well better be worth it!
Pretty obvious. If the risk is big, the reward needs to also be big.
With Premium Poker Tools, the reward never big enough. It just wasn't. This is no SpaceX.
Even if my initial ideas about the market size were correct, and there was the potential to make $10-20M, that still isn't enough to justify spending so long testing a hypothesis.
However, I don't think it's that simple. This issue is very tangled up with the planning fallacy stuff I talked about in the "Just another month or two" section. I never actually decided to spend 2+ years on this.
Show me the money
https://www.youtube.com/watch?v=FFrag8ll85w (I had to)
I have another objection to the above section. I wasn't testing a hypothesis the whole time. No. About a year into it, I already had market validation. That's right. I already had validation.
I'm not sure when exactly I would say it occurred, but a big thing is when I posted to Reddit and someone asked where they can donate. Another big thing is seeing one of the most popular poker players in the world using my app. Another was having a bunch of people tell me that they would pay for it. Another is having random people email me thanking me for building the app and telling me how great it is.
Now, I know they say talk is cheap. I know how startup people always want to see traction. Users. Money. Growth. I didn't have any of that. But I did have all of the other stuff in the above paragraph! People said so many nice things to me. Sure, conventional wisdom might say that the above paragraph isn't enough, and that you need real traction...
But I'm a Bayesian! I'm better than that! Saying that you need actual traction is like how the scientific community waits too long before accepting something as true. Being a Bayesian, I can update in inches. I can be faster than science. I can be faster than conventional startup wisdom.
Well, that's how I used to think anyways. Now I run around my apartment with my fingers in my ears yelling SHOW ME THE MONEY!!!!!!!!!
I'm exaggerating of course. I don't actually do that. And I don't actually think you should ignore everything except "actual results". No, I'm still a Bayesian, and Bayesians don't throw away evidence. But given my experiences, I've come to believe that such evidence isn't nearly as strong as I had previously thought.
Deals fall through
This is similar to the above section. In the above section, I'm basically talking about how when people say "This app is awesome! I would pay for it!" it doesn't actually mean that much. This section is going to talk about how when potential business partners say "This is really interesting! Let's talk more!" or even "I'm in!", it doesn't actually mean that much.
You can read more about it in part one, but I've just had so many deals and stuff fall through. A lot of people were telling me that they were interested in working with me. Some even said that they would work with me. A verbal "yes". I thought to myself:
Given my experiences, I've come to believe that verbal interest just isn't that telling. And this seems to mirror the experiences of others as well.
I don't want to say that verbal interest means nothing though. Honestly, I probably went too far in the above paragraph saying that it "just isn't that telling". I still have limited experience, and I'm not sure how strongly to weigh it as evidence.
But one thing does seem pretty clear: the lack of actual traction is stronger evidence than verbal interest. Eg. for me, I had a lot of people saying they're interested in partnering with me, but no one actually following through. I think the lack of follow through is stronger evidence than the verbal interest. Similarly, I had a lot of people saying they really like the app and stuff, but even when it was free I was only getting maybe 100 users/month, and they weren't spending that much time on the app. I should have paid more attention to that.
Build it and they'll come
I've always had a perspective that goes something like this:
Although it's still a little counterintuitive to me, I have to say that my perspective is different now. The perspective I described above is some version of "build it and they'll come" (BIATC). I now thing that BIATC is pretty wrong.
I'm still not quite sure what the mechanism is, though. I suppose that it takes a lot to actually get word of mouth to happen. I suppose people don't actually do that much comparison shopping, and instead lean heavily towards things that are popular, have social proof, and that they stumble across organically, eg. in blogs.
I feel like I may have overestimated BIATC due the stuff I hear from YC folks. They talk a lot about focusing heavily on the product, as opposed to marketing it. I at least get the impression from them that making something people want is what it's all about, and that if you manage to do so, you'll have success. Here's Paul Graham in How to Start a Startup:
Using the example of restaurants, I actually can think of a lot of restaurants with great food that don't do so well. Hole-in-the-wall-type places with awesome food, but that are never too busy. And on the other hand, I can think of a lot of trendier restaurants that have terrible food but a lot of customers.
Still, I don't want to completely discount BIATC. I think it can be true in some situations. I just think those situations have to be pretty extreme. You need to 10x your competition. You need to be solving a hair on fire problem. You need to be building a painkiller, not a vitamin. You need your users to really, really love you. When you totally blow the competition away, or when you truly solve an important problem that hasn't been solved yet, yeah, when you build it, they'll come. Maybe that's what YC is trying to convey.
Customer acquisition is hard
Of course, I didn't actually just build my app and sit there expecting people to come to me. I did try to acquire customers.
It just didn't work. Despite the fact that I have a product that people say all of these nice things about, I only managed to get, let's say three paid users and 100 free trial sign ups. That amount of traction seems very much not in line with the quality of product I have, which makes me think that customer acquisition is very hard. Or maybe just that I'm still bad at it.
Here's what I tried, how it went, and what I learned:
Affiliate partnerships
This has always been Plan A. Find some people who already have huge followings eg. on YouTube, and piggyback off of that by offering them revenue share. Should be pretty simple. It's free money for them, and the product is good, so why wouldn't they want to do it? Especially when I ended up offering them 50%.
Well, I'm still not quite sure what the answer to that is.
Maybe they know how small the market is and how little money they'd make? Perhaps. But still, it's so low effort for them to throw a link in the description. And they're often already using poker software in their posts/books/videos, so it isn't any extra effort for them to use mine. In fact, mine is a better fit because mine is the only web app, and they could link to simulation results.
In talking to them, the response I usually get is that they've been meaning to check the app out but just haven't had time. This makes no sense to me because as poker professionals, I'd think that they are already spending time studying with software, so why not substitute mine in? Maybe they don't actually study? Maybe they don't want to spend the time messing around with a product they're unfamiliar with? Who knows.
My takeaway here is that if the deal you're offering people is only marginally beneficial, you might just end up with no partners.
YouTube and blogging
I think I have produced solid content with my YouTube channel and blog. People have told me that. But I've gotten only a minuscule amount of hits.
Again, my initial thinking was what I described in BIATC. That the amount of hits I get should be at least roughly proportional to the quality of my content.
Nope! That didn't happen! In retrospect, this makes sense. How are people supposed to discover your YouTube videos? YouTube recommends videos that are already popular, and chooses the videos that are already popular to put high in their search results. Similar with blogging. It's a chicken-egg problem that I have yet to figure out.
Paid advertising
There are a ton of huge companies built off of ad revenue: Google, Facebook, Instagram, Twitter, Reddit, etc. So then, that gives me the impression that paid ads are a huge thing that everyone is doing. And if they're doing it, they're doing it because it works for them, presumably. So paid ads have always been something that I assumed to work well.
Nope! The first place I learned this was in Julian Shapiro's guide. He said something that made it actually seem pretty clear in retrospect. If you were able to get an ad channel working profitably for you, you'd just be able to scale up with that channel, acquiring customers profitably, and making a ton of money. If every company were able to do that, entrepreneurship would be pretty easy.
Since reading his stuff, I started to hear other people say the same things, that it's really hard to do profitably and most companies never manage to do so. I still decided to give paid ads a shot, because it's a low risk high reward thing, but it turned out not to work for me.
Direct sales
I spent a pretty good amount of time doing direct sales. Some in person networking at a poker meetup, playing poker at the casinos, meeting people in coffee shops, Ubers, etc. Then I also spent some time DMing people on poker forums, and emailing poker players and coaches. I didn't go too crazy because I don't want to be spammy, but I definitely did some.
None of it really worked though. I think my takeaway mirrors the BIATC stuff, where you need to have something really valuable to actually get peoples attention.
I again found this surprising. I figured that when you actually are in a conversation with someone, they'd sort of give you the benefit of the doubt. But no, I didn't find that to be true.
Refer a friend
There was a point where I was offering $20 for every friend you refer. Pretty good, right?! Seemed pretty generous to me, and like something people would want to take advantage of. But no, that didn't happen. Still confusing to me, but I guess the lesson again is that it really takes a lot to get peoples attention.
Sharable simulation links
(I talk more about this in this post.)
Poker players spend a lot of time in forums discussing hands. They'll say things like, "you only have 33% equity, so you should fold". My app lets you link to a simulation that shows that you only have 33% equity. For non-users, it's in readonly mode. For users, when they could click the link and play around with the assumptions.
I thought this would be huge. Making it easier for people to discuss hands in the forums. But no, it did nothing. I think a big part of it is that most people in the forums approach it very casually and don't want to spend a lot of time on their comments. They just want to add their two cents, and then leave.
Free
The app was even free for a long period of time. I would have thought that this would attract a ton of users, but no, that didn't happen. More evidence that it takes a lot to actually get people to look in your direction, and that customer acquisition is hard.
People are lazy and irrational
Here's what I mean. It's really true that if you're a poker player and are actually trying to get better, that you should have some sort of poker software to study with. Coaches and professionals say this all the time. But people are still too lazy to do so.
Hell, they are just too lazy to study in general. I've said this before, but they prefer passive things. Watching a video and having someone "tell you the answers", even though that sort of passive study never works, regardless of the field. Eg. with math, you have to actually do a lot of practice problems yourself to get it to stick.
I've also heard a lot of coaches complain about students who pay them $100+/hr not do the homework that they are assigned. The coaches beg and plead, but the students just don't want to do the work.
That's the lazy part. I guess the irrational part plays into that, but what I really had in mind is that poker software is pretty easily a +ROI investment, but people still don't care. And if poker players aren't persuaded by +ROI investments, then I'm not sure what other demographic would be persuaded.
Everything else
There are definitely things that I'm forgetting. Hopefully I'll add to this post as I remember them.
If any readers out there have any advice, I'm all ears! I want to try to learn as much as I can from this.