From Anneal's post last year

If your org is shaped like a Y-combinator company, you can spend dozens of hours absorbing high-quality, expert-crafted content.... How likely is org building success, in this premier reference class

5%. 

An AI safety lab is not the same as a Y-combinator company.  

It is. WAY. FUCKING. HARDER.

The implication is that you should expect AI safety labs for have an even higher failure rate. Is that fair?

Consider:

• In the stat Anneal links, "success" means becoming a "unicorn", i.e. achieving a billion dollar valuation.
• The standard YC deal is that they take 7% of your company for $125k.
• 7% of a billion dollar company is $70m.^[1]

Now imagine that the success rate for YC was much higher, say 50%. In that case, they would be paying $125k to get $70m * 0.5 - $35m in expectation, or a 280x return. 

What would you do if you had a slot machine that gave you 280x returns in expectation? You would put way more money into it!

Unfortunately the deal doesn't last. As you increase the supply of capital seeking startups, two things happen:

1. Valuations go up, meaning you have to pay more for the same amount of equity
2. Company quality goes down since you have to lower the bar to fund more companies.

So returns diminish. But at 28x, this remains fantastic deal!

Here's what I'm getting at: the success rate of startups is not some fixed quantity determined by "how difficult it is to build an org", it is a dynamic rate set by an equilibrium. The success rate is as low as it can be while still delivering returns good enough for LPs to continue pumping money into YC.

Of course, that's only have the equation. The other side is the willingness of startup founders to start companies. Similarly, imagine if half the time you started a startup, you ended up running a billion dollar company. It would be very popular! Overtime, that popularity drives more people to start companies who wouldn't have otherwise, and the success rate drops. Again, nothing to do with the inherent difficulty, this is just things falling into equilibrium with potential founder's next-best alternatives (working for someone else or retiring).

Some of what I'm saying here is just theoretical Econ 101 stuff that you shouldn't take too literally. But it really is historically true that YC's class size has grown 33x from 8 companies in the first batch to 260 companies in the latest batch. And that their standard deal has grown from $20k for 6% to $125k for 7%.

The mistake Anneal and many others make is to confuse a dynamic system for a static one. The low success rate of startups does not reflect the inherent difficult of building organizations, rather it reflects a willingness (from the standpoint of LPs who put money into companies and founders who start them) to throw more bodies and capital into this system until the expected value equilibrates with those people's next best alternatives. 
 

There are other considerations here, LPs might not like illiquidity, founders might not like risk, but at a very rough first approximation, this is the kind of dynamic you should have in mind when making inferences about analogous challenges.

1. ^{^}

    I'm ignoring various considerations, including:

   • YC also puts in $375,000k on an MFN Safe (meaning that it converts at the best terms that you offer future investors)
   • YC's investment will get diluted over time, very roughly maybe around 30% per round over 5 rounds between seed and becoming a unicorn, so down to .7^5 = 17% of their original ownership.
   • Some YC companies are going to be worth much much more than a single billion, so it doesn't really make sense to use what is actually the floor of the population to calculate an expected value.

   But these cut in opposite directions and don't substantially impact the point of my post.