"startup" doesn't seem to be a well defined category.

Here is Paul Graham's definition:

A startup is a company designed to grow fast. Being newly founded does not in itself make a company a startup. Nor is it necessary for a startup to work on technology, or take venture funding, or have some sort of "exit." The only essential thing is growth. Everything else we associate with startups follows from growth.

That is from http://paulgraham.com/growth.html where we also find an explanation for why startups might outperform other classes of non-publicly-traded investments. Specifically, the explanation is that startups have less need for costly financial controls to protect the interests of the investors:

The other way to get returns from an investment is in the form of dividends. Why isn't there a parallel VC industry that invests in ordinary companies in return for a percentage of their profits? Because it's too easy for people who control a private company to funnel its revenues to themselves (e.g. by buying overpriced components from a supplier they control) while making it look like the company is making little profit. Anyone who invested in private companies in return for dividends would have to pay close attention to their books. The reason VCs like to invest in startups is not simply the returns, but also because such investments are so easy to oversee. The founders can't enrich themselves without also enriching the investors.

EDIT. Fixed the URL.

This reminds me of a piece of advice I've read-- before making an investment, ask yourself whether you'd make the investment if the only person you could tell about it is your accountant. In other words, suppress the desire for bragging rights.

If the desire to make an unlikely win is common among investors, then too much money will be put into risky investments.

http://www.economist.com/news/finance-and-economics/21571443-investors-may-have-developed-too-rosy-view-equity-returns-beware-bias is relevant and recent.