The downsides are that you have to put some significant time into this topic to know what a good deal is, learn how the market and auction houses operate and to build a diverse or highly specialized and sought-after collection that is very likely to rise in value compared to other possible collections you could compile (which overall will almost certainly rise in value too, but maybe not as much as a collection you put some thought into).

Those don't seem like substantial downsides, and ones that would be incurred already by a lot of smart philatelists.

The efficient markets hypothesis asks: why can stamps be a decent investment compared to something like an index fund? Especially since there are things like hedge funds for collectibles, and stamps are a leading suspect (along with wine and art and comic books).