The downsides are that you have to put some significant time into this topic to know what a good deal is, learn how the market and auction houses operate and to build a diverse or highly specialized and sought-after collection that is very likely to rise in value compared to other possible collections you could compile (which overall will almost certainly rise in value too, but maybe not as much as a collection you put some thought into).
Those don't seem like substantial downsides, and ones that would be incurred already by a lot of smart philatelists.
The efficient markets hypothesis asks: why can stamps be a decent investment compared to something like an index fund? Especially since there are things like hedge funds for collectibles, and stamps are a leading suspect (along with wine and art and comic books).
Thus spake Eliezer:
It seems that many here might have outlandish ideas for ways of improving our lives. For instance, a recent post advocated installing really bright lights as a way to boost alertness and productivity. We should not adopt such hacks into our dogma until we're pretty sure they work; however, one way of knowing whether a crazy idea works is to try implementing it, and you may have more ideas than you're planning to implement.
So: please post all such lifehack ideas! Even if you haven't tried them, even if they seem unlikely to work. Post them separately, unless some other way would be more appropriate. If you've tried some idea and it hasn't worked, it would be useful to post that too.