It is pretty easy to imagine lots of smallish filters as responsible. If I am doing my math correctly, then 9 filters that each weed out only 9 out of 10 solar systems would reduce the expected population of intelligent life in the Milky Way to just .1 systems. Since we know so much more about star formation I am a lot more comfortable saying "habital planet" will weed out 90% of systems, that is not nearly a big enough filter to account for what we see by itself but as you stack them filters of this size are quite substantial.

edit: er no that's wrong, off by a few orders of magnitude. So yeah, gosh where is everyone?