I think killing babies is uniquely horrible rather than uniquely harmful.

Professor Moriarty's evil plan to destroy the world and kill everyone is nearing completion. All he has to do is to press the big red button on the world-destroying machine. You are standing nearby with a hand grenade and could kill Moriarty. But, anticipating this sort of problem, he has taken care to have the whole area filled with cute babies.

You should probably throw the hand grenade even though it will kill lots of babies. But if your response to this situation is anything like "Yessss! Finally I get to kill some babies!" then, although I suppose in some sense I'm glad it's you rather than someone more scrupulous in this bizarre situation, you are a terrible person and the world needs fewer people like you and in the less-artificial contexts that make up maybe 99.9999% of real situations your enthusiasm for baby-killing will not be any sort of advantage to anyone.

(There is a big difference, in principle, between the questions "Will doing X make the world a better place on balance than not doing X?" and "Is X the sort of thing a good rather than a bad person would do?". Unfortunately, everyday moral discourse doesn't make that distinction very clearly. Fortunately, in most actually-arising situations (I think) the two questions have similar answers.)