Historically, we have dismissed very obviously sapient people as lacking moral worth (people with various mental illnesses and disabilities, and even the freaking Deaf). Since babies are going to have whatever-makes-them-people at some point, it may be more likely that they already have it and we don't notice, rather than they haven't yet. That's why I'm a lot iffier about killing babies and mentally disabled humans than pigs.