"don't think about it either way" does not necessarily mean indifference, it means reverting to default behaviour.

Humans are (mostly) pro-social animals with empathy and would not crush another human who just happens to be in their way - in that they differ from a falling rock. In fact, that's the point of hate, it overrides the built-in safeguards to allow for harmful action. According to this view, to genuinely not give a damn about someone's life is a step further. Obviously.

The thing about built-in default behaviour given by evolution is that it will not trigger in some cases.

Rationality and the English Language

"Unreliable elements were subjected to an alternative justice process"—subjected by who? What does an "alternative justice process" do? With enough static noun phrases, you can keep anything unpleasant from actually happening.

or HPMoR Ch.48

Your brain imagines a single bird struggling in an oil pond, and that image creates some amount of emotion that determines your willingness to pay. But no one can visualize even two thousand of anything, so the quantity just gets thrown straight out the window.

or HPMoR Ch.87

Because the way people are built, Hermione, the way people are built to feel inside [...] is that they hurt when they see their friends hurting. Someone inside their circle of concern, a member of their own tribe. That feeling has an off-switch, an off-switch labeled 'enemy' or 'foreigner' or sometimes just 'stranger'. That's how people are, if they don't learn otherwise.

My point with that is, it's completely in line with what Eliezer usually talks about, so you know it's a perspective he holds, not just rationalization.

For completeness' sake,

Not like certain people living in certain countries, who were, it was said, as human as anyone else; who were said to be sapient beings, worth more than any mere unicorn. But who nonetheless wouldn't be allowed to live in Muggle Britain. On that score, at least, no Muggle had the right to look a wizard in the eye. Magical Britain might discriminate against Muggleborns, but at least it allowed them inside so they could be spat upon in person.

still feels off. Oh, wait, I know! Maybe Harry is being Stupid here. Or Eliezer is being a Bad Writer. Again.