Although the reason it may be hard to justify is that I have to explain to others who are somewhat distant, "Well, I care about you, but I don't really CARE care, you know? Because of X, Y, and Z you see. You understand." That's not really something that's polite to say to someone's face, so if you say in public "Yeah, I don't really care about everyone who isn't my friends and family, I just want to be sufficiently nice to all of you that I don't get hurt out of spite or guilt." That's offensive.

I see it as an unfortunate fact of limited resources. Caring about the entire world in enough detail is impossible because each of us only has a few neurons for every other person on Earth. Until we can engineer ourselves for more caring we will have to be satisfied with putting most other people into classes such that we care about the class as a whole and leave individual caring up to other individuals in that class. Being sufficiently nice to any particular individual of a class is probably the most you can rationally do for them barring extenuating evidence. If they are obviously out of the norm for their class (starving, injured, in danger, etc.) and you can immediately help them significantly more efficiently than another member of the class then you should probably spend more caring on them. This avoids the bystander effect by allowing you to only care about overall accidents and injuries in the class in general but also care specifically about an individual if you happen to be in a good position to help.

Otherwise try to maximize the utility of the class as a whole with regard to your ability to efficiently affect their utility, weighted against your other classes and individuals appropriately.