There is a very similar quote from Ayn Rand as well:

It is not justice or equal treatment that you grant to men when you abstain equally from praising men’s virtues and from condemning men’s vices. When your impartial attitude declares, in effect, that neither the good nor the evil may expect anything from you—whom do you betray and whom do you encourage?

Your point is good. Sometimes it's just a matter of allocation of resources -- and yes, may sound like "you're with us or you're against us" depending on the tone.