in this world I am 2 orders of magnitude more likely to have a crime perpetrated against me in said neighborhood.

Presumably, then, your actual objection is to living in predominantly high-crime neighborhoods, and you'd be fine with poor/black/hispanic neighborhoods? I'd certainly suspect racism if the objection was "but poor = crime!" rather than "looking at the actual statistics, that neighborhood has high crime".