Did I miss any sort of deeper reasons I could be using for this?

"That one guy I know and the stuff they say" is usually not a great proxy for some system of belief in general; therefore, to the extent you care about the same stuff they care about when they say "social justice", a knee-jerk rejection of thinking about that stuff systematically or in terms of axioms and soforth should probably come off to you as self-defeating or shortsighted.