the only reason I can think of to not call the people I'm describing racists is because it might offend them

If they believed you, consistency bias might make them lean more toward racist-2 and racist-3. Or it might shame them into lowering their belief in the entire reactionary memeplex, which would be epistemically sub-optimal. It might lower their status, or even their earning ability if justified accusations of racism became associated with their offline identities. There's many ways leveraging emotionally loaded terms can have negative effects.