Oh, I agree. However, folks often take the claim "Hey, that thing you just said was kinda racist" as meaning "YOU ARE AN AWFUL RACIST SCUMBAG GO DIE IN A FIRE" and respond accordingly.

That's not too surprising given that ① people generally don't like receiving criticism of their views or actions, and get defensive; and ② many people seem to believe that only "racists" (a kind of person, usually found in Nazi or Klan uniforms) do, say, or believe racist things — and therefore that if someone says you did something racist, they are calling you "a racist" and thereby predicting that you're going to go commit hate crimes.

It's unfortunate though.

It probably doesn't help that people use "racism" to mean several different things, including:

1. Racial prejudice — having false negative beliefs about people according to their race
2. Racial privilege — the situation where some people receive social, political, or economic advantages and others receive disadvantages on the basis of their race (see also: invisible knapsack)
3. Racial hatred — having malicious intentions towards people on the basis of their race

Notably, people can do racist₂ things — perpetuating racial privilege — without being racist₁ or racist₃.