Because, believe it or not, it's often hard to tell the difference, especially if you know about mind projection fallacy. I tend to give people the benefit of the doubt.
Plus most people avoid creating a confrontation because that also could put them in a negative light. (Not only do you have to think what the other person is doing is bad, but the majority of other people have to think that as well, for you to get any value out of the confrontation.)