I might downvote comment C1 by user U1 because of my understanding of C1 informed by the context established by U1's contributions taken as a whole, even if an identical comment C2 by user U2 would instead cause me to reply to C2, or just ignore it.

More generally, individual comments aren't events in isolation, and I don't necessarily respond to them as if they were.