It happens to everyone [therefore] it doesn't matter to me

It could, but OTOH a person's level of annoyance for something isn't always a monotonically decreasing function of how frequent that is: for example, the more often people post something I'm not interested in (e.g. gossip about celebrities, rooting for football teams, pictures of attractive people in underwear of the gender other than the one I'm attracted to) on Facebook, the more annoyed I am by seeing that particular kind of thing (ceteris paribus).

BTW, I've recently noticed a couple examples the reverse phenomenon of "it doesn't matter to me [therefore] you can get over it", namely "it annoys me [therefore] you should be outraged". For example, last night Bob introduced me to Alice; I don't remember exactly what Alice told me, except that I thought it was supposed to be playful banter; this morning Bob apologized to me on Facebook for introducing “that boorish blonde”, because Carol had told him that Alice had been rude to me. I'm guessing that Carol would be annoyed by... whatever it was that Alice told me, so she assumed that I would be annoyed too. (I have asked Carol herself for clarifications on Facebook.) This has raised my estimate of how much the typical person is annoyed by what would sound to me like playful banter, incidentally making me take PUA critics' concerns about “negging” more seriously.