I don't think this actually happens. In my experience most people who hate modern art hate it because it's more-or-less uniformly absolutely awful.

In my experience, most people who hate it do so because it's extremely unfamiliar to them, because they've only experienced a handful of examples of it (often the most difficult or "shocking"), and because they mentally associate it with snobbiness.

Also (at the risk of sounding snobby!), it's generally referred to as "contemporary art." "Modern art" refers to a period of art history that's been over for several decades.