Progressives by and large see the world through the following lens: There are some people who are oppressed and others who oppress them. Government policy ought to focus on emancipating the oppressed and punishing/overthrowing the oppressors. Criminal Justice: white people oppressing brown people. Abortion: Christian men oppressing women. Foreign policy: America oppressing the rest of the world (unless it's America saving some oppressed foreigners from an oppressor). Housing policy: landlords oppressing tenants. Labor: captital oppressing unions. Taxes: the one percent oppressing the 99%. Marriage equality: straight Christians oppressing LGBT people. Progressives aren't generally concerned about utility: they're concerned about justice.

I think you're rather generalizing Social Justice Movement mentality to progressives as a whole. They're a vocal subset, but I think a lot more people would identify as "progressive" given an explanation of the options than would ascribe to the oppressed/oppressor lens.