I do not doubt that there are plenty of people sitting around complaining about society rather than accomplishing their goals. But the discussions of privilege I've seen were written from a position of (at least moderate) success, and the point wasn't to identify reasons they failed, but to identify flaws in a system and try to fix them.

The existence of people who use it to complain isn't relevant to whether the problem exists and how to fix it if it does.