Yes, that depends on how widely you take "the game". Nevertheless, in the contexts where I run across the phrase, changing the players in question would suffice.

There are definitely situations where it goes differently, however. One example that came up in conversation today (without this phrase) is a draftee in a war, who is forced to shoot at people to avoid being shot. Changing all of the players in this position would work, but only if the players on both sides change at once. I would not blame such a person, if they don't actually want to be there.

So I seem to have just come to this conclusion: It's illegitimate to blame (to state the ethical culpability of) any player who doesn't want to play the game but is unable to quit. That includes a lot of examples, just not the ones where I've met this phrase.