It's good to understand the player's actions as being part of a particular game. But it's okay to punish the player, if you're feeling altruistic or vengeful enough (that is, you want to do your part to discourage people from playing that game).

It is similarly okay for the player or, indeed, a third party to consider your 'altruistic' punishment to be itself blameworthy or anti-social and subject it to punishment. After all, encouraging 'altruists' to punish the kind of player who is not powerful enough to deter punishment is typically just another part of the game, one step up in sophistication.