So here you and I might disagree; maybe I would abstract the concept further than you would. I presume you're not limiting "Prisoner's Dilemma" to actual prisoners, because that seems tremendously silly. So how far would you limit it?

Approximately the same. I wouldn't use Prisoner's Dilemma to describe a Tragedy of the Commons myself but would be unlikely to correct it. In some such cases I'd prefer to just use "Newcomblike", which takes the abstraction a step further (removing the strict necessity for symmetry) but is also overtly an abstraction.

But are there really people who go around applying the term "Prisoner's Dilemma" to things like Stag Hunts or zero-sum games?

Yes.