This is generalizing from the inside of my head, but I think part of what drives that sort of annoyance is fear that if one doesn't resist other people taking part in an activity, one would be obligated to do it too.

Meh. Those sorts of feelings usually drive reasoning, rather than being driven by it. (Which is not to say that you might not also be correct about your personal case. Perhaps you learned that it's disloyal (i.e. worthy of disapproval) to not do what your group is doing? If so, then that'd be a source of self-disapproval even if you merely lacked a positive desire to do what the group is doing.)