There are a couple of different aspects to fandom, and it's worth splitting them apart. I'm a HUGE fan of GEB and the Well Tempered Clavier, and have been for over 25 years. Neither of these are the basis of any social group I'm part of (though GEB fandom is rife in a some, it's not the core of any).

The aspects of fandom you're looking at - the clique or socially-separate group, would seem to come not from bad elements of the work, but from a desire to form tribes and share hardship, normally by repelling out-group attack (or perceived attack). Note that an attack on the group does not require an attack on the work, though less-generally-liked reasons for group membership probably make it easier to find attackers.

I'd recommend you avoid changes in your book to cater to such tribal instincts, unless you're using it as an example of how people trick themselves.