Fandom is a subculture that grows up around people who are passionate about a work when the rest of the world isn't.

If the work is part of the dominant culture, nobody has to build a fandom around it. The Well-Tempered Clavier is assigned to every piano student -- nobody has to organize clubs to listen to Bach in secret.

To have a fandom, a work doesn't have to be bad. It can just be overlooked, forgotten, or left behind by the mainstream. Gilbert and Sullivan operas are pretty good, but they have a fandom made up of old-fashioned Anglophiles and intellectual showoffs. (I'm one of them, naturlich.)

It helps if there's something totalizing about the work itself -- if the author insists that it should change how you see everything about the world, then those who like it will make fandom part of their identity. (See: Wagner, Rand, Kerouac.)