A handful of points, without any particular axe to grind, from a professional music scholar:

(1) The Great Fugue is difficult to like, difficult to know what to make of -- even most of its passionate advocates would agree to that -- and there's no particular reason to think that opinions from wildly positive to wildly negative are not all within the realm of the reasonable responses to this piece. A huge amount of scholarly ink has been spilled on why it, and the late string quartets, and the Missa Solemnis, are so peculiar.

(2) Relatedly, people who love it and think that it's obviously, uncomplicatedly lovable may well be putting on airs or signaling. And as with any piece of music that has gigantic prestige built up around it (partly due to its reputation for being super-profound and inscrutable), all opinions are probably to be somewhat taken with some suspicion of signaling behavior.

(3) Think of someone who has repeatedly shown herself to be a brilliant, extremely sound thinker. You come to trust her opinions on a wide range of topics. When she says something you find absolutely bizarre or inscrutable, you're going to at a very minimum think carefully about what she says to see if the fault is with you. If you're a fan of most of the music Beethoven writes, I encourage you to give him a similar benefit of the doubt.

(4) I myself find the Great Fugue remarkable but not at all pleasant -- in fact, while Beethoven holds me enraptured right up through the Last Five Sonatas and the Ninth Symphony, he loses me a bit with the Missa Solemnis and the late string quartets, with the exception of a few isolated movements. You're certainly not wrong to suggest that admitting these views in academic music circles is low-prestige (although not as much so as it used to be), but a major factor in this is my point (3) above: Beethoven has generally earned the benefit of the doubt. Also, it's equally low-prestige in those circles to run around gushing about how amazing the Great Fugue is without having some interesting things to say about why you think so.

(5) I am totally baffled why you are so convinced that quality must be something that inheres to a piece of music. Quality is subjective, or at most inter-subjective, and aesthetic judgments do not contain truth value.

(6) Whatever you think you mean by suggesting that the music of Alban Berg (not sure why you picked him) lacks "basic music theory," I can completely guarantee you that you are wrong. Music theory is not a property of musical compositions any more than linguistics is a property of language. If what you mean is that Alban Berg was not a composer of tonal music in the 18th- and 19th-century sense, then that is true, but (a) his music contains structure, just not tonal structure; (b) the relativism of aesthetic judgments means that that is neither a bad thing nor a good thing except insofar as the pleasure some people take in his music is good; and (c) if you are hinting at the claim that people who say they like Alban Berg's music don't actually like it but are just signaling social prestige, then that may be true for some individuals but is false in the general sense.

(7) Liking has a great deal more to do with familiarity than you think it does, and substantial music cognition research backs this up.

(8) It is probably impossible to separate individual aesthetic pleasure from socially-pressured aesthetic pleasure as thoroughly as you want to. (I'm reminded of the famous Judgment of Paris wine-tasting episode (link is to Wikipedia, tinyurl is the only way I could get it not to be broken).) We are social beings, so we should release ourselves from the imagined obligation to make all our aesthetic judgments in a social vacuum. Even the pleasure you take from the things you think you like in the most genuine and uncomplicated way is to some degree socially determined. Liking things is something that we're in many ways primed to do by what we hear from others -- if my best friend recommends me a novel, I'll read it with somewhat more patience knowing that someone whose opinion I value has vouched for it. If in the end I like it, even if I wouldn't have liked it otherwise, there's no reason to think of that liking as being less genuine or less valuable.

(7+8) If you listen to the Great Fugue a hundred more times, unless you find something viscerally unpleasant about it (which, make no mistake, some people really do, since it's pretty loud and screechy), you will probably like it, because familiarity and social conditioning tend to do that to us. If you like it, stop driving yourself crazy and just like it. If you can't stand to like something thinking that there's some element of social conditioning driving you to do so, then by all means stop listening to the Great Fugue.

(9) That said, many people do find that it's interesting or pleasant to expend a little effort to see if they can learn to like something that they don't immediately like but have some reason to think they may like eventually. That's what an acquired taste is. If you give it a shot and it doesn't take, then let yourself off the hook. And you can always take some pleasure in being the aggressive countersignaller who goes around telling anyone who'll listen that the Great Fugue is totally overrated (some people will take a lot more pleasure in that than they ever could in the piece itself (the politest, but by no means only, word for those people is "contrarians")).