I interpret your underlying point to be that LW is used by the most prolific contributors not only as a place to discuss and learn about rationality and related topics, but also as a place to socialize with like-minded people who share their unusual interests in topics such as sci-fi that are not of interest to most people -- and often the socializing aspect is combined with the rationality-dojo aspect in a single post or comment, when people give fun references to sci-fi or D&D or whatever or just go off on tangents about things of interest to them.
There are clearly advantages and disadvantages to not having a norm of encouraging (mostly) 'generic rationality' discussion with no (or minimal) references to things that would be unfamiliar to most (or too many) of the target audience (what is the target audience specifically? how many is too many?).
I don't have a clear idea of what the advantages and disadvantages are, or what the alternatives, but I'm curious how many people who are strongly interested in rationality are turned away by such things. Do we think that many people are seriously discouraged despite having a strong interest in rationality, or are we talking about people who clicked a random link and left immediately because it was "too weird"?
Less Wrong is as a community extremely nerdy. That's true for almost any definition of "nerd" that captures anyone's intuition for the word. However, to a large extent, many aspects of nerdiness are not connected to rationality at all, even though nerdiness may be associated with more rationality in some limited aspects. For example, fantasy literature is probably not in any deep way connected to either intelligent or rational thinking except for historical reasons.
Yet LW is full of references to science fiction, fantasy literature, anime and D&D. In one recent example, a post started with an only marginally connected tidbit from Heinlein. Moreover, substantial subthreads have arisen bashing aspects of other subcultures. For example, see this subthread where multiple users discuss how spectator sports are "banal" and "pointless". I suspect that this attitude may be turning away not only non-nerds but even the somewhat nerdy who enjoy watching sports, and see it has harmless tribalist fun, not very different than friends arguing over whether Star Wars or Star Trek is superior which has about the same degree of actual value here.
There's a related issue which is a serious point about rationality and human cognition: Our hobbies are to a large extent functions of our specific upbringings and surrounding culture. That some people prefer one form of fantastic escapism involving imaginary spaceships isn't at some level very different than the escapism of watching some people throw and catch objects. Looking down on other people because of these sorts of preferences is unhelpful tribalism. It might feel good, and it might be fun, but it isn't helpful.