I often seem to run into problems when I use the de facto label for this group. For example, when I say, "I've been hanging out with rationalists lately," I notice that many people immediately go on the defensive. They might ask why you need a group in order to be rational, or they might say that they don't believe that people are inherently rational. Of course, I made none of those claims, I simply indicated that I was hanging out with rationalists.
You might think that "rationalist" is simply a descriptive label, but it carries positive connotations -- and what people tend to hear is "I'm a superior thinker to you," or maybe "I'm a part of this group, which ascribes the label 'rationalist' to itself, to make ourselves seem higher status than we really are."
Why does this matter?
The community doesn't exist in a vacuum; how the community is viewed from the outside matters. As the community grows and as people gain awareness of it, branding becomes important. People talk to each other, and communities gain reputations. Even if you believe that we are a loose collection of individuals, as soon as you assign a name to yourself, that is sufficient to form a group identity.
The people we interact with tend to share similar interests. The population of New York may be in the millions, and yet I run into the same people at different functions without coordination.
The more negative perceptions associated with a group, the more rapidly evaporative cooling of groups will occur.
What to do?
It's far better to talk about good things that you've gained from being in the group. It's better to say what the group does, not what the group is.
But beyond that, it's about time the community picked a better label to use. I have one idea, but I'll hold off on proposing solutions.
Not sure that's actually true, at least if you read "magical" as "special; privileged". Mainline Objectivism is a closed system: it explicitly disavows thinking that builds on, expands, or (especially) critiques Rand's work, treating it as a self-contained totalizing philosophy. As far as I'm concerned that immediately puts it fairly high up on any scale of founder-worship, roughly on par with Scientology and higher than most strains of Marxism. It's also the sort of criterion that should be accessible from an inside or an outside view, which is convenient if you're asking questions about groups you belong to.
I understand there was a big rift in Objectivism over precisely this, with one group led by David Kelley splitting off because they were for a more intellectually tolerant Objectivism.