It may be relevant that asexuals started coming out after homosexuals did.
I don't think asexuals were really in the closet until relatively recently. After all, many denominations of Christianity provide people with reasonably high status positions that require that the person abstain from sex. Even the denominations that don't have monastic traditions wouldn't look down on someone who abstains from sex. It wasn't until the sexual liberation movement promulgated the idea that anyone who isn't interested enough in sex is a prude and probably repressed that asexuality became something unacceptable.
The value of abstaining from sex in priestly situations is signalling of willpower and piety, one must be actively resisting temptation. As such someone with no sex drive wouldn't get the same cache.
It is a cultural universal that people are discouraged from having sex as often and with as many people as they want to. Every culture I've ever heard of imposes many restrictions on sex. I've never heard of a culture that shames people for being too stingy with sex.
If we assume that culture is adaptive, this means that the human sex drive is too strong for humans in society. Why is this? As sex drive is a phenotypic feature with extraordinarily strong selective pressure, why haven't we evolved to have the proper sex drive?
One reason could be that reduced sex drive is selected for at the level of the group, while higher sex drive is selected for at the level of the individual.