Well, most people are not signaling their interest in sex most of the time, so not signaling interest in sex isn't noticeably odd. That someone never signals interest in sex is difficult to notice, especially if one believes that absence of evidence is not evidence of absence.
The analogous thing isn't true about homosexuality. It's much easier to be forced out of the closet as gay, and it takes much more effort to successfully stay in it, than as asexual.
So it's not too surprising that asexuals started coming out of the closet only after homosexuals did.
It is a cultural universal that people are discouraged from having sex as often and with as many people as they want to. Every culture I've ever heard of imposes many restrictions on sex. I've never heard of a culture that shames people for being too stingy with sex.
If we assume that culture is adaptive, this means that the human sex drive is too strong for humans in society. Why is this? As sex drive is a phenotypic feature with extraordinarily strong selective pressure, why haven't we evolved to have the proper sex drive?
One reason could be that reduced sex drive is selected for at the level of the group, while higher sex drive is selected for at the level of the individual.