I guess we could understand catcalling better by seeing its equivalent in more primitive societies, or preferably at apes. Or perhaps by putting a hidden camera on a person who does it frequently, and examining the consequences.

My guesses:

1) Some women react positively to catcalling. Even if one in a hundred, then it would be enough, because the cost is low. As an analogy, receiving spam is also annoying, but a tiny fraction of humans react by sending their money, which rewards the spammers.

2) Catcalling may be a defection in a Prisonners' Dilemma of a group of men meeting a woman. A more polite group would be more likely to impress her positively. But even in the best case scenario, she would most likely choose only one of them as her sexual partner. By catcalling, a man positions himself as a "speaker" of the group, as the dominant male. He slightly increases his personal chance by decreasing the chances of the group as a whole.

3) In its most primitive form, catcalling could be an encouragement to a group rape. It is not a signal for the woman. It is a signal for the fellow men to join the action.