"You have an ugly face, so you're wrong" is ad hominem. "You have an ugly face" is not. It's just a statement. Did the speaker imply the second part? Maybe... but probably not.

I contest the empirical claim you are making about human behaviour. That reply in that context very nearly always constitutes arguing against the point the other is making. In particular, the example to which you are replying most definitely is an example of a fallacious ad hominem.

A: The ugliness of my face has no bearing on moral...

In common practice it does. The rules do change based on attractiveness. (Tangential.)