That's not even an example of the ad hominem fallacy.

"You have an ugly face, so you're wrong" is ad hominem. "You have an ugly face" is not. It's just a statement. Did the speaker imply the second part? Maybe... but probably not. It was probably just an insulting rejoinder.

Insults, i.e. "Attacking you, not your argument", is not what ad hominem is. It's a fallacy, remember? It's no error in reasoning to call a person ugly. Only when you conclude from this that they are wrong do you commit the fallacy.

So:

A: It's wrong to stab your neighbor and take their stuff.
B: Your face is ugly.
A: The ugliness of my face has no bearing on moral...
B, interrupting: Didn't say it does! Your face is still ugly!