Ad hominems are not always fallacious, but the ad hominem fallacy is. The character, circumstances, or actions of the arguer have no bearing on whether the argument's conclusion follows from its premises.

The character of the speaker could be a good reason to question the truth-value of the speaker's claims. And given limited time/resources, it might be a good enough reason to not bother to listen to the speaker's argument. But it is never what makes an argument invalid, regardless of whether one is reasoning deductively. If you disagree, please give a counterexample.

ETA: it's also never what makes it the case that p is false, unless the speaker's character is directly linked to the proposition somehow. (like in the proposition, "I am a good person")