Remind yourself that the panhandler is defecting (in the Prisoner's Dilemma sense) by putting you in that situation. Remind yourself that they are actively and premeditatedly manipulating you through a set of known exploitable psychological levers. There is a strain of Dark Arts to this advice, because you are choosing to preemptively deflect your empathy with a feeling of defensiveness. It is nonetheless true that the panhandler is being rude, definitionally, and that you are being tricked.