An idea: Next time try to estimate how much money such person makes. As a rough estimate, divide the money you gave them by the length of your interaction. (To get a more precise estimate, you would have to follow them and observe how much other people give them, but that could be pretty dangerous for you.)

Years ago I made a similar estimate for a beggar on a street (people dropped money to his cap, so it was easy to stand nearby, watch for a few minutes and calculate), and the conclusion was that his income was above average for my country.

By the way, these people destroy a lot of social capital by their actions. They make life more difficult for people who genuinely want to ask for the time, or how to get somewhere, or similar things. They condition people against having small talk with people they don't know. -- So if you value people being generally kind to strangers, remember that these scammers make their money by destroying that value.