Is anger at panhandlers justifiable?

Dear all,

Lately I find myself getting fed up with panhandlers, not with the act of panhandling itself but with their pleading. There's an old black man outside Grand Central every day who sits curled up on a bucket, hunched over and shaking, pleading "Pleeease! Pleease!" like a freezing man begging for shelter.
I help him sometimes out of guilt, but yesterday after putting some change in his cup, I said, "You're out here EVERY SINGLE DAY like this!" He told me that he couldn't get over his mother's death and that all he wants is to really go back to work. I didn't really buy it honestly. I think for me the pleading eventually feels like the boy who cried wolf: a constant emergency begins to feel like no emergency at all. And if every day is the same, then what's the sense of helping him at all?
I don't know how Jesus would have responded to this: he was great at healing people and calling for support to the poor, but I don't remember many instances of him helping beggars financially. Heck, he didn't have a lot of money to begin with!
I always remember that moment when the woman spent her money to make him (perfume? was it?) and the disciples protested that the money could have been spent on the poor. Jesus said "The poor you will always have with you," but that he was only going to be there for a while. Some might use that phrase to dismiss working for the poor, but I think it was just meant to say that worship of God and Jesus must come first in all our endeavors.
My anger towards that man begins with the fact that he is breaking my heart and the sneaking suspicion that he is taking advantage of my compassion. Perhaps, there is no way around it but to admit that anger and help anyway. Or perhaps it is better to give my money to a cause that makes a bigger difference and then give the man a referral to it: maybe that'd be a bit more of the "arise and walk" response, leaving him the choice to make himself well.
I welcome your thoughts.