Annoying Pan Handlers

I am not sure how to say this or if it is even politically correct but I have had it with pan handlers.  When I first moved to the area I would give them change but recently I decided to stop.  Since I have decided to stop I have noticed that the pan handlers have been a lot more resistant and sometimes scary.  A few of them have followed me for a short distance and kept nagging me for money and it really scares me.  I see them all the time with signs that say all kinds of sad things.  Chris laughs at a lot of the signs because he usually notices things on them that the holder intends to use the money for drugs or alcohol.  He tells me all he time that he sees the same ones in the same spots every day so he doesn't have much empathy for them.  He also doesn't have a lot of problems when he tells them no, which he always does, because he is really big.  I however am a lot smaller and the pan handlers don't really listen.  I don't know what to do.  I can't afford to give them money because we have a limited supply but I also want them to leave me alone.  Any ideas because I am out.