In my experience, enough people feel that way that it's rare for anyone to starve or freeze unless he behaves so badly that he doesn't deserve to be helped.

There is a nice critique of this libertarian view of ethics here.