http://blip.tv/tech-love-live/osb09-donnie-berkholz-assholes-are-killing-your-project-2464449
This is specifically about why it's important to get assholes out of open source projects, but it applies in general. It includes an analysis of the social cost of keeping people around who frequently make other people unhappy, and in particular a way to balance the social costs (distraction, people doing much less work or leaving, useful volunteers not joining, assholes recruiting other assholes, etc.) of assholes against the useful work some of them do.
To some extent. Although from what I understand there are many who underestimate the practical (and even raw financial) consequences of certain cultural aspects.
The book by that name focuses on specific destructive behaviours to prohibit or watch for.