In 2007, the Department of Children, Youth, and Families (DCYF) held a seminar for the nonprofits vying for a piece of $78 million in funding. Grant seekers were told that in the next funding cycle, they would be required — for the first time — to provide quantifiable proof their programs were accomplishing something.
The room exploded with outrage. This wasn't fair. "What if we can bring in a family we've helped?" one nonprofit asked. Another offered: "We can tell you stories about the good work we do!" Not every organization is capable of demonstrating results, a nonprofit CEO complained. He suggested the city's funding process should actually penalize nonprofits able to measure results, so as to put everyone on an even footing. Heads nodded: This was a popular idea.
Actually, these objections might not be quite as insane as they might sound at first.
The issue is that rigorously measuring results is hard, and frequently when people try to quantify results, they screw it up and force people to spend their time gaming a dysfunctional metric instead of doing real work. Just look at everyone who complains about academia forcing researchers to publish everything they can in as small bites as possible in order to maximize citations, instead of being able to do things in a way that'd be more useful for everyone. Or look at the software companies that used to measure programmer productivity in terms of lines of code written, and - as far as I know - still haven't managed to come up with any very good objective metric for comparing their workers.
The fact is that there are plenty of cases where we know something, but don't have any way of showing it in an objective and easy-to-quantify way. A boss might know for sure who's a valuable researcher or programmer on the basis of her interactions with them, but be unable to prove it rigorously. And these are still relatively simple domains - take something very open-ended like "the impact of nonprofits", and things get even worse.
Given that people are generally bad at designing good ways of quantifying such things, and that bad measures will produce worse results than no measures at all, then it can actually make perfect sense for somebody interested in helping people to object to the creation of such measures. Better (the thought goes) to give everyone money and end up funding both useless and high-impact organizations, than to concentrate all the money to a few organizations which are good at gaming the metrics and most probably all useless.
The issue is that rigorously measuring results is hard, and frequently when people try to quantify results, they screw it up and force people to spend their time gaming a dysfunctional metric instead of doing real work.
This is a problem in business as well. Marketo is able to charge companies thousands per month for tracking online advertising outcomes in companies with long, relationship-based B2B sales cycles (who might be aiming to make a few huge sales per year).
John Wanamaker: "Half the money I spend on advertising is wasted; the trouble is I don't know which half."
The Worst-Run Big City in the U.S.
A six page article that reads as a very interesting autopsy of what institutional dysfunction in the intersection of government and non-profits looks like. I recommend reading the whole thing.
You don't say?
Reading this I had to bite my hand in frustration.
What the efficient charity movement has done so far looks much more impressive in light of this. Reading the rest of the article I think you can on your own identify the problems caused by lost purposes, applause lights and a dozen or so other faults we've explored here for years.
Discussions here are in many respects a comforting illusion, this is what humanity is like out there in the real world, almost at its best, well educated, wealthy and interested in the public good.
Yes it really is that bad.