You don't need to come up with hypothetical difficult-to-measure charities. As I've noted before, the WIkimedia Foundation is a tricky one. We do want your money, because it takes money to run, but for the current funding drive we've actually deliberately chosen a strategy that maximises the number of donors, to protect the projects' editorial independence (not to be potentially beholden to major donors).
Pretty much all of the valued output (the projects) is not done by the staff, but by volunteers who write stuff and take photos and so forth. This already made our GuideStar rating difficult, for instance.
And how do we measure? What do we choose to measure, knowing that we'll just get what we measure? Everyone has an opinion on what they want out of Wikipedia, for example, but how do we assess the CEV of our users in a way that we can measure the results of then acting upon and not just get variants on "more of the same please"? At present we're launching a quite speculative programme to get stuff happening on the wikis in India, for example. It's a defensibly good idea, but what measurement system would come up with that goal?
There must surely be other extant charities that are quite difficult to quantify results per dollar. jsalvatier suggests open source software projects, for example. Your money is somewhat useful, but your time is far more useful.
I'm not sure if this necessarily warrants a new discussion, or if there's an existing article/thread that addresses this topic.
There's a lot of discussion recently about charity, and how to give effectively. I've been looking over givewell.org and it definitely is the single most important thing I've found on lesswrong. But one discouraging thing is that by focusing on easy to measure charities, there's not a lot of info on charities that are trying to accomplish long term less measurable goals. The best charity there that matches my priorities was an educational agency in India that put a lot of emphasis on self improvement.
My *think* my ideal charity would be something similar to Heifer International, but which also focuses on reproductive health and/or women's rights. Feeding people fish for a day means you just need to feed them again tomorrow, and if they have a bunch of kids you haven't necessarily accomplished anything. From what I've read, in places where the standard of living improves and women get more equality, overpopulation becomes less of an issue. So it seems to me that addressing those issues together in particular regions would produce sustainable longterm benefit. But Givewell doesn't seem to have a lot of information on those types of charities.