I think it's worth distinguishing between cases where being correct has pragmatic advantages, and cases where it doesn't.
For example, there are plenty of people who reject the philosophical underpinnings of modern medicine. There always have been. Those people don't go to medical school, don't go to conferences, etc.; eventually a whole discussion emerges to which they are not invited. Believers in medicine don't convince the believers in homeopathy, they ignore them and concentrate on doing medicine.
And because doctors achieve more valuable things more reliably than homeopaths, over time they displace the homeopaths... they create their own community within which a belief in medicine is pervasive. That the homeopaths are not convinced isn't actually important; it just means they aren't part of that community.
Of course, if believing in homeopathy doesn't correlate with skill at carpentry, then the medicine/homeopathy disagreement may continue to exist among carpenters. But so what? Who cares what carpenters think about medicine? Why is resolving that disagreement worth devoting energy to? Better for medicinists to devote their efforts to advancing medicine.
Similarly, there are plenty of people who reject atheism. There always will be. The thing for the atheists to do is work in areas where atheism gives them a pragmatic advantage. Over time they will displace the theists in that area, and the atheism/theism disagreement will disappear in that area. If atheism confers no demonstrable advantage to carpenters, then the disagreement will continue to exist among carpenters. But, again, so what?
By the same token, if a philosophical problem turns out to not have any pragmatic implications -- that is, if there is no area where people with the correct answer can do something valuable that people with the incorrect answer can't do, or can't do as well -- then the disagreement will continue to exist everywhere. But, again, so what?
Within the more practical sorts of philosophy, like logic, epistemology, and morality, there are potentially huge gains to society for getting it right. But these can only be "tested" (in the sense of creating a society that revolves around certain philosophical ideas) on a multi-decade time scale with a huge investment of resources and possible human suffering if you're wrong, and all experiments are necessarily imperfect (communists still argue their principles would have worked if the situation had been different).
That means there are practical gains from having good philosophy, but not in a way that means it can be decided by experiment.
This thread has produced some interesting commentary around whether philosophers actually want to answer their own questions, or whether they enjoy sounding profound by debating big questions but don't want to lose that opportunity for profundity by finding single correct answers to them.
I don't quite disagree with the latter theory: the main reason I quit academic philosophy was exasperation that people were still debating questions where the right answer seemed obvious to me (like theism vs. atheism, or whether there was a universally compelling morality/aesthetics of pure reason), and worry that my philosophical career would involve continuing to debate these issues ad nauseum rather than helping to solve them and move on to the next problem.
But when I explained this to a particularly sarcastic friend, he summarized it as "So you think philosophy is useless because not everyone agrees with you?"
The problem isn't that philosophers never come up with solutions. The problem is that they come up with too many different solutions.
Science has solved many scientific problems, and anyone wondering what the solution is can look it up in a book or on Wikipedia. Philosophers have also solved many philosophical problems, but it is full of so many distractions and false solutions that anyone wondering which proposed solution is correct will have to become nearly as good a philosopher as the person who solved it in the first place. It's much easier for science to settle its disputes via experiment than for philosophy to settle its disputes via debate.
I am wary of criticizing the discipline of philosophy simply on the grounds that not everyone in it agrees with me. But I also don't want to let it off and say it's okay that they've managed to go so long without coming to any answers, when it seems to me that settling at least some of the easier problems is not that difficult.
How do we tell the difference between a discipline that doesn't really seek answers and a discipline which honestly seeks answers but just can't agree within itself? And how can philosophy do something about its level of internal disagreement without having to apply the "kick out everyone who disagrees with Less Wrong" solution?