if you really want to know how valid a particular idea you've read is, there are quantitative ways to get closer to answering that question.

The ultimate in quantitative analysis is to have a system predict what your opinion should be on any arbitrary issue. The TakeOnIt website does this by applying a collaborative filtering algorithm on a database of expert opinions. To use it you first enter opinions on issues that you understand and feel confident about. The algorithm can then calculate which experts you have the highest correlation in opinion with. It then extrapolates what your opinion should be on issues you don't even know about, based on the assumption that your expert agreement correlation should remain constant. I explained the concept in more detail a while ago on Less Wrong here, but have since actually implemented the feature. Here are TakeOnIt's predictions of Eliezer's opinions. The more people add expert opinions to the database, the more accurate the predictions become.

Note that the website currently requires you to publicly comment on an issue in order to get your opinion predictions. A few people have requested that you should be able to enter your opinion without having to comment. If enough people want this, I'll implement that feature.

one of my dreams is that one day we could develop tools that ... allowed you to estimate how contentious that claim was, how many sources were for and against it... and links to ... tell you about who holds what opinions, and allowed you to somewhat automate the process of reading and making sense of what other people wrote.

That's more or less the goal of TakeOnIt. I'd stress that the biggest challenge here is populating the database of expert opinions rather than building the tools.

An even more ambitious project: making a graph of which studies invalidate or cast doubt on which other studies, on a very big scale, so you could roughly pinpoint the most certain or established areas of science. This would require some kind of systematic method of deducing implication, though.

Each issue on TakeOnIt can be linked to any other issue by adding an "implication" between two issues. Green arrows link supporting positions; red arrows link contradictory positions. So for example, the issue of cryonics links to several other issues, such as the issue of whether information-theoretic death is the most real interpretation of death (which if true, supports the case for cryonics).