I was just about to respond by asking if you would advocate a website in the beliefs of the members are aggregated based on their reliability, then I remembered: prediction markets.

I'm guessing you're not pushing a real prediction market due to legal issues, but why not create one that uses token points instead of real money?

My first thought was slightly different: have testable predictions, as in a market, but the system treats each persons' likelihood ratios as evidence (as well as the tags for the prediction, to account for each person's area of expertise)

It seems to me that the real issue still is a supply of testable problems.