After reading the Arbital postmortem, I remembered some old ideas regarding a tool for claim and prediction aggregation.
First, the tool would have the basic features. There would be a list of claims. Each claim is a clear and concise statements that could be true or false, perhaps with a short explanation. For each claim, the users could vote on its likelihood. All these votes would be aggregated into a single number for each claim.
Second, the tool would allow the creation of composite claims by combining two existing claims. In particular, a conditional claim IF B THEN A would represent the conditional probability P(A|B). For every claim, it should be easy to find the conditionals it participates in, or the claims it is composed of. Conditionals are voted on same as simple claims (I would even consider a version where only conditionals are voted on).
Third, the tool would understand the basic probability laws and use this to direct the users' attention. For example, if three claims don't satisfy the law P(A|B) P(B) < P(A), users might be alerted about this error. On the other hand, if P(A|B) = P(B|A) = 1, the two claims might be merged or one could be discarded, to reduce the clutter.
Fourth, given a claim the tool might collect every other claim that supports it, follow every chain of argument and assemble them into a single graph, or even a semi-readable text, with the strongest arguments and counterarguments most visible.
Let's consider a possible workflow. Suppose you browse the list of claims, and find a ridiculous claim X assigned a high likelihood. You could just vote to decrease the likelihood and perhaps leave an offensive comment, however this is unlikely to have much effect. Instead you could find a convincing counterargument Y, then add both P(Y) = 1 and P(X|Y) = 0 to the list of claims. Now other users would be notified of the resulting inconsistency and would reply by voting on one of these claims, changing their vote on X, or by creating additional arguments that contradict Y or support X. In turn you would attack these new arguments, eventually creating a large graph of reasoning. Perhaps at some point there would be enough general claims that people from different debates could reuse some, instead of duplicating them, and only create new conditional claims, making the graph dense.
I took some time to write a small prototype. It is initialized with some AI related claims, and it implements first, second, and a tiny bit of third paragraph. Now for some meta. The prototype is a single page app, using angular1, backed by nodejs and mongodb. The actual features took between 1 and 2 hours to write. The backend and deployment took a couple more hours, largely because I hadn't done that in a while. Therefore I think that it's quite feasible to make similar prototypes for other ideas. Is there any value in it though?
I understand the concern, but I'm hoping that as claims about the future are explicit and distinguished from conditional claims, this might not be a problem. That is, if the user has already set P(Trump nominated) = 0.01 and P(Trump president) = 0.009, they will be satisfied with having rejected the claims, and will be able to consider that P(Trump is president | Trump was nominated) = 0.8, in isolation. Also, the conditional P(Trump was nominated | Trump is president) is obviously almost 1, and that should prevent anyone from setting P(Trump is president | Trump was nominated) too low. Also, P(Trump nominated) and P(Trump president) should always have reasonable values on prediction markets, which would set some reasonable bounds on the conditionals.
More generally, I suspect that the Multiple-Stage Fallacy comes from confusing event probabilities with conditional probabilities, and ultimately, I believe that all problems and confusions can be solved with more rigour.