I'm very glad I asked for more clarification. I'm going to call this system The Reviewer's Dilemma, it's a very interesting solution for allowing non-software analysis to occur in a trusted manner. I am somewhat worried about a laziness bias (it's much easier to agree than disprove), but I imagine that there is a similar bounty for overturning previous results this might be handled.

I'll do a little customer development with some friends, but the possibility of reviewers being added as co-authors might also act as a nice incentive (both to reduce laziness, and as addition compensation).