If there's a tradition of such adjudication - and if there's a good supply of rationalists - then people will hire them as long as they can agree in advance on submitting to arbitrage.

You know, this actually makes (entirely unintended) sense. If the rationalists are obliged to express their evaluations in the form of carefully designed and discrete bets then they are vulnerable to exploitation by others extracting arbitrage.