Excellent point.
Is that a fixable problem? Maybe, though I agree it seems hard. One way to show you're more right than others (at least to other rationalists) might be to explicitly describe contradicting evidence and ways in which you could be wrong.
Holden Karnofsky of GiveWell.org interviewed Jasen Murray of SIAI and published his notes (Edit: PDF, thanks lukeprog!), with updates from later conversations. Lots of stuff to take an interest in there - thanks to jsalvatier for drawing our attention to it. One new bit of information stands out in particular: