If you're against vaccines, but you've seen the fact-checker bot be correct 99 other times, then you might give credence to its claims.

That's subject to Goodhart's Law. If you start judging bots by their behavior in other cases, people will take advantage of your judging process by specifically designing bots to do poor fact checking on just a couple of issues, thus making it useless to judge bots based on their behavior in other cases.

(Of course, they won't think of it that way, they'll think of it as "using our influence to promote social change" or some such. But it will happen, and has already happened for non-bot members of the media.)