Here's where your understanding, by the way, is breaking down: the difference between practical behavior and valid behavior. Bayesian rationality in particular is highly susceptible to this problem, and it's one of my main objections to the system in principle: that it fails to parse the process of forming beliefs from the process of confirming truth.
Here's what may be tripping you up. Very often it doesn't even make sense for humans to pay attention to small bits of evidence, because we can't really process them very effectively. So for most bits of tiny evidence (such as most very weak appeals to authority) often the correct action given our limited processing capability is to simply ignore them. But this doesn't make them not evidence.
Say for example, I have a conjecture about all positive integers n, and I check it for every n up to 10^6. If I verify it for 10^6+1, working out how much more certain I should be is really tough, so I only treat the evidence in aggregate. Similarly, if I have a hypothesis that all ravens ravens are black, and I've checked it for a thousand ravens, I should be more confident when I checked the 1001st raven and find it is black. But actually doing that update is difficult.
The question then becomes for any given appeal to authority, how reliable is that appeal?
Note that in almost any field there's going to be some degree of relying on such appeals. In math for examples, there's a very deep result called the classification of finite simple groups. The proof of the full result is thousands of pages spread across literally hundreds of separate papers. It is possible at some point that some people have really looked at almost the whole thing, but the vast majority of people who use the classification certainly have not. Relying on the classification is essentially an appeal to authority.
LessWrongers as a group are often accused of talking about rationality without putting it into practice (for an elaborated discussion of this see Self-Improvement or Shiny Distraction: Why Less Wrong is anti-Instrumental Rationality). This behavior is particularly insidious because it is self-reinforcing: it will attract more armchair rationalists to LessWrong who will in turn reinforce the trend in an affective death spiral until LessWrong is a community of utilitarian apologists akin to the internet communities of anorexics who congratulate each other on their weight loss. It will be a community where instead of discussing practical ways to "overcome bias" (the original intent of the sequences) we discuss arcane decision theories, who gets to be in our CEV, and the most rational birthday presents (sound familiar?).
A recent attempt to counter this trend or at least make us feel better about it was a series of discussions on "leveling up": accomplishing a set of practical well-defined goals to increment your rationalist "level". It's hard to see how these goals fit into a long-term plan to achieve anything besides self-improvement for its own sake. Indeed, the article begins by priming us with a renaissance-man inspired quote and stands in stark contrast to articles emphasizing practical altruism such as "efficient charity"
So what's the solution? I don't know. However I can tell you a few things about the solution, whatever it may be:
Whatever you may decide to do, be sure it follows these principles. If none of your plans align with these guidelines then construct a new one, on the spot, immediately. Just do something: every moment you sit hundreds of thousands are dying and billions are suffering. Under your judgement your plan can self-modify in the future to overcome its flaws. Become an optimization process; shut up and calculate.
I declare Crocker's rules on the writing style of this post.