Simply put, no. No it is not. Not unless the physicist can provide a reason to believe he is correct. Now, in common practice we assume that he can -- but only because it is normal for an expert in a given field to actually be able to do this.
That's what makes the physicist an authority. If something is a reliable source of information "in practice" then it is a reliable source of information. Obviously if the physicist turns out not to know what she is talking about then beliefs based on that authority's testimony turn out to be wrong.
Here's where your understanding, by the way, is breaking down: the difference between practical behavior and valid behavior. Bayesian rationality in particular is highly susceptible to this problem, and it's one of my main objections to the system in principle: that it fails to parse the process of forming beliefs from the process of confirming truth.
The validity of a method is it's reliability.
No researcher could ever get away with saying, "Dr. Knowsitall states that X is true -- not without providing a citation of a paper where Dr. Knowsitall demonstrated that belief was valid."
The paper where Dr. Knowsitalll demonstrated that belief is simply his testimony regarding what happened in a particular experiment. It is routine for that researcher to not have personally duplicated prior experiments before building on them. The publication of experimental procedures is of course crucial for maintaining high standards of reliability and trustworthiness in the sciences. But ultimately no one can check the work of all scientists and therefore trust is necessary.
Here is an argument from authority for you: This idea of appeals to authority being legitimate isn't some weird Less Wrong, Bayesian idea. It is standard, rudimentary logic. You don't know what you're talking about.
LessWrongers as a group are often accused of talking about rationality without putting it into practice (for an elaborated discussion of this see Self-Improvement or Shiny Distraction: Why Less Wrong is anti-Instrumental Rationality). This behavior is particularly insidious because it is self-reinforcing: it will attract more armchair rationalists to LessWrong who will in turn reinforce the trend in an affective death spiral until LessWrong is a community of utilitarian apologists akin to the internet communities of anorexics who congratulate each other on their weight loss. It will be a community where instead of discussing practical ways to "overcome bias" (the original intent of the sequences) we discuss arcane decision theories, who gets to be in our CEV, and the most rational birthday presents (sound familiar?).
A recent attempt to counter this trend or at least make us feel better about it was a series of discussions on "leveling up": accomplishing a set of practical well-defined goals to increment your rationalist "level". It's hard to see how these goals fit into a long-term plan to achieve anything besides self-improvement for its own sake. Indeed, the article begins by priming us with a renaissance-man inspired quote and stands in stark contrast to articles emphasizing practical altruism such as "efficient charity"
So what's the solution? I don't know. However I can tell you a few things about the solution, whatever it may be:
Whatever you may decide to do, be sure it follows these principles. If none of your plans align with these guidelines then construct a new one, on the spot, immediately. Just do something: every moment you sit hundreds of thousands are dying and billions are suffering. Under your judgement your plan can self-modify in the future to overcome its flaws. Become an optimization process; shut up and calculate.
I declare Crocker's rules on the writing style of this post.