You should also not revise your belief if you have already counted the evidence on which the expert's beliefs depend.

What if Yudkowsky says that he believes X with probability 90%, and you only believe it with probability 40% - even though you don't believe that he has access to any more evidence than you do?

Also, I agree that in practice there is a "noise level" - i.e. very weak evidence isn't worth paying attention to - but in theory (which was the context of the discussion) I don't believe there is.

You are mistaken. It is exactly the same claim, rephrased to match the circumstances at most -- containing exactly the same assertion: "Authorities on a given topic can be mistaken; and as such no appeal to an authority can be, absent any other actual supporting materials, treated as a rational motive for belief. In all such cases, it is those supporting materials which comprise the nature of the argument." In the specific case you are taking as supporting evidence your justified belief that the individual had in fact directly observed the event, and the justified belief that the individual would relate it to you honestly. Those, together, comprise justification for the belief.

So in other words, you would like to distinguish between “appeal to authority” and “supporting materials” as though when someone refers you to the sayings of some authority, they expect you to consider these sayings as data purely “in themselves”, separately from whatever reasons you may have for believing that the sayings of that authority are evidentially entangled with whatever you want to know about.

Firstly, “appeal to authority” is generally taken simply to mean the act of referring someone to an authority during an argument about something – there is no connotation that this authority has no evidential entanglement with the subject of the argument, which would be bizarre (why would anyone refer you to him in that case?)

Secondly, if someone makes a statement about something, that in itself implies that there is evidential entanglement between that thing and their statement – i.e. the thing they are talking about is part of the chain of cause and effect (however indirect) that led to the person eventually making a statement about it (otherwise we have to postulate a very big coincidence). Therefore the idea that someone could make a statement about something without there being any evidential entanglement between them and it (which is necessary in order for it to be true that you should not update your belief at all based on their statement) is implausible in the extreme.

You started off by using “appeal to authority” in the normal way, but now you are attempting to redefine it in a nonsensical way so as to avoid admitting that you were mistaken (NB: there is no shame in being mistaken).

If you have read Harry Potter and the Methods of Rationality, you may remember the bit where Quirrell demonstrates “how to lose” as an important lesson in magical combat. Correspondingly, in future I would advise you not to create edifices of nonsense when it would be easier just to admit your mistake. Debate is after all a constructive enterprise, not a battle for tribal status in which it is necessary to save face at all costs.

I’ll say again that the error that is facilitating your confusion about beliefs and evidence is the idea that belief is a binary quality – I believe, or I do not believe – when in fact you believe with probability 90%, or 55%, or 12.2485%. The way that the concept of belief and the phrase “I believe X” is used in ordinary conversation may mislead people on this point, but that doesn’t change the facts of probability theory.

This allows you to think that the question is whether you are “justified” in believing proposition X in light of evidence Y, when the right question to be asking is “how has my degree of belief in proposition X changed as a result of this evidence?” You are reluctant to accept that someone’s mere assertions can be evidence in favour of some proposition because you have in mind the idea that evidence must always be highly persuasive (so as to change your belief status in a binary way from “unjustified” to “justified”), otherwise it isn’t evidence – whereas actually, evidence is still evidence even if it only causes you to shift your degree of belief in a proposition from 1% to 1.2%.

See also

Evidence itself can be mistaken. If your theory says an event is rare, and it happens, then that is evidence against the theory. If the theory is correct, it should be overwhelmed by evidence for the theory. If the statements of experts statistically correlate with reality, then you should update on the statements of experts until they are screened off by evidence/arguments you have looked at more directly or the statements of other experts.

I'm really confused about why you're not understanding this. Authorities are reliable to different degrees about different things. If I tell you I'm wearing an orange shirt that is clearly evidence that I am wearing an orange shirt. If a physicist tells you nothing can accelerate past the speed of light that is evidence that nothing can accelerate past the speed of light. Now, because people can be untrustworthy there are many circumstances in which witness testimony is less reliable than personal observation. But it would be rather bothersome to upload a picture of me in my shirt to you. It can also be difficult to explain special relativity and the evidence for it in a short time span. In cases like these we must settle for the testimony of authorities. This does not make appeals to authority invalid.

Now of course you might have some evidence that suggests I am not a reliable reporter of the color of my shirt. Perhaps I have lied to you many times before or I have incentives to be dishonest. In these cases it is appropriate to discount my testimony to the degree that I am unreliable. But this is not a special problem with appeals to authority. If you have reason to think you are hallucinating, perhaps because of the LSD you took an hour ago, you should appropriately discount your eyes telling you that the trees are waving at you.

Now since appeals to authority, like other kinds of sources of information, are not 100% reliable it makes sense to discuss the judgment of authorities in detail. Even if Eliezer is a reliable authority on lots of things it is a good idea to examine his reasoning. In this regard you are correct to demand arguments beyond "Eliezer says so". But it is none the less not the case that "appeals to authority are always fallacious". On the contrary modern science would be impossible without them since no one can possibly make all the observations necessary to support the reliability of a modern scientific theory.