You should also not revise your belief if you have already counted the evidence on which the expert's beliefs depend.

What if Yudkowsky says that he believes X with probability 90%, and you only believe it with probability 40% - even though you don't believe that he has access to any more evidence than you do?

Also, I agree that in practice there is a "noise level" - i.e. very weak evidence isn't worth paying attention to - but in theory (which was the context of the discussion) I don't believe there is.

You are mistaken. It is exactly the same claim, rephrased to match the circumstances at most -- containing exactly the same assertion: "Authorities on a given topic can be mistaken; and as such no appeal to an authority can be, absent any other actual supporting materials, treated as a rational motive for belief. In all such cases, it is those supporting materials which comprise the nature of the argument." In the specific case you are taking as supporting evidence your justified belief that the individual had in fact directly observed the event, and the justified belief that the individual would relate it to you honestly. Those, together, comprise justification for the belief.

So in other words, you would like to distinguish between “appeal to authority” and “supporting materials” as though when someone refers you to the sayings of some authority, they expect you to consider these sayings as data purely “in themselves”, separately from whatever reasons you may have for believing that the sayings of that authority are evidentially entangled with whatever you want to know about.

Firstly, “appeal to authority” is generally taken simply to mean the act of referring someone to an authority during an argument about something – there is no connotation that this authority has no evidential entanglement with the subject of the argument, which would be bizarre (why would anyone refer you to him in that case?)

Secondly, if someone makes a statement about something, that in itself implies that there is evidential entanglement between that thing and their statement – i.e. the thing they are talking about is part of the chain of cause and effect (however indirect) that led to the person eventually making a statement about it (otherwise we have to postulate a very big coincidence). Therefore the idea that someone could make a statement about something without there being any evidential entanglement between them and it (which is necessary in order for it to be true that you should not update your belief at all based on their statement) is implausible in the extreme.

You started off by using “appeal to authority” in the normal way, but now you are attempting to redefine it in a nonsensical way so as to avoid admitting that you were mistaken (NB: there is no shame in being mistaken).

If you have read Harry Potter and the Methods of Rationality, you may remember the bit where Quirrell demonstrates “how to lose” as an important lesson in magical combat. Correspondingly, in future I would advise you not to create edifices of nonsense when it would be easier just to admit your mistake. Debate is after all a constructive enterprise, not a battle for tribal status in which it is necessary to save face at all costs.

I’ll say again that the error that is facilitating your confusion about beliefs and evidence is the idea that belief is a binary quality – I believe, or I do not believe – when in fact you believe with probability 90%, or 55%, or 12.2485%. The way that the concept of belief and the phrase “I believe X” is used in ordinary conversation may mislead people on this point, but that doesn’t change the facts of probability theory.

This allows you to think that the question is whether you are “justified” in believing proposition X in light of evidence Y, when the right question to be asking is “how has my degree of belief in proposition X changed as a result of this evidence?” You are reluctant to accept that someone’s mere assertions can be evidence in favour of some proposition because you have in mind the idea that evidence must always be highly persuasive (so as to change your belief status in a binary way from “unjustified” to “justified”), otherwise it isn’t evidence – whereas actually, evidence is still evidence even if it only causes you to shift your degree of belief in a proposition from 1% to 1.2%.

See also

Evidence itself can be mistaken. If your theory says an event is rare, and it happens, then that is evidence against the theory. If the theory is correct, it should be overwhelmed by evidence for the theory. If the statements of experts statistically correlate with reality, then you should update on the statements of experts until they are screened off by evidence/arguments you have looked at more directly or the statements of other experts.

I'm really confused about why you're not understanding this. Authorities are reliable to different degrees about different things. If I tell you I'm wearing an orange shirt that is clearly evidence that I am wearing an orange shirt. If a physicist tells you nothing can accelerate past the speed of light that is evidence that nothing can accelerate past the speed of light. Now, because people can be untrustworthy there are many circumstances in which witness testimony is less reliable than personal observation. But it would be rather bothersome to upload a picture of me in my shirt to you. It can also be difficult to explain special relativity and the evidence for it in a short time span. In cases like these we must settle for the testimony of authorities. This does not make appeals to authority invalid.

Now of course you might have some evidence that suggests I am not a reliable reporter of the color of my shirt. Perhaps I have lied to you many times before or I have incentives to be dishonest. In these cases it is appropriate to discount my testimony to the degree that I am unreliable. But this is not a special problem with appeals to authority. If you have reason to think you are hallucinating, perhaps because of the LSD you took an hour ago, you should appropriately discount your eyes telling you that the trees are waving at you.

Now since appeals to authority, like other kinds of sources of information, are not 100% reliable it makes sense to discuss the judgment of authorities in detail. Even if Eliezer is a reliable authority on lots of things it is a good idea to examine his reasoning. In this regard you are correct to demand arguments beyond "Eliezer says so". But it is none the less not the case that "appeals to authority are always fallacious". On the contrary modern science would be impossible without them since no one can possibly make all the observations necessary to support the reliability of a modern scientific theory.

No. What you should do is ask for a justification of the belief. If you do not have the resources available to you to do so, you can fail-over to the trust system and simply accept the physicist's statement unexamined -- but utilization of the trust-system is an admission of failure to have justified beliefs.

If you want to increase the reliability of your probability estimate, you should ask for a justification. But if you do not increase your probability estimate contingent on the physicist's claim until you receive information on how he established that belief, then you are mistreating evidence. You don't treat his claim as evidence in addition to to evidence on which it was conditioned, you treat it as evidence of the evidence on which it was conditioned. Once you know the physicist's belief, you cannot expect to raise your confidence in that belief upon receiving information on how he came to that conclusion. You should assign weight to his statement according to how much evidence you would expect a physicist in his position to have if he were making such a statement, and then when you learn what evidence he has you shift upwards or downwards depending on how the evidence compares to your expectation. If you revised upwards on the basis of the physicist's say-so, and then revised further upwards based on his having about as much evidence as you would expect, that would be double-counting evidence, but if you do not revise upwards based on the physicist's claim in the first place, that would be assuming zero correlation of his statement with reality.

Justification of belief cannot be "A person who usually is right in this field claims this is so" but can be "A person who I have reason to believe would have evidence on this matter related to me his assessment of said evidence."

You do not need the person to relate their assessment of the evidence to revise your belief upward based on their statement, you only need to believe that it is more likely that they would make the claim if it were true than if it were not.

The difference here is between having a buddy who is a football buff who tells you what the Sportington Sports beat the Homeland Highlanders by last night -- even though you don't know whether he had access to a means of having said information -- as opposed to the friend you know watched the game who tells you the scores.

Anything that is more likely if a belief is true than if it is false is evidence which should increase your probability estimate of that belief. Have you read An Intuitive Explanation of Bayes' Theorem, or any of the other explanations of Bayesian reasoning on this site?

If you have a buddy who is a football buff who tells you that the Sportington Sports beat the Homeland Highlanders last night, then you should treat this as evidence that the Sportington Sports won, weighted according to your estimate of how likely his claim is to correlate with reality. If you know that he watched the game, you're justified in assuming a very high correlation with reality (although you also have to condition your estimate on information aside from whether he is likely to know, such as how likely he is to lie.) If you do not know that he watched the game last night, you will have a different estimate of the strength of his claim's correlation with reality.