Continuation of:  Argument Screens Off Authority

In the art of rationality there is a discipline of closeness-to-the-issue—trying to observe evidence that is as near to the original question as possible, so that it screens off as many other arguments as possible.

The Wright Brothers say, "My plane will fly."  If you look at their authority (bicycle mechanics who happen to be excellent amateur physicists) then you will compare their authority to, say, Lord Kelvin, and you will find that Lord Kelvin is the greater authority.

If you demand to see the Wright Brothers' calculations, and you can follow them, and you demand to see Lord Kelvin's calculations (he probably doesn't have any apart from his own incredulity), then authority becomes much less relevant.

If you actually watch the plane fly, the calculations themselves become moot for many purposes, and Kelvin's authority not even worth considering.

Black Belt Bayesian (aka "steven") tries to explain the asymmetry between good arguments and good authority, but it doesn't seem to be resolving the comments on Reversed Stupidity Is Not Intelligence, so let me take my own stab at it:

Scenario 1:  Barry is a famous geologist.  Charles is a fourteen-year-old juvenile delinquent with a long arrest record and occasional psychotic episodes.  Barry flatly asserts to Arthur some counterintuitive statement about rocks, and Arthur judges it 90% probable.  Then Charles makes an equally counterintuitive flat assertion about rocks, and Arthur judges it 10% probable.  Clearly, Arthur is taking the speaker's authority into account in deciding whether to believe the speaker's assertions.

Scenario 2:  David makes a counterintuitive statement about physics and gives Arthur a detailed explanation of the arguments, including references.  Ernie makes an equally counterintuitive statement, but gives an unconvincing argument involving several leaps of faith.  Both David and Ernie assert that this is the best explanation they can possibly give (to anyone, not just Arthur).  Arthur assigns 90% probability to David's statement after hearing his explanation, but assigns a 10% probability to Ernie's statement.

It might seem like these two scenarios are roughly symmetrical: both involve taking into account useful evidence, whether strong versus weak authority,  or strong versus weak argument.

        "...then our people on that time-line went to work with corrective action.  Here."
        He wiped the screen and then began punching combinations.  Page after page appeared, bearing accounts of people who had claimed to have seen the mysterious disks, and each report was more fantastic than the last.
        "The standard smother-out technique," Verkan Vall grinned.  "I only heard a little talk about the 'flying saucers,' and all of that was in joke.  In that order of culture, you can always discredit one true story by setting up ten others, palpably false, parallel to it."
                —H. Beam Piper, Police Operation

Piper had a point.  Pers'nally, I don't believe there are any poorly hidden aliens infesting these parts.  But my disbelief has nothing to do with the awful embarrassing irrationality of flying saucer cults—at least, I hope not.

You and I believe that flying saucer cults arose in the total absence of any flying saucers.  Cults can arise around almost any idea, thanks to human silliness.  This silliness operates orthogonally to alien intervention:  We would expect to see flying saucer cults whether or not there were flying saucers.  Even if there were poorly hidden aliens, it would not be any less likely for flying saucer cults to arise.  p(cults|aliens) isn't less than p(cults|~aliens), unless you suppose that poorly hidden aliens would deliberately suppress flying saucer cults.  By the Bayesian definition of evidence, the observation "flying saucer cults exist" is not evidence against the existence of flying saucers.  It's not much evidence one way or the other.

This is an application of the general principle that, as Robert Pirsig puts it, "The world's greatest fool may say the Sun is shining, but that doesn't make it dark out."

Followup to:  Guessing the Teacher's Password, Artificial Addition

A classic paper by Drew McDermott, "Artificial Intelligence Meets Natural Stupidity", criticized AI programs that would try to represent notions like happiness is a state of mind using a semantic network:

STATE-OF-MIND
     ^
     | IS-A
     |
HAPPINESS

And of course there's nothing inside the "HAPPINESS" node; it's just a naked LISP token with a suggestive English name.

So, McDermott says, "A good test for the disciplined programmer is to try using gensyms in key places and see if he still admires his system.  For example, if STATE-OF-MIND is renamed G1073..." then we would have IS-A(HAPPINESS, G1073) "which looks much more dubious."

Or as I would slightly rephrase the idea:  If you substituted randomized symbols for all the suggestive English names, you would be completely unable to figure out what G1071(G1072, 1073) meant.  Was the AI program meant to represent hamburgers?  Apples?  Happiness?  Who knows?  If you delete the suggestive English names, they don't grow back.