Community

The Center For Applied Rationality (CFAR) checklist is a heuristic for assessing the admissibility of one's own testimony. 

What of the challenge of evaluating the testimony of others?

Slapping the label of a bias on a situation?

Arguing at the object level by provision of evidence to the contrary?

This risks Gish Gallop. For those who prefer to pick their battles, I commisioned this post of my time, a structural intervention into the information ecosystem.

We need not event the wheel, for legal theorists have researched this issue for years, while practitioners and courts have identified heuristics useful to lay people interested in this field. 

Precedent 

The Daubert standard provides a rule of evidence regarding the admissibility of expert witnesses' testimony during United States federal legal proceedings. Pursuant to this standard, a party may raise a Daubert motion, which is a special case of motion in limine raised before or during trial to exclude the presentation of unqualified evidence to the jury. The Daubert trilogy refers to the three United States Supreme Court cases that articulated the Daubert standard:

-https://en.wikipedia.org/wiki/Daubert_standard

Further reading on the case is available here on Google Scholar

Practice

How can this be applied in practice? 

What is the first principle of skepticism. It's effectively synonymous: 'question'

What question? This isn't the 5 W's of primary school, after all.

I have summarized critical questions to a reading here to get the ball rolling:

Issues to consider when contesting and evaluating expert opinion evidence

 

A. Relevance (on the voir dire)

I accept that you are highly qualified and have extensive experience, but how do we know that your level of performance regarding . . . [the task at hand — eg, voice comparison] is actually better than that of a lay person (or the jury)?

What independent evidence... [such as published studies of your technique and its accuracy] can you direct us to that would allow us to answer this question?

What independent evidence confirms that your technique works?

Do you participate in a blind proficiency testing program?

Given that you undertake blind proficiency exercises, are these exercises also given to lay persons to determine if there are significant differences in results, such that your asserted expertise can be supported?

B. Validation 

Do you accept that techniques should be validated?

Can you direct us to specific studies that have validated the technique that you used?

What precisely did these studies assess (and is the technique being used in the same way in this case)?

Have you ever had your ability formally tested in conditions where the correct answer was known? (ie, not a previous investigation or trial)

Might different analysts using your technique produce different answers?

Has there been any variation in the result on any of the validation or proficiency tests you know of or participated in?

Can you direct us to the written standard or protocol used in your analysis?

Was it followed?

C. Limitations and errors

Could you explain the limitations of this technique?

Can you tell us about the error rate or potential sources of error associated with this technique?

Can you point to specific studies that provide an error rate or an estimation of an error rate for your technique?

How did you select what to examine?

Were there any differences observed when making your comparison . . . [eg, between two fingerprints], but which you ultimately discounted? On what basis were these discounted?

Could there be differences between the samples that you are unable to observe?

Might someone using the same technique come to a different conclusion?

Might someone using a different technique come to a different conclusion?

Did any of your colleagues disagree with you?

Did any express concerns about the quality of the sample, the results, or your interpretation?

Would some analysts be unwilling to analyse this sample (or produce such a confident opinion)?

...

D Personal proficiency 

...

Have you ever had your own ability... [doing the specific task/using the technique] tested in conditions where the correct answer was known?

If not, how can we be confident that you are proficient?

If so, can you provide independent empirical evidence of your performance?

E Expressions of opinion 

...

Can you explain how you selected the terminology used to express your opinion? Is it based on a scale or some calculation?

If so, how was the expression selected?

Would others analyzing the same material produce similar conclusions, and a similar strength of opinion? How do you know?

Is the use of this terminology derived from validation studies?

Did you report all of your results?

You would accept that forensic science results should generally be expressed in non-absolute terms?

More

Standard methods of inferring knowledge about the world are based off premises that I don’t know the justifications for. Any justification (or a link to an article or book with one) for why these premises are true or should be assumed to be true would be appreciated.

Here are the premises:

• “One has knowledge of one’s own percepts.” Percepts are often given epistemic privileges, meaning that they need no justification to be known, but I see no justification for giving them epistemic privileges. It seems like the dark side of epistemology to me.

• “One’s reasoning is trustworthy.” If one’s reasoning is untrustworthy, then one’s evaluation of the trustworthiness of one’s reasoning can’t be trusted, so I don’t see how one could determine if one’s reasoning is correct. Why should one even consider one’s reasoning is correct to begin with? It seems like privileging the hypothesis, as there are many different ways one’s mind could work, and presumably only a very small proportion of possible minds would be remotely valid reasoners.

• “One’s memories are true.” Though one’s memories of how the world works gives a consistent explanation of why one is perceiving one’s current percepts, a perhaps simpler explanation is that the percepts one are currently experiencing are the only percepts one has ever experienced, and one’s memories are false. This hypothesis is still simple, as one only needs to have a very small number of memories, as one can only think of a small number of memories at any one time, and the memory of having other memories could be false as well.

A discussion of four errors which lead to false positives-- neglecting maturation (that brains change with time, even without intervention, learning effects (people who take a test more than once get better at it), regression to the mean (people who are unusually good or bad at something will probably have a more average score on subsequent attempts), and the placebo effect.

The link above is a summary of a lecture which isn't playing for me, so any further information about the lecture would be greatly appreciated.

Since it came to my attention that signing up for cryonics is not as pointless as I'd once thought, I've been pondering how to sell my dad on the idea.

This is somewhat urgent for a couple of reasons. First, he's already pushing sixty and would meet increased resistance in acquiring another life insurance policy at a much steeper rate than myself. Second, even given his age, he could afford to sign us both up easily, and after some consideration, convincing him seems like the more efficient path to being signed up myself than trying to arrange it for myself alone. And, well, he's my dad, and while he has his flaws, he's kind of awesome.

The sticking point, I can easily predict, will be getting passed his Cached Skepticism towards the concept of Cryonics. He is proud of being Skeptical; it's important to his self-concept, so I need to hit him with an opening that he can't easily dismiss. I would predict that if I simply linked him to Alcor's webpage, absolutely nothing would happen. I need something that will motivate him to investigate.

The frustrating part, is that I'm quite sure that if he'd never heard of cryonics before, he'd have no resistance to the idea beyond the basic pain of dispelling the emotional numbness towards a Certain Doom by suggesting it might not be certain after all. Unless my model of him is very much inaccurate, his Cached Skepticism of Cryonics is the only notable obstacle. I would appreciate any recommendations on which if any articles in particular would be best for initially getting through that.

An article at The Edge has scientific experts in various fields give their favorite examples of theories that were wrong in their fields. Most relevantly to Less Wrong, many of those scientists discuss what their disciplines did that was wrong which resulted in the misconceptions. For example, Irene Pepperberg not surprisingly discusses the failure for scientists to appreciate avian intelligence. She emphasizes that this failure resulted from a combination of different factors, including the lack of appreciation that high level cognition could occur without the mammalian cortex, and that many early studies used pigeons which just aren't that bright.