Community

The Center For Applied Rationality (CFAR) checklist is a heuristic for assessing the admissibility of one's own testimony. 

What of the challenge of evaluating the testimony of others?

Slapping the label of a bias on a situation?

Arguing at the object level by provision of evidence to the contrary?

This risks Gish Gallop. For those who prefer to pick their battles, I commisioned this post of my time, a structural intervention into the information ecosystem.

We need not event the wheel, for legal theorists have researched this issue for years, while practitioners and courts have identified heuristics useful to lay people interested in this field. 

Precedent 

The Daubert standard provides a rule of evidence regarding the admissibility of expert witnesses' testimony during United States federal legal proceedings. Pursuant to this standard, a party may raise a Daubert motion, which is a special case of motion in limine raised before or during trial to exclude the presentation of unqualified evidence to the jury. The Daubert trilogy refers to the three United States Supreme Court cases that articulated the Daubert standard:

-https://en.wikipedia.org/wiki/Daubert_standard

Further reading on the case is available here on Google Scholar

Practice

How can this be applied in practice? 

What is the first principle of skepticism. It's effectively synonymous: 'question'

What question? This isn't the 5 W's of primary school, after all.

I have summarized critical questions to a reading here to get the ball rolling:

Issues to consider when contesting and evaluating expert opinion evidence

 

A. Relevance (on the voir dire)

I accept that you are highly qualified and have extensive experience, but how do we know that your level of performance regarding . . . [the task at hand — eg, voice comparison] is actually better than that of a lay person (or the jury)?

What independent evidence... [such as published studies of your technique and its accuracy] can you direct us to that would allow us to answer this question?

What independent evidence confirms that your technique works?

Do you participate in a blind proficiency testing program?

Given that you undertake blind proficiency exercises, are these exercises also given to lay persons to determine if there are significant differences in results, such that your asserted expertise can be supported?

B. Validation 

Do you accept that techniques should be validated?

Can you direct us to specific studies that have validated the technique that you used?

What precisely did these studies assess (and is the technique being used in the same way in this case)?

Have you ever had your ability formally tested in conditions where the correct answer was known? (ie, not a previous investigation or trial)

Might different analysts using your technique produce different answers?

Has there been any variation in the result on any of the validation or proficiency tests you know of or participated in?

Can you direct us to the written standard or protocol used in your analysis?

Was it followed?

C. Limitations and errors

Could you explain the limitations of this technique?

Can you tell us about the error rate or potential sources of error associated with this technique?

Can you point to specific studies that provide an error rate or an estimation of an error rate for your technique?

How did you select what to examine?

Were there any differences observed when making your comparison . . . [eg, between two fingerprints], but which you ultimately discounted? On what basis were these discounted?

Could there be differences between the samples that you are unable to observe?

Might someone using the same technique come to a different conclusion?

Might someone using a different technique come to a different conclusion?

Did any of your colleagues disagree with you?

Did any express concerns about the quality of the sample, the results, or your interpretation?

Would some analysts be unwilling to analyse this sample (or produce such a confident opinion)?

...

D Personal proficiency 

...

Have you ever had your own ability... [doing the specific task/using the technique] tested in conditions where the correct answer was known?

If not, how can we be confident that you are proficient?

If so, can you provide independent empirical evidence of your performance?

E Expressions of opinion 

...

Can you explain how you selected the terminology used to express your opinion? Is it based on a scale or some calculation?

If so, how was the expression selected?

Would others analyzing the same material produce similar conclusions, and a similar strength of opinion? How do you know?

Is the use of this terminology derived from validation studies?

Did you report all of your results?

You would accept that forensic science results should generally be expressed in non-absolute terms?

More

 

Hey all - I typed this out to help me understand, well... how to understand things:

 

Mental clarity is the ability to read reality accurately. 

 

I don't mean being able to look at the complete objective picture of an event, as you don't have any direct access to that. I'm talking about the ability to read the data presented by your subjective experience: thoughs, sights, sounds, etc. Once you get a clear picture of what that data is, you can then go on and use it to build or falsify your ideas about the world.

This post will focus on the "getting a clear picture" part.

I use the word "read" because it's no different than reading from a book, or from these words. When you read a book, you are actually curious as to what the words are saying. You wouldn't read anything into it that's not there, which would be counterproductive to your understanding.

 

You just look at the words plainly, and through this your mind automatically recognizes and presents the patterns: the meaning of the sentences, their relation to the topic, the visual imagery associated with them, all of that. If you want to know a truth about reality, just look at it and read what's there.

Want to know what the weather's like? Look outside - read what's going on.

Want to know if the Earth revolves around the Sun, or vice versa? Look at the movement of the planets, read what they're doing, see which theory fits better.

Want to check if your beliefs about the world are correct? Take one, read the reality that the belief tries to correspond to, and see how well they compare.

This is the root of all science and all epiphanies.

But if it's so simple and obvious, why am I talking about it?

It's not something that we as a species often do. For trivial matters, sure, for science too, but not for our strongly-held opinions. Not for the beliefs and positions that shape our self-image, make us feel good/comfortable, or get us approval. Not for our political opinions, religious ideas, moral judgements, and little white lies.

If you were utterly convinced that your wife was faithful, moreso, if you liked to think of her in that way, and your friend came along and said she was cheating on you, you'd be reluctant to read reality and check if that's true. Doing this would challenge your comfort and throw you into an unknown world with some potentially massive changes. It would be much more comforting to rationalize why she still might be faithful, than to take one easy look at the true information. It would also more damaging.

Delusion is reading into reality things which aren't there. Telling yourself that everything's fine when it obviously isn't, for example. It's the equivalent of looking at a book about vampires and jumping to the conclusion that it's about wizards.

Sounds insane. You do it all the time. You'll catch yourself if you're willing to read the book of your own thoughts: flowing through your head, in plain view, is a whole mess of opinions and ideas of people, places, and positions you've never even encountered. Crikey!

That mess is incredibly dangerous to have. Being a host to unchecked or false beliefs about the world is like having a faulty map of a terrain: you're bound to get lost or fall off a cliff. Reading the terrain and re-drawing the map accordingly is the only way to accurately know where you're going. Having an accurate map is the only way to achieve your goals.

So you want to develop mental clarity? Be less confused, or more successful? Have a better understanding of the world, the structure of reality, or the accuracy of your ideas? 

Just practice the accurate reading of what's going on. Surrender the content of your beliefs to the data gathered by your reading of reality. It's that simple.

 

It can also be scary, especially when it comes to challenging your "personal" beliefs. It's well worth the fear, however, as a life built on truth won't crumble like one built on fiction.

 

Truth doesn't crumble.

 

Stay true.

Further reading:

Stepvhen from Burning true on truth vs. fantasy.

Kevin from Truth Strike on why this skill is important to develop.