This is a post about evaluating possible experts in fields you're not in, where you are somewhat qualified to evaluate their claims.

I try to think along these lines when talking to other engineers about their jobs, but also when figuring out which of two people has got their story straight in online drama.

I frame my model using competence claims, but I think you can use it whenever people are talking about extraordinary events that they witnessed and you haven't, as long as you've got a mental model of what people are likely to lie or exaggerate about.

You can even use my lines of reasoning for legal content once you know what some of the common heterodox legal takes are. (For instance, "fire in a crowded theatre" is not current precedent, but many people act like it is.)

I think my rules of thumb are pretty good, but I really recommend doing diligent research before you boost anything or publicly take a side in anything! When you do that, people other than you have the potential to be hurt or misled.

OK, here's my idea

When someone claims to be fourish standard deviations off the mean -- 160 IQ, for instance -- they're probably lying or mistaken.

Most of the time, when someone says "I'm competent," you have to weigh the odds that they really are against the odds that they're completely wrong. Especially when it's a really tall claim!

Many statements aren't claims of competence per se, but they have strong implications about competence. If you're in France and some guy comes in from England and you ask, "how did you get here?" then  four worlds are possible. I'll assign them arbitrary percentage probabilities to make my point:

• (9%) They say "I swam," but they didn't
• (1%) They say "I swam," and they did
• (89%) They say "I didn't swim," and they didn't
• (1%) They say "I didn't swim," and they did

Well, based on these numbers and Bayes' Law:

• 2% of the time, they swam
• If they say they swam, then 10% of the time, they swam

(The numbers are arbitrary, but you get the same conclusions as long as you assume that swimming is rare and people rarely lie in a way that implies they're less competent than they actually are.)

So this is all me building up to a general observation, which is that certain claims -- usually claims that sound crazy -- are equally likely to come from a competent or an incompetent person, because they remove the middle.

For instance, in software, you're likely to hear these statements from some people:

• The author of this codebase is an idiot.
• We should build our own engine.
• I hate ORMs.

These statements can strongly suggest "I'm unwilling to adapt to other people's rules" which is a basic career skill that a person might be lacking, or "I don't think other people can do things better than me" -- which in the case of competent people is accurate and in the case of incompetent people is very wrong. 

If you frame it Monty Hall-ishly, it looks like this. Before someone says something like that, you have the options "mundane," "sharp," "crazy" to classify them. Once they've said something like that, you know they're weird, so you eliminate "mundane."

(Of course, some claims are so wrong that they immediately eliminate "sharp" too -- you don't need to keep an open mind around Flat Earthers.)

Sharp, crazy

Well, they put their foot in it, so now you have another question to answer: are they sharp or are they crazy?

From there it's wise to look at one or two other data points.

You can test for Chesterton's Fence:

• Why do you think the code looks like this?
• How do you feel about Unity's scene system?
• Why do you think other people like ORMs?

You can probe for basic experience:

• How long have you worked with this code?
• What engines have you used?
• What ORMs have you used?

You can be Socratic:

• Why do you think that?

These questions are often terribly uninformative for mundane people (or people who have sophisticated social strategies) -- you will likely get coy answers or the other person will try to sniff out your opinion and repeat it.

But because you already know the other person is willing to say things that are weird and socially unacceptable, that's less likely to happen -- meaning you can often use these kinds of questions to quickly differentiate sharp people from crazy people.

(Note that while the last four questions may seem basic, very brazen liars will often be unable to convincingly answer really basic questions, especially when they could be convicted of fraud if caught, or when they are also lying to themselves. For instance, health scammer Belle Gibson couldn't substantiate any of her claimed personal history when she was finally asked.)

There's a minor subcategory of people who will recant their wild opinion if questioned. You can sometimes tell how crazy they are by how meek they look while recanting, but this is pretty unreliable, in my opinion, because it's a strong tell that people care about image.

Mundane, sharp

If someone's claiming to be really, really sharp, and you don't have any other evidence suggesting they are? Well, sharp people are rare, so you can just assume "crazy." (But don't say it out loud, because they'll hate you!)

But not everyone who's sharp will say something that eliminates "mundane" as an option. Even if they do, you're not likely to recognize it. This makes them harder to identify than crazy people.

Sharp people typically have at least one wildly heterodox opinion. As far as being right is concerned, if sharp people had the same opinions as everyone else, then anyone with the consensus view would be right about as often as them.

But much of smarts is technique; great pianists and chess players likely agree on 90% of theory, even though it's obvious they do something different from everyone else. Their heterodox opinions are likely to be about niche issues you only recognize once you have the broad base of concepts that comes from knowing a lot of technique. Or, in a completely non-linguistic way, they just understand things better than you.

It's also pretty common to misrecognize actually heterodox opinions as dull conservative ones. On a lot of issues, the prevailing opinion changes kind of arbitrarily for social reasons, but people who have an actual reason for their preference are going to be perceived as stodgy or unwilling to follow the times when their preference goes out of fashion. 

Additionally, sharp people may not voice their heterodox opinions unless they're comfortable around you.

I think that unfortunately, in a situation where all you have is a glancing look at what people are saying and no access to their careers or personal history, you are probably not going to be able to tell with certainty if they're hiding deep insight.

My best recommendation is to see if they're saying things that (1) don't make sense to you (2) obviously aren't intended to impress; if so, presume they could know what they're talking about, while looking for evidence that they don't.

The nice thing about loud clashy arguments between people you don't know is that usually everyone says something that sounds crazy eventually, so you're less likely to wind up on this branch of the fork. (The not-so-nice thing is that people who get into online drama are a lot less likely to be right about basically everything.)

(BTW: I migrated this from my personal offsite blog! I hope this doesn't violate any norms.)