One of the red flags was "disagreeing with experts". While all the preceding tools apply here, there's a suite of special options for examining this particular scenario.
I think a formative moment for any rationalist– our “Uncle Ben shot by the mugger” moment, if you will– is the moment you go “holy shit, everyone in the world is fucking insane.”
First, you can say “holy shit, everyone in the world is fucking insane. Therefore, if I adopt the radical new policy of not being fucking insane, I can pick up these giant piles of utility everyone is leaving on the ground, and then I win.”
Second, you can say “holy shit, everyone in the world is fucking insane. However, none of them seem to realize that they’re insane. By extension, I am probably insane. I should take careful steps to minimize the damage I do.”
I want to emphasize that these are not mutually exclusive. In fact, they’re a dialectic (…okay, look, this hammer I found is really neat and I want to find some place to use it).
(I would define a "dialectic" as two superficially opposite statements, both of which are true, in such a way that resolving the apparent paradox forces you to build a usefully richer world-model. I have not run this definition past Ozy, much less Hegel.)
But, speaking first to the basic dichotomy that’s being proposed, the whole point of becoming sane is that your beliefs shouldn’t reflect what sort of person you are. To the extent you’re succeeding, at least, your beliefs should just reflect how the world is.
Good News, Everyone!
I did an empirical test and found no sign of a general factor of trusting your own reasoning.
But I did find lots of disagreement. What are people deciding based on?
Types of Disagreement
Explicit
The expert actually said the opposite of your conclusion.
This is the simplest form of disagreement. The expert could still be wrong, or lying, or communicating unclearly and not actually saying what you think they said. But at least there's no complications from the form of disagreement itself.
The "communicating unclearly" hypothesis should always be high in your probability space. Communication is hard. (It could be spun as "you misunderstood", but this makes the theory less pleasant without altering its substance, so generally don't bother.)
The lying theory is tricky, as it can explain anything. A good lying-based theory should explain why the expert told that particular lie out of millions of possibilities. Without that, the theory contains a contrived co-incidence, and should be penalized accordingly.
With explicit disagreement, you may be able to find the expert's reasoning. If so, try to find the crux and recurse. If you can't follow the reasoning, this is a sign that the expert either understands something important that you don't, or is spouting complete bullshit. Look for signs that the expert has substance, such as real-world accomplishments or specific endorsements from notably perceptive and honest people.
Superlative or Silence
If an expert oncologist says "glycolisis inhibitors are the best available treatment for this sort of cancer" -- and you think it might be better to use oil-drop immersion to sequence the tumor cells, find the root mutation, and CRISPR an aggressive protease into that spot -- then technically you are disagreeing. This is similar to if she is asked for something better than glycolisis inhibitors and says nothing.
But it's not really an active disagreement. Most likely she's never considered the sequence-and-CRISPR possibility.
(Alternative theory: she doesn't consider a therapy to be "available" if it isn't well-tested and endorsed by appropriate parties, even if it is something an agenty patient with moderate wealth could probably obtain. Communication is hard!)
Nobody considers every possibility. What people who try usually do in practice is depend on a large community to do the search. This means that if an individual disagrees with you via superlative or silence, your real disagreement is with the community they depend on.
Implied by Action
An expert takes actions which suggest a disagreement.
A venture capitalist declines to invest in a cancer-curing startup. Does this mean he thinks the cure doesn't work?
Maybe there's other factors in the decision. Maybe he thinks the founder has no head for business, and the company is fatally underestimating regulatory hurtles, but he would invest in the same technology in better hands. Unless you have a very close seat of observation, this looks the same to you.
Or maybe his values are not what you think. Maybe he's very focused on short-term returns, and a company which is five years away from revenue is of no interest to him.
Types of Experts
Individuals
By far the most straightforward.
Do consider how much of an expert the person is on this exact question. Many experts have broad knowledge, which comes at an opportunity cost of depth. Do you know more about medicine than a randomly chosen doctor? Almost certainly not. (Unless you are a doctor yourself, of course.) Do you know more about one particular disease that said doctor doesn't specialize in? Perhaps one that you have? Entirely possible. More about the exact variation which you have? (Cases of a disease are not identical.) Quite likely.
Also consider how much effort the expert put into forming their opinion. Do they have skin in the game? Is it their job to worry about questions like this? (And if so, how dutiful are they?) If this is their way of brushing you off? (If so, respect that decision and go ponder elsewhere.) The further down the list, the more likely they could be wrong because they don't care enough to be right.
Institutions or Communities
Eliezer just wrote an entire book on the subject. You'd think I wouldn't have much to add.
In fact I do. In addition to everything he wrote, consider the size of the community. This is especially relevant to superlative or silence disagreements, where the key question is "Did anybody evaluate this?"
The ratio of particle physicists to types of fundamental particles is about a thousand to one, and the number of ways those particles can interact is pretty limited. The odds that you've thought of a good question about particle physics which no established physicist already considered are pretty low. (Unless you have studied deeply in the field, of course.)
The ratio of entomologists to species of insect is closer to one to a thousand. Asking a good question about insects which no entomologist has seriously considered is pretty easy.
(The ratio of intestinal microbiologists to species in the intestinal microbiome is unknown -- we don't have a good estimate of the latter.)
I suspect this is an important piece of why Eliezer was able to outdo the psychiatric community regarding Seasonal Affective Disorder. True, there are a lot of psychiatric researchers, but there are a lot of conditions for them to study, and SAD isn't a high priority. Once you zoom in to those working on treatmentment-resistant-SAD, the numbers aren't that high, and things fall through the gaps.
An order-of-magnitude fermi estimate can be useful here.
Traditions or Biology
Traditions and biology have something in common: they are formed by evolution.
Evolution does not share your values. You can often tell what its values are, but it's a constant effort to remember this.
It also has a tendency to take undocumented dependencies on circumstances. It can be really hard to figure out which aspects of the environment cause a solution evolution picked to be the ideal one.
One thing I'd add is low-friction. This is especially important for refining odds far from 50%. If someone's offering a bet on "pi will equal 3 tomorrow" and I can buy "no" for $0.96 and get $1.00 when it resolves, but the betting site will take 5% of my winnings as a fee, I'm not going to bet. So the odds on the proposition will stay at the absurdly high 4%.
Tying up money that has been bet counts as a type of friction. If $0.96 will get me $1.00 in a year with no fee, but index funds are getting 5%/year and I can't do both, I have very little reason to take the bet. Labor can also be a type of friction, especially if bet sizes are capped. If I could gain $100 but it would require five hours of wrestling with terrible UIs or APIs (or studying cryptocurrencies), there's little incentive to bet.
Zvi describes how friction like this can drive people away from a market and make it too small to be useful. And that may well be the primary effect. But it can also cause asymmetric distortions, driving all probabilities toward the middle.
Previously: Testing for Rationalization
One of the red flags was "disagreeing with experts". While all the preceding tools apply here, there's a suite of special options for examining this particular scenario.
The "World is Mad" Dialectic
Back in 2015, Ozymandias wrote:
(I would define a "dialectic" as two superficially opposite statements, both of which are true, in such a way that resolving the apparent paradox forces you to build a usefully richer world-model. I have not run this definition past Ozy, much less Hegel.)
To which Eliezer replied in 2017:
Good News, Everyone!
I did an empirical test and found no sign of a general factor of trusting your own reasoning.
But I did find lots of disagreement. What are people deciding based on?
Types of Disagreement
Explicit
The expert actually said the opposite of your conclusion.
This is the simplest form of disagreement. The expert could still be wrong, or lying, or communicating unclearly and not actually saying what you think they said. But at least there's no complications from the form of disagreement itself.
The "communicating unclearly" hypothesis should always be high in your probability space. Communication is hard. (It could be spun as "you misunderstood", but this makes the theory less pleasant without altering its substance, so generally don't bother.)
The lying theory is tricky, as it can explain anything. A good lying-based theory should explain why the expert told that particular lie out of millions of possibilities. Without that, the theory contains a contrived co-incidence, and should be penalized accordingly.
With explicit disagreement, you may be able to find the expert's reasoning. If so, try to find the crux and recurse. If you can't follow the reasoning, this is a sign that the expert either understands something important that you don't, or is spouting complete bullshit. Look for signs that the expert has substance, such as real-world accomplishments or specific endorsements from notably perceptive and honest people.
Superlative or Silence
If an expert oncologist says "glycolisis inhibitors are the best available treatment for this sort of cancer" -- and you think it might be better to use oil-drop immersion to sequence the tumor cells, find the root mutation, and CRISPR an aggressive protease into that spot -- then technically you are disagreeing. This is similar to if she is asked for something better than glycolisis inhibitors and says nothing.
But it's not really an active disagreement. Most likely she's never considered the sequence-and-CRISPR possibility.
(Alternative theory: she doesn't consider a therapy to be "available" if it isn't well-tested and endorsed by appropriate parties, even if it is something an agenty patient with moderate wealth could probably obtain. Communication is hard!)
Nobody considers every possibility. What people who try usually do in practice is depend on a large community to do the search. This means that if an individual disagrees with you via superlative or silence, your real disagreement is with the community they depend on.
Implied by Action
An expert takes actions which suggest a disagreement.
A venture capitalist declines to invest in a cancer-curing startup. Does this mean he thinks the cure doesn't work?
Maybe there's other factors in the decision. Maybe he thinks the founder has no head for business, and the company is fatally underestimating regulatory hurtles, but he would invest in the same technology in better hands. Unless you have a very close seat of observation, this looks the same to you.
Or maybe his values are not what you think. Maybe he's very focused on short-term returns, and a company which is five years away from revenue is of no interest to him.
Types of Experts
Individuals
By far the most straightforward.
Do consider how much of an expert the person is on this exact question. Many experts have broad knowledge, which comes at an opportunity cost of depth. Do you know more about medicine than a randomly chosen doctor? Almost certainly not. (Unless you are a doctor yourself, of course.) Do you know more about one particular disease that said doctor doesn't specialize in? Perhaps one that you have? Entirely possible. More about the exact variation which you have? (Cases of a disease are not identical.) Quite likely.
Also consider how much effort the expert put into forming their opinion. Do they have skin in the game? Is it their job to worry about questions like this? (And if so, how dutiful are they?) If this is their way of brushing you off? (If so, respect that decision and go ponder elsewhere.) The further down the list, the more likely they could be wrong because they don't care enough to be right.
Institutions or Communities
Eliezer just wrote an entire book on the subject. You'd think I wouldn't have much to add.
In fact I do. In addition to everything he wrote, consider the size of the community. This is especially relevant to superlative or silence disagreements, where the key question is "Did anybody evaluate this?"
The ratio of particle physicists to types of fundamental particles is about a thousand to one, and the number of ways those particles can interact is pretty limited. The odds that you've thought of a good question about particle physics which no established physicist already considered are pretty low. (Unless you have studied deeply in the field, of course.)
The ratio of entomologists to species of insect is closer to one to a thousand. Asking a good question about insects which no entomologist has seriously considered is pretty easy.
(The ratio of intestinal microbiologists to species in the intestinal microbiome is unknown -- we don't have a good estimate of the latter.)
I suspect this is an important piece of why Eliezer was able to outdo the psychiatric community regarding Seasonal Affective Disorder. True, there are a lot of psychiatric researchers, but there are a lot of conditions for them to study, and SAD isn't a high priority. Once you zoom in to those working on treatmentment-resistant-SAD, the numbers aren't that high, and things fall through the gaps.
An order-of-magnitude fermi estimate can be useful here.
Traditions or Biology
Traditions and biology have something in common: they are formed by evolution.
Evolution does not share your values. You can often tell what its values are, but it's a constant effort to remember this.
It also has a tendency to take undocumented dependencies on circumstances. It can be really hard to figure out which aspects of the environment cause a solution evolution picked to be the ideal one.
Markets
Zvi has written about this at length.
One thing I'd add is low-friction. This is especially important for refining odds far from 50%. If someone's offering a bet on "pi will equal 3 tomorrow" and I can buy "no" for $0.96 and get $1.00 when it resolves, but the betting site will take 5% of my winnings as a fee, I'm not going to bet. So the odds on the proposition will stay at the absurdly high 4%.
Tying up money that has been bet counts as a type of friction. If $0.96 will get me $1.00 in a year with no fee, but index funds are getting 5%/year and I can't do both, I have very little reason to take the bet. Labor can also be a type of friction, especially if bet sizes are capped. If I could gain $100 but it would require five hours of wrestling with terrible UIs or APIs (or studying cryptocurrencies), there's little incentive to bet.
Zvi describes how friction like this can drive people away from a market and make it too small to be useful. And that may well be the primary effect. But it can also cause asymmetric distortions, driving all probabilities toward the middle.