I have been stewing about this question in general for a while. When I look back at my long (so far) life, I think of the many times I have been misled by so-called experts.
From which I learned that experts, even real ones:
On top of that a lot of self-proclaimed and even highly credentialed 'experts' don't actually have much of a clue. Because:
Important **None of this is to say that an amateur with google and ten minutes to spare can do better**. be cautious. It is very hard to do better than flawed experts.
Personally I have worked out, over time, some heuristics which have proven useful to sort out actual experts. Some things that mark out an actual expert:
1. They can make surprisingly accurate predictions. Better than most people, and better than simple techniques like linear extrapolation.
2. They can fix things that are broken. Whether broken machines, or dysfunctional social systems, or sick people.
3. They can explain things in a way that is as simple as possible, illuminating, and gives one clues as to how things might be better.
Not only that, but they have evidence for this. An example of the opposite: After thirty-five years of Freudian psychoanalysis, someone thought to do a study of whether they actually helped people get better more than doing nothing. No, they did not.
Things that do not mark out an expert:
1. Status among peers. The peers may be equally clueless or useless.
2. Great confidence. This is more a sign of arrogance than of competence. In "A Mind for Numbers" it is pointed out that claims of skill or competence or knowledge not accompanied by proof are actually far worse than acknowledged incompetence.
3. Ornate certificates on the wall.
4. Having attended high status institutions.
5. Having been successful, after taking huge risks. They may be a lucky idiot - look closely.
6. A few lucky breaks.
In the current context, I am willing to listen to experts who have a proven track record, who have relevant experience, and who have the skills needed to do the job. Even then I look hard for biases.
I welcome any additions/corrections/clarifications to all this.
It's not hard to find amateurs who got it right, after the fact. Amateur advice is of lower expected quality, but of much higher variance than expert advice.
Apart from filtering out the obvious crazies, can we identify high quality amateur advice ahead of time?
These articles were widely shared at the time, and people were taking them seriously. https://medium.com/@noahhaber/flatten-the-curve-of-armchair-epidemiology-9aa8cf92d652 and https://medium.com/@ameliahoovergreen/listen-to-actual-experts-on-coronavirus-please-1b0e7f2c763e were a response to that saying "please stop paying attention to these widely shared articles because they are not written by experts". I'm saying that the attention was reasonable given the circumstances and the contents of the articles.
If the "please stop paying attention" articles had contained substantive criticisms of the articles ("they say Italy's CFR is X but they're missing Y") I would feel very differently.
But other people were sharing other articles saying different things ("this is all overblown"), or just something more moderate like "we'll have to social distance later but not yet" and other people were also taking those seriously. So I still don't know how to answer the question of "at the time, how should we have known who to listen to?"
Here's my answer:
There is an important distinction between "object level arguments" and "appeals to authority". Contrary to how it's normally spoken about, appeal to authority is not really fallacious and at times absolutely necessary. If I am unable to parse the object level arguments myself, I have to defer to experts. The only issue is whether I have the self awareness and integrity to say "I'm not capable of evaluating this myself, so unfortunately I have to defer to the people I trust to get these things right. Maybe you're right and I'm just not smart enough to see it". However, this must ground out somewhere. If you listen to people who only appeal to authority (whether it is their own or others) and there are never any attempts to ground things in object level arguments, then there is nothing this trust is founded on and so your beliefs can float away with no connection to reality.
What I do is take into consideration all object level arguments which I am not personally qualified to evaluate, and then weigh my trust in the various "authorities" based on how capable they seem in actually getting into the object level and making at least as much sense as the people they're arguing against. As it applies here, the amateurs linked to actually got into the object level and made very plausible sounding arguments. I didn't see any major holes in the main premise, even if I could pick less important nits. I never saw any credentialed authority engaging in the object level and making even plausibly correct counterarguments which negated the main point of these amateur models. There were a lot of "don't worry, nothing to see here", but there weren't any that were backed up by concrete models that didn't have visible holes.
The people I'm going to listen to (regardless of how capable I personally am of evaluating the object level arguments) are those who 1) have been willing to stick their neck out and make actual arguments, and 2) haven't had their neck chopped off by people pointing out identifiable mistakes in ways that are either personally verifiable or agreed upon by a more compelling network of "authority".
I think this heuristic worked pretty well in this case.
Neil Ferguson and Ian Lipkin, heads of their respective prominent epidemiological institutes, have caught or are suspected of having caught COVID-19. Being a top epidemiologist seems to be a strong risk factor for being infected, given that fewer than 1% of the English speaking world has probably been infected so far and there are not that many epidemiologists who are as prominent as these two. (This seems worth noting here as a surprising/interesting fact to update on. I'm not suggesting we should draw strong conclusions from this.)
ETA: See also my previous comment about a couple of authorities on economics.
The story about Neil Ferguson suggests he got it while being in the press annoucement. It might very well be that the risk of getting infected was worth the PR of making that press annoucement as best as it could be.
Right, I have some uncertainty that there's a reasonable explanation along those lines, which is (in part) why I said "I’m not suggesting we should draw strong conclusions from this." But I cringe at public officials, including public health officials doing press conferences while not observing social distancing guidelines that they themselves are promoting. Wouldn't it actually be more effective PR to show people what to do and signal that it's something they themselves are taking seriously? On a positive note on this front, I noticed that yesterday Trump and other top officials did a Fox interview / virtual townhall outdoors while being appropriately spaced out. I can't see how it would have been worse PR if all the officials had done that to begin with.
I don't think this decision is up to the public health officials. Indeed, you can read an interview with Fauci where he is making this exact point and saying that he has asked, and asked, and asked, for social distancing to be observed at press conferences.
Notalgebraist posted a reasonable critique of Flattening The Curve Is a Deadly Delusion, explaining how it's incorrect to assume that flattenings won't reduce the total number of cases, and that it doesn't make sense to assume a normal distribution. I think that the the piece's main point was correct, though: that the curve would need to get crazy flat for hospitals to not be overloaded.
3 days later, the Imperial College study made the same point in a much better and more rigorous way (among lots of other good points), but it was less widely shared on social media.
I'm not sure what the conclusion here is. Non-experts will sometimes make false assumptions and get the details wrong, but they're still capable of making good points that only require you to multiply numbers together, and will do so a few days faster and in a way that's more memetically fit than papers from experts?
Can you link to Nostalgebrist saying that his post was a math error at SSC? I can't find it. Also, see Nostalgebrist's update at the start of the critique:
To be clear, Bach’s use of a Gaussian is not the core problem here, it’s just a symptom of the core problem.
The core problem is that his curves do not come from a model of how disease is acquired, transmitted, etc. Instead they are a convenient functional form fitted to some parameters, with Bach making the call about which parameters should change – and how much – across different hypothetical scenarios.
Having a model is crucial when comparing one scenario to another, because it “keeps your accounting honest”: if you change one thing, everything causally downstream from that thing should also change.
Without a model, it’s possible to “forget” and not update a value after you change one of the inputs to that value.
That is what Bach does here: He assumes the number of total cases over the course of the epidemic will stay the same, whether or not we do what he calls “mild mitigation measures.” But the estimate he uses for this total – like most if not all such estimates out there – was computed directly from a specific value of the replication rate of the disease. Yet, all of the “mild mitigation measures” on the table right now would lower the replication rate of the disease – that’s what “slowing it down” means – and thus would lower the total.
Nowhere in that comment does he say that his post was or contained a "math error". The closest thing I can find is this:
Great point about the step function. That convinces me that Bach would not have drawn a different qualitative conclusion if he had used a different functional form, no matter which one. I’ve updated my post with a note about this.
[EDIT: AFAICT Douglas_Knight is saying that Nostalgebrist's initial guess that Bach's post was sensitive to the functional form is a "math error". I wouldn't call it that, but perhaps reasonable people could disagree about this.]
You've lost track of the object level here.
What did his post originally mean? I'm not allowed to read people's minds. He admits that no one took from it what he wanted them to take from it. Lanrian said that it was "a reasonable critique...that it doesn't make sense to assume a normal distribution." That was a qualitative complaint and he admitted that it was qualitatively wrong.
The gaussian assumption makes no difference.
As I said, I do agree that the piece's qualitative conclusion was correct. However, the gaussian assumption does make a large quantitative difference. Comparing it to the extreme: If we always have the maximum number of people in ICUs, continuously, the time until herd-immunity would be 4.9 years, which is a factor 3 less than what the normal assumption gives you. Although it is still clearly too much, it's only one or two additional factors of 3 away from being reasonable. This extreme isn't even that unrealistic; something like it could plausibly be achieved if the government continuously alternated between more or less lock-down, keeping the average R close to 1.
To be clear, I think that it's good that the post was written, but I think it would have been substantially better if it had used a constant number of infected people. If you're going to use unrealistic models (which, in many cases, you should!) it's good practice to use the most conservative model possible, to ensure that your conclusion holds no matter what, and to let your reader see that it holds no matter what. In addition, it would have been simpler (you just have to multiply/divide 3 numbers), and it would have looked less like realistic-and-authoritative-math to the average reader, which would have better communicated its real epistemic status.
Notalgebraist posted a reasonable critique of Flattening The Curve Is a Deadly Delusion, explaining how it's incorrect to assume that flattenings won't reduce the total number of cases, and that it doesn't make sense to assume a normal distribution.
I see the Deadly Delusion post saying "My back-of-the-envelope calculation is not a proper simulation, or a good model of what’s going on either. Don’t cite it as such! In reality, the spread of a disease does not follow a normal distribution. The main bump of the curve will be on the left, with a long tail on the right. There is always going to be some effective mitigation (prevention of public gatherings, conferences, non-essential travel). The model is quite sensitive to the length of the stay in the ICU. If we get that down, fewer people will need these resources simultaneously, and the peaks of the curves will come down. We may be able to fight the inflammation during pneumonia, and reduce the number of critical cases. The available medical resources will increase over time to deal with the need. Regulations will be dropped, new treatments will be explored, and some of them will work. At some point in the near future, we may have to blow into a tube before we enter an airplane or an important public building, and a little screen tells us within seconds if our airways hold COVID-19, H1N1 or the common flu. But the point of my argument is not that we are doomed, or that 6% of our population has to die, but that we must understand that containment is unavoidable, and should not be postponed, because later containment is going to be less effective and more expensive, and leads to additional deaths." This seems to address Notalgebraist's concerns pretty well? I thought maybe it was added to the post in response to feedback, but I see it in the first wayback capture https://web.archive.org/web/20200314031533/https://medium.com/@joschabach/flattening-the-curve-is-a-deadly-delusion-eea324fe9727 which looks like it's older than https://nostalgebraist.tumblr.com/post/612592471097147392/flattening-the-curve-is-a-deadly-delusion
3 days later, the Imperial College study made the same point in a much better and more rigorous way (among lots of other good points), but it was less widely shared on social media.
While the paper itself wasn't widely shared (not too surprising!) lots of news stories that cited it and passed on its conclusions were shared: https://www.nytimes.com/2020/03/17/world/europe/coronavirus-imperial-college-johnson.html https://www.washingtonpost.com/world/europe/a-chilling-scientific-paper-helped-upend-us-and-uk-coronavirus-strategies/2020/03/17/aaa84116-6851-11ea-b199-3a9799c54512_story.html https://www.cnn.com/2020/03/17/health/coronavirus-uk-model-study/index.html
I actually do not think I've seen any honest recounting of the Imperial College study. Unless I'm misreading it, the study appears to be saying that we either need to have essentially a complete lock down for the next 18 months (which **definitely has not** been reported by most media organizations) or we shouldn't be implementing full social distancing, but rather an optimal policy of social distancing of those over 70 and school closures. Complete social distancing followed by relaxation before the development of a vaccine leads to almost as many deaths as unmitigated spread.
I've seen so much discussion of how the Imperial College paper has influenced governmental policy, but our current approach (which seems to be - hard, social isolation for a month or two followed by relaxation) is exactly the approach proscribed by the paper.
I would love if someone could weigh in with how I'm misinterpreting the paper because the distance between what the paper seems to say and the rhetoric I've seen online seems to be incredibly large. For instance, if I am interpreting the paper correctly, China is not a success case but will soon start seeing exponential growth again once they re-open unless they implement SDOL_70.
The excess deaths in the model are caused by a red line of critical care capacity, which the study assumes is fixed at 14 per 100,000 population in the US. If the curve of cases requiring critical care rises too high, 100% of cases above the red line will die even though 50% could have been saved.
But the coronavirus doesn't need the entire package of critical care a hospital might provide, just a ventilator and a bed. So the US is aiming to simply build enough ventilators to hike the red line above the curve. (I would assume the UK is doing the same thing but I haven't been following their response.)
From the Imperial College paper:
For countries able to achieve it, this leaves suppression as the preferred policy option. [...] The major challenge of suppression is that this type of intensive intervention package--or something equivalently effective at reducing transmission--will need to be maintained until a vaccine becomes available (potentially 18 months or more)--given that we predict that transmission will quickly rebound if interventions are relaxed. We show that intermittent social distancing--triggered by trends in disease surveillance--may allow interventions to be relaxed temporarily in relative short time windows, but measures will need to be reintroduced if or when case numbers rebound.
So it's not really saying "lock down for 18 months or do nothing". It's saying lock down until the problem is controlled and then squash new outbreaks quickly. The Hammer and the Dance article makes this point more clearly in my opinion, especially pointing out that a temporary lockdown would give us time to build up test and treatment capacity, protective equipment supplies, etc, and implement strategies for tracing and suppressing the new cases that arise after it's lifted. However it does say "a few weeks" of strict lockdown will be enough without really supporting it well, and that seems optimistic (Bill Gates, for instance, has been saying 6-10 weeks).
Hm. Doesn't the paper go on to say that full lockdowns would need to be in effect for 2/3 of those 18 months? I will read that article, but I don't think the paper is saying that we could return to normal with monitoring and case tracing and localized lockdown to quash outbreaks for the remainder of the 18 months.
e: Okay, read the paper. Respectfully, many of the estimated numbers there seem entirely inconsistent with the literature that I've been reading from epidemiology experts. I haven't seen a single paper estimating 10 million deaths in the United States, and I'm not inclined to trust an uncredentialed medium post (I know this is relevant to the OP topic).
I also really value modeling. Ferguson ran the numbers and seems to suggest that "the dance" would not be an effective approach for suppression long term and that we would need to go under frequent shelter-in-places again. This medium article doesn't seem to cite any sort of number crunching that the hammer followed by the dance would work for long-term suppression.
It said the fraction would be a bit lower for the US because local outbreaks could be dealt with by state-level lockdowns, but I didn't see a hard number. Still, intermittent lockdowns for 18 months seems much more achievable than a continuous lockdown. The Hammer article is definitely more optimistic than the Imperial paper, though it still doesn't quite imply "return to normal".
Since "do nothing" is not a real option, what will actually happen in the US (I'm moderately confident) is some degree of lockdown for several weeks to several months depending on how effective it is. The sooner we start it the better it will work. After that we will either: 1) give up, if the lockdown was ineffective and a large fraction of the country is infected (this is "flattening"), or 2) if the lockdown succeeded in reducing the number of cases substantially, we'll move into a period of intermittent and possibly localized lockdowns interspersed with trying to test and contact trace. The fraction of time we spend in intermittent lockdowns will depend on how effective the testing and tracing is.
Proscribe or prescribe?
"Prescribe means "to set down authoritatively for direction" or "to set down a medical procedure in order to cure or alleviate symptoms." The noun form is prescription, that is, something prescribed. Proscribe means "prohibit or limit" or "ostracize or avoid in a social sense"
Yep, I meant it in the correct fashion :) Prescribe would imply the opposite of what the paper said.
Since the very beginning of LW, there has been a common theme that you can't always defer to experts, or that the experts aren't always competent, or that experts on a topic don't always exist, or that you sometimes have to do your own reasoning to determine who the experts are, etc. (E.g. LW on Hero Licensing, or on the Correct Contrarian Cluster, or on Inadequate Equilibria; or ACX in Movie Review: Don't Look Up.)
I don't think this post makes a particularly unique contribution to that larger theme, but I did appreciate its timing, and how it made and referenced its examples.
Hmmmm. I am wondering about the information transmission/propagation process and how that might effect outcomes.
The "experts" are know to be connected with some subject area. When people (particularly media and government) the "experts" are brought in. LW has had at least one thread on this situation and what some of the problems might be.
The "armchair" people are probably two forms -- either they are some form of known "celebrity" type of smart person or they are unknowns to the world generally. In either case the utterances from these people run through a different filter before their claims become part of the general information about X.
In this particular context -- a pandemic, we don't have too many real experts in the sense of "I've done this before and I have see this playing out". Yes, we've seen outbreak and understand the transmission processes and models pretty well. However, some of this seems to be different than SARS, MERS and similar more contained or localized events.
As someone mentioned, we can find a bunch or pretty silly analysis or recommendation from the armchair side (even that it's a hoax, just like a cold/flu or only a problem for the really old and already sick not the healthy). We when we make the claim about how the armchair crowd has done much better than the experts I think we gloss over how those good armchair positions came into the general information set. They were the cream of the crop and filters via a number of social filtering mechanisms.
We should not compare the best armchair position against some average expert position (where the experts may in fact not really be experts).
This might however suggest the selection mechanism used by both media and government in situations where we are dealing with something somewhat new may have some weaknesses that we want to review.
You're right about the selection process being a major factor. Part of the reason I read LessWrong is that the selection mechanism for posts here seems really, really good - good enough that I often sanity check actual doctors/experts by cross referencing against LW and seeing what matches up.
On this particular topic, some actual (official) experts are starting to come forward with good material.
I like this one, with the new insight that quarantine in isolation was very important. Beyond "please stay home".
Presentation
https://www.medrxiv.org/content/10.1101/2020.03.03.20030593v1
Slides
https://drive.google.com/file/d/1QFfBbs-7-qCqDJ93_Ww6QQUQlMVp77Pl/view
Here is an expert trying to reverse engineer the government's strategy. [I do have some concerns with his approach, he seems to have an exaggerated view of how accurately you can manage R0. There is such a narrow zone for R0 between "it blows up fast and high" and "it rapidly declines close to zero", that IMHO it is futile to try to finesse how much slackness we can get away with.]
The specific evidence you’ve cited is weak. (1) You write that “The argument that we should be listening to experts and not random people would make a lot of sense if the "armchair" folks didn't keep being right.” It is extremely easy to be right on a binary question (react more vs less). That many non-experts were right is therefore more-or-less meaningless. (I can also cite many, many examples of non-experts being wrong. I think what we want is the fraction of experts vs non-experts who were right, but that seems both vague and unobtainable.)
(Note that this is importantly different, and stronger, than the claim you made in the final paragraph. I agree with that claim.)
(2) For many, but probably not all, of the policy failures you describe, there is little reason to attribute them to experts. The United States is not a technocracy.
The posts I'm referring to made claims that were much stronger than "we should be reacting more". If you look through https://medium.com/@tomaspueyo/coronavirus-act-today-or-people-will-die-f4d3d9cd99ca and https://medium.com/@joschabach/flattening-the-curve-is-a-deadly-delusion-eea324fe9727, and the follow-up https://medium.com/@tomaspueyo/coronavirus-the-hammer-and-the-dance-be9337092b56 they're making detailed claims about how the world is and how it will soon be.
People are writing a lot about the coronavirus, and I've seen a lot of pushback on how pieces often haven't been written by people with epidemiology or public health credentials. For example, Flatten the Curve of Armchair Epidemiology, Listen To Actual Experts On Coronavirus, and comments like this one. The argument that we should be listening to experts and not random people would make a lot of sense if the "armchair" folks didn't keep being right.
Let's look at the articles they're criticizing for having non-expert authors:
Coronavirus: Act Today or People Will Die (3/10): Argues that we need to get everyone to stay home immediately.
Cancel Everything (3/10): Argues essentially the same point.
Flattening The Curve Is a Deadly Delusion (3/13): Argues that our medical capacity is so much lower than the likely peak that we need an immediate lockdown and a renewed focus on containment.
With two weeks of perspective, however, these articles were exactly right. They clearly laid out the case for decisive action, and if we had followed their prescriptions more closely we would be in much better shape right now.
This goes beyond a few articles, however. All the aspects of this crisis that have involved planning more than a couple weeks out have been very poorly handled:
People weren't told to stock up on food so that they'd be able to reduce trips outdoors, and so that they'd have food in case they were quarantined for 2+ weeks. Stores weren't told to prepare for a rush. A government that was on top of things could have started advocating this in early February in an "if you can afford to" way. This would have spread people's buying over a longer period and avoided the empty shelves we see now. Instead, once restaurants were closing and people realized that they could be quarantined at any time, everyone simultaneously tried to buy weeks worth of ingredients and we had widespread shortages of basic goods starting in mid March.
Hospitals weren't told (or allowed?) to ration personal protective equipment such as masks until they had shortages. The CDC didn't publish guidelines for sanitization and reuse, and start telling people to conserve. In weeks of handling initial cases, hospitals burned through amounts that would have lasted months with careful rationing.
The federal government, state governments, or even hospitals could have placed emergency ventilator production orders in February, but didn't. Because we don't allow price gouging, ventilator companies can't ramp up production speculatively figuring that if there is really an epidemic then they'll make their money back. By mid March it was obvious that we were far short of where we needed to be and the companies started ramping up but we lost about a month of production increase.
Masks were sitting on shelves across the country, and the government could have requisitioned them for emergency medical use, or even just gone and bought them. Instead the Surgeon General tweeted a request that people not buy them.
Testing has been completely messed up, though it's hard to tell how much was bad luck vs reasonable rationing of scarce tests. But we should have been quarantining people who seemed to have it based on symptoms, instead of saying "well, since we can't test you we have to assume you don't have it, so you're welcome to continue living your life".
We could have planned and built out COVID-specific facilities to reduce the risk of others getting sick and make more efficient use of ventilators and personal protective equipment. We're just starting to do this now, much too late.
The passengers of the Grand Princess were offered testing but told that if they tested positive they would be required to undergo quarantine. Of 858 passengers, 568 declined testing and were released without even advice to self-isolate.
The CDC is still (3/24) not recommending the general public avoid contact with others unless you're sick, they're sick, or you know covid-19 is spreading in your community. They're only recommending people stay home if they're sick.
And I understand: this is moving very quickly, and authorities aren't used to needing to respond so rapidly. But there was a meme going around:
This joke and its many copycats feature the CDC we wish we had. A CDC that would have been pushing social distancing a month ago, when it would have helped so much more.
Google Trends: "social distancing"
If we had listened to the warnings and prepared better we would have the experts we need, with the influence to get policies changed, and we wouldn't need the advice of the armchair epidemiologists. But that's not the world we've found ourselves in, and the amateurs have been doing critical work filling in for them in pushing policy.
A policy of "listen to random experts" is better than a policy of "listen to random amateurs". But rejecting the arguments of amateurs who were making clear arguments, solely on the grounds of their non-expert status, was harmful here.