LESSWRONG
LW

Comment Permalink

(Sources and so on are in https://www.lesswrong.com/posts/wQLXNjMKXdXXdK8kL/fauci-s-emails-and-the-lab-leak-hypothesis)

I don't factor Bayes factors together when I'm on Metaculus either, so that's not really how I reason. If you want them I would be interested in yours for the following pieces of evidence.

How do you get a probability as high as 35% for a natural origin? Can you provide Bayes factors for that?

There's a database for Coronaviruses. We (the international community) funded it to help us for the time when we have to deal with a coronavirus pandemic. The Wuhan Institute of Virology (WIV) took it down in late 2019 and doesn't give it to anyone. Nobody complains about that in a way that brings this in the mainstream narrative even when all those virologists who believe in their field should think that the database is valuable for fighting the pandemic. If it isn't why are we funding that research in the first place?

According to the US government information there was unusual activity in the WIV in October 2019 with at least significantly reduced cell phone traffic and likely also road blocks.

Three people from the WIV seems to went to hospital in November 2019 with flu or COVID-19 like symptoms in the same week.

Huang Yanling who was in the beginning of the pandemic called patient zero was a WIV employee and US government requests who account for what's up with her currently go unanswered.

The security at the WIV was so bad that they asked the US for help in 2018 because they didn't have enough skilled people to operate their biosafety 4 lab safely. Chinese care about saving face, things need to have been bad to get them to tell the US that they didn't have enough people to operate their lab safely.

There are six separate biological reasons of why the virus looks like it came from a lab.

The bats are more then 1000km away from Wuhan. It's quite unclear how they would have naturally infected people in Wuhan.

An amazing amount of effort went into supressing the story, likely with a lot of collateral damage that made us react less well to the pandemic. Google, Facebook and Twitter started censoring in early February and that might have been part of the reason why it took us so long to respond.

As Bret Weinstein said, that given that the virus looks so much like it comes from Wuhan the next likely alternative explantion would be that someone went through a lot of effort and released it in Wuhan to make the WIV look bad. If that would be the case it's however unclear why the WIV doesn't allow outside inspections to clear their name.

If all the information we had was that Huang Yanling who was in the beginning called 'patient zero' was a WIV employee that alone might warrent more then 65%. I mean what are the odds that 'patient zero' for a pandemic caused by Coronavirus is randomly an employee of a lab studying Coronaviruses?

Then a lab studying Coronaviruses with known safety problems?

See in context

56

[ Question ]

Why did no LessWrong discourse on gain of function research develop in 2013/2014?

by ChristianKl

18th Jun 2021

2 min read

4 45

56

While I wasn't at 80% of a lab leak when Eliezer asseted it a month ago, I'm now at 90%. It will take a while till it filters through society but I feel like we can already look at what we ourselves got wrong.

In 2014, in the LessWrong survey more people considered bioengineered pandemics a global catastrophic risk then AI. At the time there was a public debate about gain of function research. On editoral described the risks of gain of function research as:

Insurers and risk analysts define risk as the product of probability times consequence. Data on the probability of a laboratory-associated infection in U.S. BSL3 labs using select agents show that 4 infections have been observed over <2,044 laboratory-years of observation, indicating at least a 0.2% chance of a laboratory-acquired infection (5) per BSL3 laboratory-year. An alternative data source is from the intramural BSL3 labs at the National Institutes of Allergy and Infectious Diseases (NIAID), which report in a slightly different way: 3 accidental infections in 634,500 person-hours of work between 1982 and 2003, or about 1 accidental infection for every 100 full-time person-years (2,000 h) of work (6).
A simulation model of an accidental infection of a laboratory worker with a transmissible influenza virus strain estimated about a 10 to 20% risk that such an infection would escape control and spread widely (7). Alternative estimates from simple models range from about 5% to 60%. Multiplying the probability of an accidental laboratory-acquired infection per lab-year (0.2%) or full-time worker-year (1%) by the probability that the infection leads to global spread (5% to 60%) provides an estimate that work with a novel, transmissible form of influenza virus carries a risk of between 0.01% and 0.1% per laboratory-year of creating a pandemic, using the select agent data, or between 0.05% and 0.6% per full-time worker-year using the NIAID data.

Even at the lower bar of 0.05% per full-time worker-year it seems crazy that society continued playing Russian Roulette. We could have seen the issue and protested. EA's could have created organizations to fight against gain-of-function research. Why didn't we speak every Petrov day about the necessity to stop gain of function research? Organizations like OpenPhil should go through the 5 Why's and model why they messed this up and didn't fund the cause. What needs to change so that we as rationalists and EA's are able to organize to fight against tractable risks that our society takes without good reason?

World Optimization

Frontpage

56

Mentioned in

21Matt Taibbi's COVID reporting

New Answer

New Comment

4 Answers sorted by
top scoring

habryka

Jun 19, 2021

210

I feel like this just happened? There was a good amount of articles written about this, see for example this article by the Global Priorities Project on GoF research:

http://globalprioritiesproject.org/wp-content/uploads/2016/03/GoFv9-3.pdf

I also remember a number of other articles by people working on biorisk, but would have to dig them up. But overall I had a sense there was both a bunch of funding and a bunch of advocacy and a bunch of research on this topic.

[-]ChristianKl4y150

I searched LessWrong itself for "gain of function" and it didn't bring up much. Searching for it on OpenPhil finds it mentioned a few times, so it seems that while OpenPhil got in contact with the topic they failed to identify it as a cause area that needs funding.

All the hits on OpenPhil are 2017 and before and 2018 the Trump administration ended the ban on gain of function research. That should have been a moment of public protest by our community.

4DirectedEvolution4y

Are you saying we should have been marching in the streets and putting up banners? Criticizing the end of the ban more in public? Or taking steps against it, somehow, using alternative mechanisms like advocacy with any contacts we might have in the world of virology?

8ChristianKl4y

The first step would be to do similar things as we do with other X-risks. For the case of OpenPhil, the topic should have been important enough for them to task a researcher with summarizing the state of the topic and what should be done. That's the OpenPhil procedere to deal with topics that matter. That analysis might have resulted in the observation that this Marc Lipsitch guy seems to have a good grasp of the subject to then fund him with a million per year to do something. It's not clear that funding Lipsitch would have been enough, but it would be on course with "we tried to do something with our toolkit". With research it's hard to know before what you find if you invest in a bunch of smart people to think about a topic and how to deal with it. In retrospect finding out that the NIH illegally funneled money to Baric and Shi in circumvention of the moratorium imposed by the Office of Science and Technology Policy and then challenging that publically might have prevented this pandemic. Being part of a scandal about illegal transfer of funds likely would have seriously damanged Shi's career given the importance of being seen as respectful in China. Finding that out at the time would have required reading a lot of papers to understand what's going on but I think it's quite plausible that a researcher who reads through the top 200 gain of function research papers attentively and tried to get a good model of what's happening might have caught it.

9habryka4y

Some relevant links: * https://www.openphilanthropy.org/sites/default/files/Lipsitch%201-29-14%20%28public%29.pdf * OpenPhil conversation notes from 2014 with Lipsitch * https://www.openphilanthropy.org/focus/global-catastrophic-risks/biosecurity/harvard-university-biosecurity-and-biosafety * Grant to Lipsitch in 2020 for ~$320k * https://www.openphilanthropy.org/giving/grants * All OpenPhil grants in the biosecurity area Don't think they prove anything, but seem useful references.

4ChristianKl4y

I do think they suggest the situation is better then I initially thought given that funding the Lipsitch /Johns Hopkins Center for Health Security is a good idea. I read through their report Research and Development to Decrease Biosecurity Risks from Viral Pathogens: I do think that list is missing finding ways to reduce gain of function research and instead encourages gain of function research via funding "Targeted countermeasures for the most dangerous viral pathogens". Not talking about the tradeoffs between developing measures against viruses and the risk caused by gain of function research, seem to me a big omission. Not speaking about the dangers of gain of function research likely reduces conflicts with virologists. The report suggests to me that the led themselves be conned by researchers who suggest that developing immunity against a novel pathogen in fewer than 100 days is about developing new vaccination platforms when it was mostly about regulation and finding ways to verifying drug safety in short amounts of time. Fighting for changes in laws about drug regulation means to get in conflicts while funding vaccine platforms is conflictless. Unsexy approaches like reducing the amount of surfaces touched by multiple people or researching better airfilters/humidifiers to reduce transmission of all viruses are also off the roadmap.

[-]ChristianKl4y*90

I now read the paper and given what we saw last year the market mechanism they proposed seems flawed. If we would have an insurance company that would be responsible to paying out the damage created by the pandemic that company would be insolvent and not be able to pay for the damage and at the same time the suppression of the lab leak hypothesis (and all the counterparty risk that comes with a major insurance company going bankrupt) would have been even stronger when the existence of a billion dollar company depends on people not believing in the la... (read more)

Anders_H

Jun 19, 2021

160

Here is a data point not directly relevant to Less Wrong, but perhaps to the broader rationality community:

Around this time, Marc Lipsitch organized a website and an open letter warning publicly about the dangers of gain-of-function research. I was a doctoral student at HSPH at the time, and shared this information with a few rationalist-aligned organizations. I remember making an offer to introduce them to Prof. Lipsitch, so that maybe he could give a talk. I got the impression that the Future of Life Institute had some communication with him, and I see from their 2015 newsletter that there is some discussion of his work, but I am not sure if anything more concrete came out of of this

My impression was that while they considered this important, this was more of a catastrophic risk than an existential risk, and therefore outside their core mission.

[-]ChristianKl4y90

While this crisis was a catastrophe and no existential challenge, it's unclear why that has to be generally the case.

The claim that global catastrophic risk isn't part of the FLI mission seems strange to me. It's the thing the Global Priorities Project of CEA focus on (global catastrophic risk is more primarily mentioned on in the Global Priorities Project then X-risk).

FLI does say on it's website that out of five areas one of them is:

Biotechnology and genetics often inspire as much fear as excitement, as people worry about the possibly neg

... (read more)

[-]Anders_H4y20

Here is a video of Prof. Lipsitch at EA Global Boston in 2017. I haven't watched it yet, but I would expect him to discuss gain-of-function research: https://forum.effectivealtruism.org/posts/oKwg3Zs5DPDFXvSKC/marc-lipsitch-preventing-catastrophic-risks-by-mitigating

4ChristianKl4y

He only addresses it indirectly by saying we shouldn't develop very targeted approaches (which is what gain of function research is about) and instead fund interventions that are more broad. The talk doesn't mention the specific risk of gain of function research.

jdfaben

Jun 19, 2021

120

I can't speak for less wrong as a whole, but I looked into this a little bit around that time, and concluded that actually it looked like things were heading in the sensible direction. In particular, towards the end of 2014, the US government stopped funding gain of function research: https://www.nature.com/articles/514411a, and there seemed to be a growing consensus/understanding that it was dangerous. think anyone doing (at least surface level) research in 2014/early 2015 could have reasonably concluded that this wasn't a neglected area. That does leave open the question of what I did wrong in not noticing that the moratorium was lifted 3 years later...

[-]ChristianKl4y120

It seems that when there's a discussion of a dangerous practice being stopped pending safety review it makes sense to shedule into the future a moment to review how the safety review turned out.

Maybe a way forward would be:

Whenever there's something done by a lot of scientists is categorically stopped pending safety review, make a metaculus question about how the safety review is likely to turn out.

That way when the safety review turns out negatively, it triggers an event that's seen by a bunch of people who can then write a LessWrong post about it?

That leaves the question whether there are any comparable moratoriums out there that we should look at more.

[-]GeneSmith4y40

Eliezer seemed to think that the ban on funding for gain of function research in the US simply led to research grants going to labs outside the US (Wuhan Institute of Virology in particular). he doesn't really cite any sources here so I can't do much to fact check his hypothesis.

Upon further googling, this gets murkier. Here's a very good article that goes into depth about what the NIH did and didn't fund at WIV and whether such research counts as "gain of function research".

Some quotes from the article:

The NIH awarded a $3.4 million grant to the non-pro

... (read more)

4ChristianKl4y

I think Eliezer ignores how important prestige is for the Chinese. We got them to outlaw human cloning by telling them that doing it would put the Chinese academic community in a bad light. We likely could have done the same with gain of function research. Having their first biosafety level 4 lab for the Chinese likely was mostly about prestige. Having no biosafety 4 labs while a lot of other countries had biosafety 4 labs wasn't something that was okay for the Chinese because it suggests that they aren't advanced enough. I do think that it would be possible to make a deal that gives China the prestige for their scientists that they want without having to endanger everyone for it. The Chinese took their database with the database about all the viruses that they had in their possession down in September 26 2019. In their own words they took it down because of a hacking attack during the pandemic (which suggests that starts for them somewhere in September). If we would have the database we likely would find a more related virus in it. Given that the point of creating the database in the first place was to help us in a coronavirus pandemic taking it down and not giving it to anyone is a clear sign that there's something that would implicate them. Basically, people outside of the virology community told them that they have to stop after exposing 75 CDC scientists to anthrax and a few weeks later other scientists finding a few vials of small pox in their freezer. The reaction of the virology community was to redefine what gain of function research happens to be and continue endangering everyone. It's like Wall Street people when asked whether they do insider training saying: "According to our definition of what insider training means we didn't". I have written all my sources up at https://www.lesswrong.com/posts/wQLXNjMKXdXXdK8kL/fauci-s-emails-and-the-lab-leak-hypothesis

1GeneSmith4y

Wow, this is quite the post! I've been looking for a post like this on LessWrong going over the lab leak hypothesis and the evidence for and against it, but I must have missed this one when you posted it. I have to say, this looks pretty bad. I think I still have a major blindspot, which is I've read much more about the details of the lab leak hypothesis than I have about the natural origin hypothesis, so I still don't feel like I can judge the relative strength of the two. That being said I think it is looking more and more likely that the virus was probably engineered while doing research and accidentally leaked from the lab. Thanks for writing this up. I'm surprised more of this info doesn't show up in other articles I've read on the origins of the pandemic.

3ChristianKl4y

I was too when I researched it. I think it's telling us something about the amount of effort went into narrative control. Take for example Huang Yanling, who was in the start of the pandemic called "patient zero" till someone discovered that she works at the Wuhan Institute of Virology and the Chinese started censoring information about her. The fact that the NIH asked the EcoHealth alliance about where Huang Yanling is suggest that the US government (that has CIA/NSA who wiretap a lot and hack people to try to get some idea what's going on) does consider this to be an important piece of information. Why doesn't the name appear in the NewYorkTimes? Very odd... It seems impossible for a simple he-said/she-said article about the questions from the NIH to EcoHealth to appear in any of the major publications.

2ChristianKl4y

After reading more it seems that according to John Holdren (Head of the Office of Science and Technology Policy) the Chinese came to US politicians to discuss how topics like gain of function research should be regulated: China's leaders aren't completely irresponsible. They messed up in Wuhan by allowing the lab to without enough trained personal to operate it safely but I would expect that it's a combination of goals to have the lab on the one hand and the information about the security issues not going to the right people because the people who are responsible for the lab don't want to look bad. I doubt that Xi Jinping knew that he had a biosafety 4 lab without enough trained personal to be run safely.

1GeneSmith4y

I think the fact that mistakes like this are so understandable is precisely why gain of function research is dangerous. One mistake can lead to a multi-year pandemic and kill 10 million people. With those stakes, I don't think anyone should be doing gain of function research that could lead to human deaths if pathogens escaped.

Anders_H

Jun 21, 2021

I found the original website for Prof. Lipsitch's "Cambridge Working Group" from 2014 at http://www.cambridgeworkinggroup.org/ . While the website does not focus exclusively on gain-of-function, this was certainly a recurring theme in his public talks about this.

The list of signatories (which I believe has not been updated since 2016) includes several members of our community (apologies to anyone who I have missed):

Toby Ord, Oxford University
Sean O hEigeartaigh, University of Oxford
Daniel Dewey, University of Oxford
Anders Sandberg, Oxford University
Anders Huitfeldt, Harvard T.H. Chan School of Public Health
Viktoriya Krakovna, Harvard University PhD student
Dr. Roman V. Yampolskiy, University of Louisville
David Manheim, 1DaySooner

Interestingly, there was an opposing group arguing in favor of this kind of research, at http://www.scientistsforscience.org/. I do not recognize a single name on their list of signatories

[-]ChristianKl4y20

That's interesting. That leaves the question of why the FHI mostly stopped caring about it after 2016.

Past that point https://www.fhi.ox.ac.uk/wp-content/uploads/Lewis_et_al-2019-Risk_Analysis.pdf and https://www.fhi.ox.ac.uk/wp-content/uploads/C-Nelson-Engineered-Pathogens.pdf seem to be about gain of function research while completely ignoring the issue of potential lab leaks and only talking about it as an interesting biohazard topic.

My best guess is that it's like in math where applied researchers are lower status then theoretical researche... (read more)

24 comments, sorted by

top scoring

Click to highlight new comments since: Today at 2:16 PM

[-]gilch4y70

In 2014, in the LessWrong survey more people considered bioengineered pandemics a global catastrophic risk then AI.

It goes back further than that. Pandemic (especially the bioengineered type) was also rated as the greater risk in the 2012 survey,, and also the most feared in the 2011 survey, which was the earliest one I could find that asked the question.

It seems like it has been one of the global catastrophic risks we've taken most seriously here at LessWrong from the beginning. It's one of our cached memes. It's a large part of the reason that we rationalists, as a subculture, were able to react to the coronavirus threat so much more quickly than the mainstream. It was a possibility we had considered seriously a decade before it happened.

[-]Shmi4y40

Eliezer's X-risk emphasis has always been about extinction-level events, and a pandemic ain't one, so it didn't get a lot of attention from... the top.

[-]ChristianKl4y30

Events that kill 90% of the human population can easily be extinction level events and in 2014 more LessWrongers believed that pandemics do that then AI.

[-]Shmi4y30

I don't disagree that it was discussed on LW... I'm just pointing out that there was little interest from the founder himself.

[-]interstice4y30

Killing 90% of the human population would not be enough to cause extinction. That would put us at a population of 800 million, higher than the population in 1700.

[-]ChristianKl4y20

Shimux claims that Eliezer's emphasis was always about X-risk and not global catastrophic risks. If that's true why was the LW survey tracking global catastrophic risks and not X-risk?

[-]interstice4y10

I actually agree with you there, there was always discussion of GCR along with extinction risks(though I think Eliezer in particular was more focused on extinction risks). However, they're still distinct categories: even the deadliest of pandemics is unlikely to cause extinction.

[-]ChristianKl4y20

Modern civilisation depends a lot on collaboration. I think it's plausible that downstream of the destabilization of a deadly pandemic extinction happens, especially as the tech level grows.

[-]DPiepgrass4y10

That doesn't ring true to me. I'm curious why you think that, even though I'm irrationally short-termist: "100% is actually much worse than 90%" says my brain dryly, but I feel like a 90% deadly event is totally worth worrying about a lot!

[-]Liron4y30

A related question is why the topic of GoF research still didn’t get much LW discussion in 2020

[-]ChristianKl4y20

For my part I would say that in 2020 thinking about how to deal with the pandemic was a topic that reduced the available attention for other topics.

I also mistakenly thought that Fauci & Co are just incompetent and not actively hostile.

[-]ChristianKl4y20

After spending two days reading more 90% now feels way to low and 99% more reasonable because so many different stings of evidence point at the same conclusion that it was a lab leak.

[-]gilch4y40

I already think a lab leak is more likely than not, and did months ago when I first heard the circumstantial case for the hypothesis, but I'm nowhere near 99%. I'd say ~65%, but that's just my current prior, and I might not be calibrated that well. The fact that other rationalists are more confident about this than I am makes me want to update in that direction, but I also don't want to double-count my evidence. I'm also worried about confirmation bias creeping in. Can you summarize the strongest points and their associated Bayes factors? Or factors with error bars, if you prefer?