I think a bounty for actually ending malaria is great. I think a bounty for unilaterally releasing gene drives is probably quite bad for the world.
Like, I think malaria is really bad, and worth making quite aggressive sacrifices for to end, but that at the end of the day, there are even bigger games in town, and setting the precedent of "people are rewarded for unilaterally doing crazy biotech shenanigans" has a non-negligible chance of increasing global catastrophic risk, and potentially even existential risk.
I think the pathways towards doing that are twofold:
-
We further erode the currently very fragile and unclear norms around not unilaterally releasing pathogen-adjacent stuff. While I think specifically a malaria gene-drive is unlikely to have catastrophic consequences here, the same logic feels like it gets you much closer to stuff that does sure end up dangerous, or towards technologies that enable much more dangerous things (in-general I am pretty wary of naive consequentialist reasoning of this type. A negative utilitarian could go through the same reasoning to conclude why it's a good idea to release an omnicidal pathogen)
-
We broadly destabilize the world by having more people do naive consequentialist estimates of stuff, and then taking large world-reshaping actions to achieve them, without consulting very much with the people who are most likely to have properly thought through the consequences of those actions. In this case, my sense is that actually rushing to release gene drives is not a great idea and might very well prevent future gene drives.This is like a classical unilateralist situation, and clearly it's much worse if we somehow fuck up our gene drives forever than if we have to have a few more years of malaria. I think there is a time when it makes sense to go "wow, the current delay here is really unacceptable and someone should just go ahead and do this", but I don't think that time is right now, and this bounty feels like it pushes towards the "release faster" direction.
Like, I think actions can just have really large unintended consequences, and this is the kind of domain where I do actually like quite a bit of status quo bias and conservatism. I frequently talk to capabilities researchers who are like "I don't care about your purported risks from AI, making models better can help millions of people right now, and I don't want to be bogged down by your lame ethical concerns", and I think this reasoning sure is really bad for the world and will likely have catastrophic consequences for all of humanity. I think this posts treatment of the gene drives issue gives me some pretty similar vibes, and this is a reference class check I really don't want to get wrong.
At the object level I think actors like Target Malaria, the Bill and Melinda Gates Foundation, Open Philanthropy, and Kevin Esvelt are right to support a legal process approved by affected populations and states, and that such a unilateral illegal release would be very bad in terms of expected lives saved with biotech. Some of the considerations:
- Eradication of malaria will require a lot more than a gene drive against Anopheles gambiae s.l., meaning government cooperation is still required.
- Resistance can and does develop to gene drives, so that development of better drives and coordinated support (massive releases, other simultaneous countermeasures, extremely broad coverage) are necessary to wipe out malaria in regions. This research will be set back or blocked by a release attempt.
- This could wreck the prospects for making additional gene drives for other malaria carrying mosquitoes, schistosomiasis causing worms, Tsetse flies causing trypanosomiasis, and other diseases, as well as agricultural applications. Collectively such setbacks could cost millions more lives than the lost lives from the delay now.
- There could be large spillover to other even more beneficial controversial biotechnologies outside of gene drives. The thalidomide scandal involved 10,000 pregnancies with death or deformity of the babies. But it led to the institution of more restrictive FDA (and analogs around the world imitating the FDA) regulation, which has by now cost many millions of lives, e.g. in slowing the creation of pharmaceuticals to prevent AIDS and Covid-19. A single death set back gene therapy for decades. On the order of 70 million people die a year, and future controversial technologies like CRISPR therapies may reduce that by a lot more than malaria eradication.
I strongly oppose a prize that would pay out for illegal releases of gene drives without local consent from the affected regions, and any prizes for ending malaria should not incentivize that. Knowingly paying people to commit illegal actions is also generally illegal!
Putting aside the concerns about potential backfire effects of unilateral action[1], calling the release of gene drive mosquitoes "illegal" is unsubstantiated. The claim that actually cashes out to is "every single country where Anopheles gambiae are a substantial vector for the spread of malaria has laws that narrowly prohibit the release of release of mosquitoes". The alternative interpretation, that "every single country will stretch obviously unrelated laws as far as necessary to throw the book at you if you do this", may be true, but isn't very interesting, since that can be used as a fully general argument against doing anything ever.[2]
- ^
Which I'm inclined to agree with, though notably I haven't actually seen a cost/benefit analysis from any of those sources.
- ^
Though you're more likely to have the book thrown at you for some things than for others, and it'd be silly to deny that we have non-zero information about what those things are in advance. I still think the distinction is substantial.
Unilateral action in general might be bad, but most of these reasons you've given to not support an illegal one (if gene drives were explicitly illegal, which they're not) seem completely misguided or misleading. I can't parse whether or not this is deliberate. I'm against lying as a means of stopping unilateral action in most real world scenarios; people who want to obtain or give multilateral consensus will need to understand where actual risks come from, not made up risks designed to discourage bad actors.
Eradication of malaria will require a lot more than a gene drive against Anopheles gambiae s.l., meaning government cooperation is still required.
AFAICT this is basically incorrect. What government cooperation do you suspect is necessary? You have to release a "significant amount" of mosquitoes across a "large enough" part of Africa, yes, but unless the local population is unusually isolated, the mosquitoes travel to other countries. Metacelsus estimates you could do it with maybe ~50k on a shoestring budget and most of the work would be catching the mosquitoes.
If what you really mean is "you have to not be actively hunted by every local government that has jurisdiction over a suitable place to start the gene drive" then A. that level of cooperation currently exists, and B. I think you are both underestimating the number of governments in Sub-Saharan Africa and overestimating their state capacity.
Resistance can and does develop to gene drives, so that development of better drives and coordinated support (massive releases, other simultaneous countermeasures, extremely broad coverage) are necessary to wipe out malaria in regions. This research will be set back or blocked by a release attempt.
Resistance may develop to the doublesex gene, and it may not. So far none of the trials done by Target Malaria have suggested this happens in practice. It is of course still theoretically possible, for that gene, that a resistance could develop at larger population sizes, which is one of the only good reasons why you shouldn't unilaterally release gene drive mosquitoes. But there are other gene drives you could deploy that would simply engineer mosquitoes not to transmit the parasite for malaria, so this is not a concern about gene drives in general, just the genes that introduce sterility.
Other than resistances, the claim "research will be set back or blocked by a release attempt" is also obviously incorrect. If you somehow only exterminate a completely isolated subsection of the larger population of Anopheles gambiae (which again, I find pretty unlikely) you can just try again. The Senegalese government is not going to both detect and then ban gene drives in response to a deployment that failed to reach escape velocity.
This could wreck the prospects for making additional gene drives for other malaria carrying mosquitoes, schistosomiasis causing worms, Tsetse flies causing trypanosomiasis, and other diseases, as well as agricultural applications. Collectively such setbacks could cost millions more lives than the lost lives from the delay now.
Read one way, this is a bizarrre technical misconception: the doublesex gene does not work on worms. A gene drive on Anopholese Gambiae would involve epsilon risk introducing a resistance on schistosomiasis causing worms.
Read another way:
There could be large spillover to other even more beneficial controversial biotechnologies outside of gene drives. The thalidomide scandal involved 10,000 pregnancies with death or deformity of the babies. But it led to the institution of more restrictive FDA (and analogs around the world imitating the FDA) regulation, which has by now cost many millions of lives, e.g. in slowing the creation of pharmaceuticals to prevent AIDS and Covid-19. A single death set back gene therapy for decades. On the order of 70 million people die a year, and future controversial technologies like CRISPR therapies may reduce that by a lot more than malaria eradication.
The FDA does not, in fact, have jurisdiction over any of the places affected by the parasites you mentioned. The U.S. government has the money, political capital, and state capacity to start institutions like the FDA, Kenya generally does not. And the U.S. government generally starts deadly regulatory crazes like that in response to things happening to U.S. citizens, not Africans.
So, altogether, what you are proposing will/may happen if someone releases a gene drive to eradicate mosquitoes without getting "government permission" first is:
- It doesn't work.
- Even though it didn't work to kill mosquitoes, it, through a completely unspecified and at this point imaginary mechanical process, kills a small number of Kenyans, makes some women bear deformed children, etc.
- This negative side effect of the gene drive is somehow detected by either e.g. the Kenyan government or NGOs from first world countries.
- Those first world nations unanimously care enough about the death of ~1/10/100 Kenyans from a bad gene drive in Kenya to block CRISPR technology in agriculture, or maybe other third world nations unanimously care enough to proactively and effectively block further attempts to destroy other parasites.
In other words: Despite the fact that the biotechnology efforts to kill those parasites you mentioned do not currently exist, we're supposed to wait until 2030 to deploy the gene drive that stops malaria, because if they did exist they might be stopped in response to a side effect that does not sound mechanically possible and does not sound like it would be detected if it were possible. Pretty ridiculous IMO, even granting the (incorrect, but common on LessWrong) model of regulation that says the existence of FDA analogs is a result of every government "copying" the FDA, rather than normies just being extremely conservative when it comes to medicine as applied to humans.
This is frankly what jimrandhomh talks about when he says the safety concerns about "gene drives" are analogous to safety concerns about "nuclear power", in that they can't be addressed by science or policy predictions because they're actually anxieties provoked by the words "gene", "biotechnology", "unilateral", etc. Would you like to make some bets regarding this story? I'll give you great odds.
And this is all in addition to RobertM's obvious critique that releasing gene drives is not, in fact, actually illegal. I'm keeping Habryka's unilateral action clause, was always going to include some clause that says we're not going to pay out the bounty if it would mean pledgers get prosecuted, solely because more people are likely to be willing to pledge that way. These concerns do not seem to be coherent enough to merit amending the bounty otherwise.
without consulting very much with the people who have tried thinking more deeply through the consequences
I'm not sure what the standard is here, but this doesn't feel quite right. Sometime people who have tried thinking a lot about an area are worth ignoring (e.g. the field of medical ethics).
1Agree that this shouldn't be measured in "amount of time thinking through something" or in "degree to which they look like they are supposed to have thought through something". I think it should just be measured in the likelihood that a group of people has actually figured out the most important considerations.
I edited it a bit. Now says "without consulting very much with the people who are most likely to have properly thought through the consequences of those actions".
Yeah, if you can find a group of people who have thought deeply about the consequences that seems great. I also think for many great actions, there may not be any such body of people. In such a case (my first thought is that) I would try to create one? Like, find some very intelligent people with enough understanding of the details to engage on the object level, and who disagree, and put in the effort to debate them, and then actually change their minds. Might be a bit more robust.
(But of course, in our civilization, there are no real adults who can make sure a decision is the right one for the whole course of the world, there are only gradations of reduction of uncertainty, and at some point you have to make decisions according to your most true-and-tried principles.)
I think a bounty for actually ending malaria is great. I think a bounty for unilaterally releasing gene drives is probably quite bad for the world.
Like, I think malaria is really bad, and worth making quite aggressive sacrifices for to end, but that at the end of the day, there are even bigger games in town, and setting the precedent of "people are rewarded for unilaterally doing crazy biotech shenanigans" has a non-negligible chance of increasing global catastrophic risk, and potentially even existential risk.
Like, I think actions can just have really large unintended consequences, and this is the kind of domain where I do actually like quite a bit of status quo bias and conservatism. I frequently talk to capabilities researchers who are like "I don't care about your purported risks from AI, making models better can help millions of people right now, and I don't want to be bogged down by your lame ethical concerns", and I think this reasoning sure is really bad for the world and will likely have catastrophic consequences for all of humanity. I think this posts treatment of the gene drives issue gives me some pretty similar vibes, and this is a reference class check I really don't want to get wrong.
Alright. You're probably right. And I don't want to increase existential risk. But I do want the person or group that ends Malaria to get >=$5,000, which is what this bounty is actually written to do, gene drive or no. Is there a way we can actually do that, or should I scrap it entirely?
Can we add an amendment that requires significant consultation with stakeholders? Should it be mandatory that it be done as part of a large nonprofit so that future individual people aren't encouraged to act unilaterally anyways? Should it be amended to spread the money also among the people doing reasonable, informed research on safety concerns?
What would responsible buy-in actually look like, for technology like this? I hope not just "get buy in from local elected political leaders", as I don't expect that to actually correlate well with risk at all.
I am literally yours here; I don't like unilateral action either, and am interested in introducing whatever safety protocols you propose. But if nobody can come up with an N-step process to turn unilateral action into multilateral consensus, the anxiety is not about this particular action being unilateral, it's about the action having unintended negative consequences at all.
In this specific case, I feel like I think the problem would basically be solved if you would just say "I am not going to award this bounty if you unilaterally release a gene drive, without also writing a post that convinces me that the effects of doing so in a rushed way were worth the costs". Basically just inverting the burden of proof in that one case.
Done then.
I'll think about this more and formalize it a bit before I actually create the nonprofit's pledge.
I agree about inverting the burden of proof in that case. I'd prefer to operationalize "unilaterally" more. Here's an alternative:
"I am not going to award this bounty if other people with a strong understanding of the science involved point out straightforward flaws that make the project appear catastrophically net negative, or if a post on LessWrong about the project leads users to point out straightforward reasons why the project is catastrophically net negative, or if I think you made no attempt to get such people to actually check and then change their minds (or engage with their arguments for 100+ hours) before going ahead."
I pledge to match the bounty of the next person to pledge $5,000, because of research showing this encourages people to donate money.
So if someone else pledges, we’ve reached a third of the median US salary.
I'll add $5,000 to the pot.
Um, I want to reserve the right to judge differently than lc, if I disagree with them about e.g. whether something turned out negative. I don't particularly expect that to happen, but just in case.
Actually it would be interesting to have a systematic way to pledge.
It would be a great way to incentivize actions and research in a domain. I am thinking of a system like petition or reverse crowdfunding. Maybe it could be easier enforced by smart contracts, without the need for a central authority in which in the trust would need to resign.
If you already know a system like this or if you have any idea on how to design solutions for the first problems I encountered by thinking about it for two minutes, please don't hesitate to give me your input.
Enough people have decided to pledge, here and in private messages, that during october I will work with a lawyer to create an actual nonprofit and give you guys a place to donate money/crypto/whatever to, as well as potentially draft a pledge contract for U.S. citizens. I am busy for the next week or so but I should be able to get it done during the first two weeks of October. I will notify everyone here that has stated interest/intent to pledge when that's available, and/or make another post.
Update: turns out creating a nonprofit is actually really hard and my lawyer advised against it. Lol. My personal pledge (and hopefully the pledge of the other people here) still stands informally, but I'm looking into alternative solutions to see if I can provide a place for others to donate and make it a little more official, or at least publicly logged. Will update sooner than last time to tell you all how it goes.
That's too bad. I look forward to what you come up with, but I don't think I can offer any help :-/
Many widely-known organizations in this field have publicly-identifiable persons and assets that can be pursued if they fail to follow through on their promises. For example, in the United States of America, the Internal Revenue Service requires some organizations to complete Form 990, which can be viewed by everyone.
Are there plans to provide a similar level of assurance to people who are interested in this cause?
It's perhaps somewhat morally cowardly to worry about it, but what about legal concerns? In general offering bounties for illegal acts is illegal, and often prosecuted if someone takes you up on the bounty. I don't think eliminating a mosquito species is explicitly illegal in any jurisdictions, but then again, one could certainly imagine prosecution in some scenarios.
You could add a clause like "This bounty will only be paid out if you did not break any laws of the United States in the process." Uncertainty in payouts reduces the value of a bounty, but it may be worth it to encourage additional pledges.
That aside, I think this is an excellent idea. It may be a small amount of money, but you could easily imagine a junior researcher in an org working on gene drives being influenced on the margin by a bounty like this. I'll give some thought to contributing. If I'm convinced legal issues are immaterial I'm likely to match your bounty.
I guess this was inspired by my recent post: https://denovo.substack.com/p/gene-drives-why-the-wait
As I mentioned in that post, there are good reasons to not unilaterally release gene drives, so please exercise some restraint. Also it would cost a lot more than $5000 to do it. (Maybe $50,000 on a shoestring budget.)
I didn't notice any mention of alternative to extinction type gene-drive solutions have been suggested in your investigation on the subject. Would it be that much more trouble to engineer a gene to attack malaria in the mosquito genome rather than one that sterilizes the female and leads to extinction?
Seems like that would address some/most of the ecological impact fears if we just gave the mosquito antibodies to kill the virus than killing off the mosquitos which do seem to have a role in a broader ecological context.
I think of one of the main experts here as Kevin Esvelt, the first person to suggest using CRISPR to affect wild populations. Here's an article largely based on interviews with him that he recommends, explaining why he's against unilateral action here:
"Esvelt, whose work helped pave the way for Target Malaria’s efforts, is terrified, simply terrified, of a backlash between now and then that could derail it. This is hardly a theoretical concern. In 2002, anti-GMO hysteria led the government of Zambia to reject 35,000 tons of food aid in the middle of a famine out of fear it could be genetically modified. Esvelt knows that the CRISPR gene drive is a tool of overwhelming power. If used well, it could save millions of lives, help rescue endangered species, even make life better for farm animals. If used poorly, gene drives could cause social harms that are difficult to reverse. . . .
“To the extent that you or I say something or publish something that reduces the chance that African nations will choose to work with Target Malaria by 1 percent, thereby causing a 1 percent chance that project will be delayed by a decade, the expected cost of our action is 25,000 children dead of malaria,” Esvelt tells me. “That’s a lot of kids.”"
Is the tech to create a gene drive and spread it - something a hypothetical determined person can do from their garage ?
I wish I had a better source, but in this video, a journalist says that a well-equipped high schooler could do it. The information needed seems to be freely available online, but I don't know enough biology to be able to tell for sure. I think it is unknown whether it would spread to the whole population given a single release, though.
If you want it to happen and can't do it yourself nor pay someone else to do it, the best strategy might be to pay someone to translate the relevant papers into instructions that a regular smart person can follow and then publish them online. After making sure to the best of your capabilities (i.e. asking experts the right questions) that it actually is a good idea, that is.
RE creating an instruction manual:
I strongly vote against increasing the number of people able to unilaterally decide that an arbitrary species should be extinct. I think there are already many thousands of such people, and I don't want there to be millions.
(I'm less strongly opposed if the instruction manual were somehow specific to mosquitoes and completely useless for any other plant or animal.)
I guess it might be possible to repurpose the manual, but making mosquito species extinct isn't the only possible method, e.g. https://bioengineeringcommunity.nature.com/posts/gene-drive-rescue-system-for-population-modification-of-a-malaria-mosquito .
So in order to build a gene drive, you'd need to build the DNA construct (pretty easy in a garage), introduce it into mosquitoes (not easy at all in a garage), and then breed enough to release (I'm not sure how easy in a garage, but probably not very easy).
Depending on the country you live in, you may be able to order GE organisms from specialist companies or universities, with higher likelihood of success if you do so on behalf of a company/university.
This bounty seems to be written in a way where you expect the person to be public about their actions. Is that intentional? It might also be possible to pay an anonymous actor via crypto.
This got me thinking about how an anonymous actor could prove responsibility. It occurred to me that they could write their bitcoin address into the genome of the modified mosquitos. I don't know if that's how gene drives work, but it's an interesting premise for a sci-fi story in any case.
Let's say that I am willing to do the thing and take public social/legal responsibility. Who would I contact to work out an action plan? I would really appreciate some advice, both legal and technical. Also, is anyone here in contact with people actually working on gene drive field testing? A lot of recent discussion here about why deployment has been delayed seem like conjecture.
It occurred to me recently that if a group of people eradicated malaria, say by exterminating Anopheles gambiae via gene drives, they would not be paid. That sounds like a potentially embarrassing mistake, so I'm hereby pledging $5,000 to anyone that manages to accomplish such a thing. This way if someone from Dath Ilan materializes into our world and asks us judgmentally whether or not Earth would even pay people for eradicating malaria, we can all respond "Our society definitely has an explicit, preregistered financial reward greater than $0 in place for anybody that does that" and sidestep some awkwardness.
Below I have written out the details for this new bounty in FAQ form.
Malaria Bounty FAQ
What specifically will lc wire me $5,000 for doing?
I will give you $5,000 if you permanently reduce global annual mortality from malaria by 95% or more. Eligibility for the reward is independent to how this is actually accomplished, save for two restrictions outlined below in the sections about unilateral action and negative side effects.
How will lc assign credit for solving malaria? Does developing the deployed technique count?
The money will be given to the person or group that directly causes the reduction in deaths. If a pioneering group of researchers wants to get my money, they will have to follow through by successfully managing and executing the deployment of their clever research to prevent people from dying.
I would like to handsomely reward both groups, but there are already many existing charities you can extract money from in advance, if you're working on tools to end malaria and want funding for that. So far, none of the (awesome) people developing those tools have opted to use them, partly because they insist on getting permission from local African governments, which are delaying them for political reasons. I reserve hope that I will some day have reason to send these researchers money, after a few more million people have died, but in the meantime there's not as much marginal benefit.
So can I get the bounty via unilaterally deploying biotech?
No, and this is the major restriction on payouts.
By "unilaterally" I mean: I am not going to award this bounty if other people with a strong understanding of the science or existing efforts suspect the project is/was catastrophically net negative, or if a post on LessWrong about the project leads users to point out straightforward reasons why the project is catastrophically net negative, or if I believe you made no attempt to get such people to actually check and then change their minds (or engage with their arguments for 100+ hours) before going ahead. There must be broad-based consensus among smart, informed people prior to your project's creation and launch that what you're doing is at least not going to backfire terribly. If someone thoughtful and scope-sensitive does believe your project's approach is going to backfire terribly you need to both have talked to that person and thoroughly documented your disagreements beforehand and gotten the OK from most other thoughtful people that your disagreements are obviously valid.
To give an example of a party meeting this bar: Target Malaria, if they were to decide to deploy a gene drive in the next few years, I would not consider to be acting unilaterally. They are taking great pains to do boatloads of prior safety research and get the blessings of every conceivable relevant person. They are also making sacrifices to PR that do not seem like they decrease existential risk from such a deployment, but they are definitely sufficiently cautious in the relevant ways.
So do I have to apply somewhere?
Nope. You may decide to contact me and provide relevant evidence so I am aware that malaria has been solved, but I expect I'll be proactively contacting the organization that actually does this after I hear about them doing it through the news. You just have to have respected unilateralists' curse and actively achieved buy-in from the majority of the most informed, reasonable people.
What if someone non-unilaterally eradicates malaria, in a way that still causes a legible negative side effect that someone gets angry about?
In that case, I shall construct and consult an expert technology ethics review board, staffed by myself. This ethical review board will formally determine whether or not the bad thing is veritably and legibly bad enough to outweigh saving hundreds of thousands of lives per year. If the review panel's unanimous finding is that this is indeed the case, then the party responsible will not receive the $5,000.
Examples of some malaria reduction techniques which could cause someone to be disqualified by the expert panel include:
Examples of secondary effects that are explicitly noted not to disqualify award recipients include:
In general, the review board will be asking themselves: "Would we rather 200 million people a year start becoming afflicted with malaria than this have happened?" If they conclude not, then that is a strong sign they will not disqualify you for those results.
What if someone mostly eradicates malaria, and nothing bad happens to real breathing humans as a direct consequence, but in the process of doing so they violate illegible principles of self determination?
I will still pay that person $5,000.
What if more than one person seems directly responsible for solving malaria?
If they're all members of the same organization dedicated to doing that, the reward will be sent to that organization, and then its overseers can distribute it as they wish. If instead it's a group of less than ten unaffiliated people, then I will send a fraction of the reward to each of them. If the group has ten or more such individuals I will send 500$ to a random selection of ten people, in order to avoid large potential transaction fees.
Why not wait to start the bounty until you've gathered more money?
It's probably true that this post is not going to convince anyone to solve the problem that wasn't going to anyways. However at least half of justice is rewarding people if they do good things, such as saving millions of lives. So it seems better to me that the person or group who solves malaria gets $5,000 instead of $0, even if they didn't expect to, or weren't primarily motivated by money. And I may simply never get around to gathering a bigger pile of gold for this.
On the other hand, a larger and more commensurate bounty would prevent marginal embarrassment in case the extradimensional visitor asks followup questions. So I will certainly accept additional pledges from anybody that wants to add money to what I understand today to be Earth's sole bounty for ending malaria. Feel free to express interest by commenting here, DMing me, or contacting me over session[1]. If someone does that, I'll go ahead and set up some kind of very small nonprofit with more formalized payout requirements.
Feel free to start your own bounty as well.
My id is
056724a0e9c1fefc86227f178cb8361e07bdcfd17ef78aa0d0d81c4c5ebbb2ad02