At the beginning of November, I learned about a startup called Red Queen Bio, that automates the development of viruses and related lab equipment. They work together with OpenAI, and OpenAI is their lead investor.
On November 13, they publicly announced their launch. On November 15, I saw that and made a tweet about it: Automated virus-producing equipment is insane. Especially if OpenAI, of all companies, has access to it. (The tweet got 1.8k likes and 497k views.)
In the tweet, I said that there is, potentially, literally a startup, funded by and collaborating with OpenAI, with equipment capable of printing arbitrary RNA sequences, potentially including viruses that could infect humans, connected to the internet or managed by AI systems.
I asked whether we trust OpenAI to have access to this kind of equipment, and said that I’m not sure what to hope for here, except government intervention.
The only inaccuracy that was pointed out to me was that I mentioned that they were working on phages, and they denied working on phages specifically.
At the same time, people close to Red Queen Bio publicly confirmed the equipment they’re automating would be capable of producing viruses (saying that this equipment is a normal thing to have in a bio lab and not too expensive).
A few days later, Hannu Rajaniemi, a Red Queen Bio co-founder and fiction author, responded to me in a quote tweet and in comments:
This inaccurate tweet has been making the rounds so wanted to set the record straight.
We use AI to generate countermeasures and run AI reinforcement loops in safe model systems that help train a defender AI that can generalize to human threats
The question of whether we can do this without increasing risk was a foundational question for us before starting Red Queen. The answer is yes, with certain boundaries in place. We are also very concerned about AI systems having direct control over automated labs and DNA synthesis in the future.
They did not answer any of the explicitly asked questions, which I repeated several times:
- Do you have equipment capable of producing viruses?
- Are you automating that equipment?
- Are you going to produce any viruses?- Are you going to design novel viruses (as part of generating countermeasures or otherwise)?
- Are you going to leverage AI for that?- Are OpenAI or OpenAI’s AI models going to have access to the equipment or software for the development or production of viruses?
It seems pretty bad that this startup is not being transparent about their equipment and the level of possible automation. It’s unclear whether they’re doing gain-of-function research. It’s unclear what security measures they have or are going to have in place.
I would really prefer for AIs, and for OpenAI (known for prioritizing convenience over security)’s models especially, to not have ready access to equipment that can synthesize viruses or software that can aid virus development.
340% is good enough! The bar gets higher on further levels.
(It doesn’t particularly increase the difficulty, unless you cross like 90%, in which case it adds new notes.)
Typo for Obama, thanks!
Trump still has the ability to launch nukes.
I'm very confused about how they're evaluating cost-effectiveness here. Like, no, spending $200 on vaccines in Africa to save lives seems like a much better deal than spending $200 to cause one more $400k apartment to exist.
I’m accumulating a small collection of spicy previously unreported deets about Anthropic for an upcoming post. Some of them sadly cannot publish because they might identify the sources. Others can! Some of those will be surprising to staff.
If you can share anything that’s wrong with Anthropic, that has not previously been public, DM me, preferably on Signal (@ misha.09)
Yes, I’ve read their entire post. $14.4 of “social return” per $1 in the US seems incredibly unlikely to be comparable to the best GiveWell interventions or even GiveDirectly.
Is there a write up on why the “abundance and growth” cause area is an actually relatively efficient way to spend money (instead of a way for OpenPhil to be(come) friends with everyone who’s into abundance & growth)? (These are good things to work on, but seem many orders of magnitude worse than other ways to spend money.)
1My understanding is that Habryka spent hours talking to the third party as a result of receiving the information.
I mistakenly assumed a pretty friendly/high-trust relationship in this context due to previous interactions with the Lightcone team and due to the nature of the project we (me and Habryka) were both helping with.
I think the interests of everyone involved (me, Habryka, and the third party) are all very aligned, but Habryka disagrees with my assessment (a part of the reason of sharing the information was to make him talk to the third party and figure out that they're a lot more aligned than Habryka assumed).
I did not make the offer, because from the context of the DMs had assumed that Habryka's problem with the idea was keeping secrets at all, for deontology-related reasons and not because of the personal price of how complicated that is. I would've been happy to pay a reasonable price.
(Elsewhere, Habryka said the price would've been "If you recognize that as a cost and owe me a really small favor or something, I can keep it private, but please don't take this as a given".)
Would maybe work if you have to write >500 new words every day + publish >3500 words every week, as two separate requirements.
Or maybe add explicit requirements for polishing things?
Working more on polishing is valuable, but I think most XP points are gained just from writing.
I didn't particularly present any publicly available evidence in my tweet. Someone close to Red Queen Bio confirmed that they have the equipment and are automating it here.