I'm happy that this was done before release. However ... I'm still left wondering "how many prompts did they try?" In practice, the first AI self-replicating escape is not likely to be a model working alone on a server, but a model carefully and iteratively prompted, with overall strategy provided by a malicious human programmer. Also, one wonders what will happen once the base architecture is in the training set. One need only recognize that there is a lot of profit to be made (and more cheaply) by having the AI identify and exploit zero-days to generate and spread malware (say, while shorting the stock of a target company). Perhaps GPT-4 is not yet capable enough to find or exploit zero-days. I suppose we will find out soon enough.

Note that this creates a strong argument for never open-sourcing the model once a certain level of capability is reached: a GPT-N with enough hints about its own structure will be able to capably write itself.

(Spoilers for Interstellar)

I sat next to person on a flight a few weeks ago who, upon talking about physics, said she thought the movie Interstellar was "amazing" and "scientific". I agreed with her, thinking she was talking about the realistic black hole simulations. No, she was talking about scenes where the main character reaches back in time as a ghost to influence his daughter.

This person was a first-year Ph.D. student in medicine.

So yes, even when science fiction is done relatively carefully, some people will take as "scientific" the parts which have been stretched for better storytelling.

There are a few examples in history of civilizations running out of critical resources, usually accompanied by other conflict and calamity. In the most extreme historical cases, these civilizations go extinct in ways that left us with poor documentation, so there may be an anthropic / selection bias which obscures the exact causes of civilizational collapse:

• Deforestation and the introduction of rats in Easter Island in the 1500s led to [ecological collapse](https://en.wikipedia.org/wiki/Easter_Island) and reduction in population from ca. 15k to 2k~3k. However, [some dispute this](https://en.wikipedia.org/wiki/Collapse:_How_Societies_Choose_to_Fail_or_Succeed#Criticisms) and claim the collapse was due to disease.
• The 11th century collapse of North African civilization is attributed to exhaustion of resources in concert with foreign invasion. Here exhaustion of resources refers to desertification.
• The collapse of irrigation in Mesopotamia is attributed to the 13th century invasion of Mesopotamia by Genghis Kahn is attributed and led to the transition of the region into nomadic tribes.

There are several instances of social destabilization following closely on this heels of famine and rising food prices. However, famine is almost always a result of political or economic mismanagement, rather than pure constraints on non-food resources. The Arab Spring is sort of the canonical example of food scarcity resulting in higher food prices and subsequent political instability.

However, political instability is not a de facto result of food shortages. The 20th century saw many famines with relatively limited political consequences: the [Russian Famine of 1921](https://en.wikipedia.org/wiki/Russian_famine_of_1921%E2%80%931922), the [Soviet famine of 1932](https://en.wikipedia.org/wiki/Soviet_famine_of_1932%E2%80%931933), the [Great Leap Forward/Great Chinese Famine](https://en.wikipedia.org/wiki/Great_Chinese_Famine), and the [1996 famine in North Korea](https://en.wikipedia.org/wiki/North_Korean_famine) are prominent examples. It may be noted that none of these countries were technologically advanced or democratic: perhaps free communications are a prerequisite to shortage-induced political instability?

Caveat: history is really complicated, and I am only parroting popular views. I am not a historian, nor do I have any detailed knowledge of these events.

What's the practical difference between "text" and one-hots of said "text"? One-hots are the standard for inputting text into models. It is only recently that we expect models to learn their preferred encoding for raw text (cf. transformers). By taking a small shortcut, the authors of this paper get to show off their agent work without loss of generality: one could still give one-hot instructions to an agent that is learning to act in the real life.

The last time CM was mentioned on here I looked up his old videos about Fauci. In a video from Sept 2020 he made a very confident claim that NYC had already achieved 70% infections and thereby herd immunity. That turned out to be untrue:

https://www.lesswrong.com/posts/xEFfbEMFHhtgseKz3/covid-6-10-somebody-else-s-problem

So my prior for him is now skewed toward "pundit" rather than "honest inquirer."

Immune response is generally associated with age and lifestyle. What can you tell us about those factors?

it means contact tracing becomes easier and lockdowns more effective. 

I read that oppositely. If the serial interval is shorter, contact tracers need to work faster to inform those exposed, lest those exposed become infected and transmitting. Likewise, lockdowns would only become more effective if the time each person is contagious is reduced. IIRC the delta variant, according to Indian accounts, is significantly contagious for three weeks from the date of infection as opposed to the usual two.

Re: quarantines for fully vaccinated travellers

The problem a national government has in setting quarantine standards for vaccinated people is that there are so many inconsistencies to vaccines, and the topic is wrapped in geopolitics.

• Do you exempt (from quarantine) those who claim to have gotten vaccines in countries where vaccine documentation is easy to fake? What about countries where vaccine documentation is nonstandard between regions?
• How do you (cheaply) verify vaccine documentation from each foreign country / region?
• Does refusing to exempt people from poorly-documented countries cause a diplomatic incident with those countries?
• Do you exempt those who have documented vaccines which are not authorized in your country? If so, what are your criteria for accepting vaccines?
• If not, then the home countries of inelligible vaccines will lodge a diplomatic protest, and maybe accuse you of racism.
• Or do you only exempt those who got vaccines which are produced in your country? (Chinese-style vaccine nationalism.)
• Do you exempt those who got vaccines which are authorized in your country, but are coming from countries where immunity-escaping strains are common?
• Do you exempt those who got the vaccine, but are in a demographic likely to have poor immune response?
• If you exempt people with vaccine immunity, do you exempt people with positive COVID antibody tests?
• If you exempt people with antibody tests, which tests do you exempt? See all the previous questions about reliability of documentation again...

The cost to administration of such an exemption program is high, the social, political, and medical perils are many, and there is the chance of quarantine escape in >5% of cases. Vaccine exemption from national quarantine is unlikely.