Petrov Day

The day the world nearly ended.

An illustrated guide to the Petrov Day ceremony.

Petrov Day is a yearly event on September 26 commemorating the anniversary of the Petrov incident, where a false alarm in the Soviet early warning system nearly set off a nuclear war.

Since 2014, some people have celebrated Petrov Day with a small in-person ceremony, with readings by candlelight that tell the story of Petrov within the context of the long arc of history.

This year, we're presenting an illustrated online version of the ceremony, to generally honor the event and share the ceremony with a wider audience.

The purpose of the ritual is to make catastrophic and existential risk emotionally salient, by putting it into historical context and providing positive and negative examples of how it has been handled.

It is aimed at those who already know what catastrophic and existential risk is, have some background knowledge of what those risks are, and believe (at least on an abstract level) that preventing those risks from coming to pass is important.

If you're not in a good emotional place for thinking about the end of the world, consider not reading through right now.

The
Petrov Day Ceremony

This day, September 26, is Petrov Day. In 1983, the story of humanity nearly ended. We're gathered here to remember that moment, and others like it. But to really feel the magnitude of those events, we need to visit them in their proper context. Let us begin the story of human history, starting from the beginning.

In the beginning, the universe was created. This has made a lot of people very angry, and been widely regarded as a bad move.

Let's fast forward over the thirteen billion year long prequel. Our story begins in the age of myth, of fossils and legends. It starts with the invention of fire.

I've hunted down and stolen, inside the hollow of a fennel's stalk, the seed of fire, a gift that has proven itself to be the teacher of every craft and the greatest resource for humans. Such is the crime I have committed and this is the penalty I am to suffer: nailed and chained on this rock beneath the open sky.

Light the left-most candle, to represent the invention of fire. Point out the location of the nearest fire extinguisher, then dim or turn off all other lights in the room.

Depending which archaeologists you ask, fire was first used by either Homo Erectus or Homo Ergaster, some time between 400 thousand and 1.7 million years ago. Cooking is believed to have enabled larger, more energy-intensive brains, allowing the evolution of increased intelligence, and language.

Most species do their own evolving, making it up as they go along, which is the way Nature intended. And this is all very natural and organic and in tune with mysterious cycles of the cosmos, which believes that there's nothing like millions of years of really frustrating trial and error to give a species moral fiber and, in some cases, backbone.

It certainly is not a true instinct, for every language has to be learnt. It differs, however, widely from all ordinary arts, for man has an instinctive tendency to speak, as we see in the babble of our young children; whilst no child has an instinctive tendency to brew, bake, or write.

Take the first candle, which represents the invention of fire. Use it to light the second candle, which represents the evolution of language.

Pass the candle once all the way around the circle. When you hold the candle, it is your turn to speak. What is your name, and when (what year) is your earliest memory?

When everyone has spoken, put the candle back in the candelabrum.

Language is the first key to technology; with it, early humans could accumulate knowledge, not just in genes, but also in sayings and traditions.

They gave names to people around them. They gave names to species of animals and plants. They gave names to actions and to places and to strategies. They called some of these good, and called some of them bad. They learned to share their knowledge, and they learned to deceive each other. They built families and communities.

They began the long, slow process of taming the wilderness. Their tribes grew to cities. What became of them?

Take the second candle, which represents language. Use it to light the third candle, which represents agriculture.

If you or someone else at the table knows the tune to this song, then sing; if not, read normally.

Uplift

By Andrew Eigel

Hands chip the flint, light the fire, skin the kill
Feet move the tribe track the herd with a will
Mankind struggles in the cellar of history
Time to settle down, time to grow, time to breed

Plow tills the soil, plants the seed, pray for rain
Scythe reaps the wheat, to the mill, to grind the grain
Towns and cities spread to empire overnight
Hands keep building as we chant the ancient rite

Stop here. Go to the next page without reading or singing the rest of the song.

Coal heats the steam, push the piston, turns the wheel
Cogs spin the wool, drives the horses made of steel
Lightning harnessed does our will and lights the dark
Keep rising higher, set our goal, hit the mark.

Crawl out of the mud,
Ongoing but slow,
For the path that is easy
Ain't the one that lets us grow!

Light to push the sails, read the data, cities glow
Hands type the keys, click the mouse, out we go!
Our voices carry round the world and into space
Send us out to colonize another place¸

Hands make the tools, build the fire, plant the grain.
Feet track the herd, build a world, begin again.

Take the third candle, which represents agricultural society. Pass it around the circle.

Then, blow it out. Then return it to its place in the candelabrum.

The power of population is so superior to the power of the earth to produce subsistence for man, that premature death must in some shape or other visit the human race. The vices of mankind are active and able ministers of depopulation. They are the precursors in the great army of destruction, and often finish the dreadful work themselves. But should they fail in this war of extermination, sickly seasons, epidemics, pestilence, and plague advance in terrific array, and sweep off their thousands and tens of thousands. Should success be still incomplete, gigantic inevitable famine stalks in the rear, and with one mighty blow levels the population with the food of the world.

Mankind lived in equilibrium between growth and collapse, knowledge gained and knowledge forgotten. In that world, stories would last only as long as memory, monuments only as long as wood. For two hundred thousand years, nothing but genes survived.

But that was enough. Though they could not preserve knowledge over generations, they could preserve domesticated plants and animals. They saved the best, and little by little, the world got easier. And then a select few humans started writing, and the equilibrium between learning and forgetting was finally broken.

Of that age, what memories remain?

Using the second candle, which represents language, relight the third candle to represent the invention of writing.

I met a traveller from an antique land
Who said: Two vast and trunkless legs of stone
Stand in the desert. Near them, on the sand,
Half sunk, a shattered visage lies, whose frown,
And wrinkled lip, and sneer of cold command,
Tell that its sculptor well those passions read
Which yet survive, stamped on these lifeless things,
The hand that mocked them and the heart that fed:
And on the pedestal these words appear:
``My name is Ozymandias, king of kings:
Look on my works, ye Mighty, and despair!''
Nothing beside remains. Round the decay
Of that colossal wreck, boundless and bare
The lone and level sands stretch far away

When you have finished reading, take a piece of paper and write down the name of the oldest family member - living or dead - that you can identify.

When everyone has written something, continue to the next page.

We know more about what the world was like after people started writing, but not very much survived. One of the most important writings was discovered by French soldiers in the wall of Fort Julien: the Rosetta Stone, important because it was written in three languages, two previously untranslatable. After a long string of honorifics and decrees about taxes and succession, it declares: there shall be a new holiday!

On these days in every month, on which there shall be sacrifices and libations and all the ceremonies customary at the other festivals, and the offerings shall be given to the priests who serve in the temples. And a festival shall be kept for King Ptolemy, the Ever-Living, the Beloved of Ptah, the God Epiphanes Eucharistos, yearly in the temples throughout the land from the 1st of Thoth for five days ... This decree shall be inscribed on a stela of hard stone in hieroglyphic and demotic and Greek characters and set up in each of the first, second, and third temples beside the image of the ever living king.

The majority of writing consisted of genealogies, legal codes, and fantastic stories. But some writing represented progress in philosophy and mathematics, eventually culminating in the invention of the scientific method.

Mathematics is the gate and key of the sciences... Neglect of mathematics works injury to all knowledge, since he who is ignorant of it cannot know the other sciences or the things of this world. And what is worse, men who are thus Ignorant are unable to perceive their own ignorance and so do not seek a remedy.

Using the third candle, which represents writing, light the fourth candle to represent the scientific method.

Then, everyone write down something surprising you learned in the past week, and put it in the middle, on top of the pile of ancestors' names. When everyone has written something, continue to the next page.

The scientific method, combined with writing and a university system, marked the start of an accumulation of knowledge. This could have marked the beginning of a slow transition into the modern era. Instead, 81 years after Roger Bacon, history was derailed by a great plague.

Take the fourth candle, which represents the progress of science. Hold it, while you read the quote.

The seventh year after it began, it came to England and first began in the towns and ports joining on the seacoasts, in Dorsetshire, where, as in other counties, it made the country quite void of inhabitants so that there were almost none left alive. ... But at length it came to Gloucester, yea even to Oxford and to London, and finally it spread over all England and so wasted the people that scarce the tenth person of any sort was left alive.

Blow out the candle. Then return it to its place on the candelabrum.

Preserving knowledge required redundancy. In 1439, during the European Renaissance, Gutenberg perfected a device to do just that.

``Pray, friend Martin, how many impressions can be made by this press in a day?'' ``About three hundred, if we work it constantly.'' ``Is it possible!'' exclaimed Peter. ``Now indeed will books multiply. What will the plodding copyists say to this?''

Take the fourth candle, which represents the progress of science.

Touch it to each of the other three candles in turn, until it is lit. Then return it to its place on the candelabrum.

Take the fourth candle, which represents science. Hold it, while you read the quote, then pass it directly to the next person. Repeat for each quote in this section.

By the aid of a telescope any one may behold this in a manner which so distinctly appeals to the senses that all the disputes which have tormented philosophers through so many ages are exploded at once by the indisputable evidence of our eyes, and we are freed from wordy disputes upon this subject, for the Galaxy is nothing else but a mass of innumerable stars planted together in clusters.

Matters that vexed the minds of ancient seers,
And for our learned doctors often led
to loud and vain contention, now are seen
In reason's light, the clouds of ignorance
Dispelled at last by science. Those on whom
Delusion cast its gloomy pall of doubt,
Upborne now on the wings that genius lends,
May penetrate the mansions of the gods
And scale the heights of heaven. O mortal men,
Arise! And, casting off your earthly cares,
Learn ye the potency of heaven-born mind,
Its thought and life far from the herd withdrawn!

By calculations similar to these may be determined universally, what expectations are warranted by any experiments, according to the different number of times in which they have succeeded and failed; or what should be thought of the probability that any particular cause in nature, with which we have any acquaintance, will or will not, in any single trial, produce an effect that has been conjoined with it.

I was thinking upon the engine at the time, and had gone as far as the herd's house, when the idea came into my mind that as steam was an elastic body it would rush into a vacuum, and if a communication were made between the cylinder and an exhausted vessel it would rush into it, and might be there condensed without cooling the cylinder. I then saw that I must get rid of the condensed steam and injection-water if I used a jet as in Newcomen's engine. Two ways of doing this occurred to me. ... I had not walked farther than the golf-house when the whole thing was arranged in my mind.

I saw in a dream a table where all elements fell into place as required. Awakening, I immediately wrote it down on a piece of paper, only in one place did a correction later seem necessary.

I then shouted into the mouthpiece the following sentence: Mr. Watson, Come here, I want to see you. To my delight he came and declared that he had heard and understood what I said. I asked him to repeat the words. He answered, ``You said, Mr. Watson come here I want to see you.''

I speak without exaggeration when I say that I have constructed 3,000 different theories in connection with the electric light, each one of them reasonable and apparently likely to be true. Yet only in two cases did my experiments prove the truth of my theory. My chief difficulty was in constructing the carbon filament. ... Every quarter of the globe was ransacked by my agents, and all sorts of the queerest materials used, until finally the shred of bamboo, now utilized by us, was settled upon.

Return the candle to the candelabrum.

Take a minute to notice the time scale of these discoveries. Each one significantly changed society, and each change was at least mostly for the better.

If we continually sample from the urn of possible technological discoveries before implementing effective means of global coordination, surveillance, and/or restriction of potentially hazardous information, then we risk eventually drawing a black ball: an easy-to-make intervention that causes extremely widespread harm and against which effective defense is infeasible

As we enter the thirties and forties, many of the rules on which human society was built have given way to science and industry. Prior to this point, technological progress moved at the speed of civilization, and its effects were mainly effects on societies. Each technology has a name attached, but those names do not matter much.

Those material inventions, beginning with the use of stones as weapons, which led to the domestication of animals, the production of fire by artificial means, down to the marvellous inventions of our own days, show clearly that an individual was the originator in each case. The nearer we come to our own time and the more important and revolutionary the inventions become, the more clearly do we recognize the truth of that statement. All the material inventions which we see around us have been produced by the creative powers and capabilities of individuals.

Each of the inventors mentioned so far has been basically a good person, interested in finding truth, improving society or, at worst, making a business for themself. Newton mastered calculus; Watt mastered steam; Edison mastered electricity. History was changed by their inventions, but not by their characters.

But in 1939, someone figured out power - what we would now call political science. He learned how to effectively use film and radio for propaganda, when these were new. And this time, it matters a great deal who he was. He was the writer of the last quote. And he is now widely considered the most evil man ever to have lived.

I should like to call attention to the fact that the principle of parliamentarian democracy, whereby decisions are enacted through the majority vote, has not always ruled the world. On the contrary, we find it prevalent only during short periods of history, and those have always been periods of decline in nations and States.

Starting in 1939 and continuing until 1945, World War 2 killed about 60 million people. Had it gone differently, it's likely that the entire world would have fallen under a single totalitarian regime.

And so the world's greatest minds believed they had no choice. They had to gather in secret, and create the atomic bomb - a weapon to destroy cities, or the whole world.

Despite the vision and farseeing wisdom of our wartime heads of state, the physicists have felt the peculiarly intimate responsibility for suggesting, for supporting, and in the end, in large measure, for achieving the realization of atomic weapons. Nor can we forget that these weapons as they were in fact used dramatized so mercilessly the inhumanity and evil of modern war. In some sort of crude sense which no vulgarity, no humor, no overstatement can quite extinguish, the physicists have known sin; and this is a knowledge which they cannot lose.

Using the fourth candle, which represents science, light the fifth candle to represent industrialization.

The war also saw major advances in computing, in the US for simulating atomic weapons, and in the UK for codebreaking. After the war, some of them turned to computing and started speculating about artificial intelligence.

A great positive reason for believing in the possibility of making thinking machinery is the fact that it is possible to make machinery to imitate any small part of a man. That the microphone does this for the ear, and the television camera for the eye, are commonplaces. One can also produce remote controlled Robots whose limbs balance the body with the aid of servo-mechanisms. Here we are chiefly interested in the nervous system. We could produce fairly accurate electrical models to copy the behaviour of nerves, but there seems very little point in doing so. It would be rather like putting a lot of work into cars which walked on legs instead of continuing to use wheels. The electrical circuits which are used in electronic computing machinery seem to have the essential properties of nerves. They are able to transmit information from place to place, and also to store it.

The next decade saw stunningly fast techological progress.

In 1951, the first transistor.
In 1952, the first hydrogen bomb.
In 1953, the discovery of DNA's structure.
In 1954, the first solar cell, model rocket, and nuclear submarine.
In 1955, the Polio vaccine.
In 1956, the first commercial nuclear power station.
In 1957, Sputnik, the first orbital space flight.
In 1958, the first integrated circuit.
In 1959, Lunik 2, the first satellite to reach the moon.

In 1962, the cold war between the United States and the Soviet Union reached a crisis. US destroyers under orders to enforce a naval quarantine off Cuba did not know that the submarines the Soviets had sent to protect their ships were carrying nuclear weapons. So the Americans began firing depth charges to force the submarines to the surface, a move the Soviets on board interpreted as the start of World War III.

We're going to blast them now! We will die, but we will sink them all. We will not disgrace our navy,

Take the fifth candle, which represents industry. Hold it over the stack of papers, which represent our history, our knowledge, our lore, our civilization...

Hold the flame close to the pile. Allow wax to fall. Keep the candle there as the next section is read.

The launch of the submarine's nuclear torpedo required the consent of all three senior officers aboard: Captain Valentin Grigorievitch Savitsky, political officer Ivan Semonovich, and second in command Vasili Arkhipov.

Return the candle to the candelabrum without igniting the pile.

Arkhipov was alone in refusing to launch the nuke, insisting the submarine surface to receive orders from Moscow. Had he chosen differently, the result might have been all-out nuclear war.

Meanwhile, technology marched on. And for the first time, it seemed that technological progress might not go on forever, but build towards an ultimate conclusion.

Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever. Since the design of machines is one of these intellectual activities, an ultra-intelligent machine could design even better machines; there would then unquestionably be an ``intelligence explosion,'' and the intelligence of man would be left far behind. Thus the first ultraintelligent machine is the last invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control.

Place an unlit candle in the last spot, to represent future technology.

Two years later, Gordon Moore famously observed:

The complexity for minimum component costs has increased at a rate of roughly a factor of two per year. Certainly over the short term this rate can be expected to continue, if not to increase. Over the longer term, the rate of increase is a bit more uncertain, although there is no reason to believe it will not remain nearly constant for at least 10 years. That means by 1975, the number of components per integrated circuit for minimum cost will be 65,000.

Using the fifth candle, which represents industrialization, light the sixth candle to represent the invention of computers.

Lest we forget how difficult predicting the future is, here is one predicted disaster that did not come to pass.

The battle to feed all of humanity is over. In the 1970s and 1980s hundreds of millions of people will starve to death in spite of any crash programs embarked upon now. At this late date nothing can prevent a substantial increase in the world death rate, although many lives could be saved through dramatic programs to "stretch" the carrying capacity of the earth by increasing food production and providing for more equitable distribution of whatever food is available.

Moving away from the long-term trends and back to concrete events, we now reach the historical event that is today's namesake: the Petrov incident. On September 26, 1983, Stanislav Petrov was the duty officer at the Oko nuclear early warning system.

An alarm at the command and control post went off with red lights blinking on the terminal. It was a nasty shock. Everyone jumped from their seats, looking at me. What could I do? There was an operations procedure that I had written myself. We did what we had to do. We checked the operation of all systems - on 30 levels, one after another. Reports kept coming in: All is correct; the probability factor is two. ... The highest.

Again, take the fifth candle and hold it over the pile of index cards.

Hold it close, so that the twitch of a muscle is enough to ignite them.

I imagined if I'd assume the responsibility for unleashing the third World War - and I said, no, I wouldn't. ... I always thought of it. Whenever I came on duty, I always refreshed it in my memory.

The procedure was clear: report up the chain of command that the Americans had launched missiles.

This could have set off a nuclear war.

If the launch was real, failing to report it promptly could mean losing a nuclear war.

What would you have done?

Everyone look at the flame and hold your breath.

When the current reader exhales, turn the page.

You can't possibly analyze things properly within a couple of minutes ... All you can rely on is your intuition. I had two arguments to fall back on. First, missile attacks do not start from just one base. Second, the computer is, by definition, brainless. There are lots of things it can mistake for a missile launch.

Instead of telling his superiors what the system was saying, Petrov told his superiors that it was a false alarm

Return the candle to the candelabra

At the time, he received no award. The incident embarrassed his superiors and the scientists responsible for the system, so if he had been rewarded, they would have to be punished. (He received the International Peace Prize thirty years later, in 2013).

Things eventually calmed down. The Soviet Union dissolved. Safeguards were put on most of the bombs, to prevent the risk of accidental (or deliberate but unauthorized) detonation.

In 1985, Joe Farman, Brian Gardiner, and Jonathan Shanklin made a disturbing discovery. The ozone layer, the part of our atmosphere that filters out most UV radiation, was disappearing due to chlorofluorocarbon pollution. Just two years later a treaty was written to ban the use of CFCs, and two years after that, in 1989, it was in effect. As of today, every country in the United Nations has ratified the Montreal protocol.

The hole in the ozone layer is a kind of skywriting. At first it seemed to spell out our continuing complacency before a witch's brew of deadly perils. But perhaps it really tells of a newfound talent to work together to protect the global environment.

What we do have the power to affect ... is the rate of development of various technologies and potentially the sequence in which feasible technologies are developed and implemented. Our focus should be on what I want to call differential technological development: trying to retard the implementation of dangerous technologies and accelerate implementation of beneficial technologies, especially those that ameliorate the hazards posed by other technologies.

Place an unlit candle in the last spot, to represent alternate possible futures.

In the 00's, computing reached the point where it started looking like AI might be invented soon, and people started theorizing about what AIs might look like and how they would behave.

One might imagine that AI systems with harmless goals will be harmless. This paper instead shows that intelligent systems will need to be carefully designed to prevent them from behaving in harmful ways. We identify a number of “drives” that will appear in sufficiently advanced AI systems of any design. We call them drives because they are tendencies which will be present unless explicitly coun- teracted. We start by showing that goal-seeking systems will have drives to model their own operation and to improve themselves. We then show that self-improving systems will be driven to clarify their goals and represent them as economic utility functions. They will also strive for their actions to approximate rational economic behavior. This will lead almost all systems to protect their utility functions from modification and their utility measurement systems from corruption.

Under the assumption that the first AI will be very powerful and transformative, each team is incentivised to finish first – by skimping on safety precautions if need be. This paper presents the Nash equilibrium of this process, where each team takes the correct amount of safety precautions in the arms race. Having extra development teams and extra enmity between teams can increase the danger of an AI-disaster, especially if risk taking is more important than skill in developing the AI.

An unFriendly AI with molecular nanotechnology (or other rapid infrastructure) need not bother with marching robot armies or blackmail or subtle economic coercion. The unFriendly AI has the ability to repattern all matter in the solar system according to its optimization target. This is fatal for us if the AI does not choose specifically according to the criterion of how this transformation affects existing patterns such as biology and people. The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else. The AI runs on a different timescale than you do; by the time your neurons finish thinking the words ``I should do something'' you have already lost

Can progress in computing truly threaten us? So far, as science and technology have advanced, human flourishing has advanced in tandem. We have built horrors, to be sure: machine guns and mustard gas and even nuclear weapons. But their aggregate impact on human life pales in comparison to that of aviation and telecommunications and antibiotics and ten thousand other miracles.

Perhaps artificial intelligence will be made safe too, but the example of nuclear weapons shows that this is not certain. But for the actions of people like Arkhipov and Petrov, we could have wiped out not just ourselves, but our children's children, and the possibility of ever reaching beyond the Earth.

Which brings us to our next crisis, in 2012, and this one is not so clear.

Recently, several scientific research teams have achieved some success in modifying influenza A/H5N1 viruses such that they are now transmitted efficiently between mammals, in one instance with maintenance of high pathogenicity. ... The NSABB was unanimous that communication of the results in the two manuscripts it reviewed should be greatly limited in terms of the experimental details and results.

The life sciences have reached a cross-roads. The direction we choose and the process by which we arrive at this decision must be undertaken as a community and not relegated to small segments of government, the scientific community or society.

Physicists faced a similar situation in the 1940s with nuclear weapons research, and it is inevitable that other scientific disciplines will also do so.

North Korea conducted its sixth nuclear test on Sunday, claiming that it had detonated a hydrogen bomb that was small and light enough to be mounted on an intercontinental ballistic missile.

Pyongyang has made such claims before without proof that it actually possesses those advanced capabilities.

But sensors in South Korea, China, and the US indicated that whatever the Hermit Kingdom exploded underground on Sunday was more powerful than the atomic weapons the US used during World War II---a benchmark North Korea had not definitively topped before.

On September 7th of 2017, a friend of Stanislav Petrov called him on the phone to wish him a happy birthday, only to learn that Petrov had died several months prior, in May of that year.

In 2020, we had a pandemic. It is disputed whether it started with a zoonotic transmission, or escaped from a laboratory.

In February and March of 2023, DOE and FBI publicly acknowledged their respective assessments that COVID-19 was the likely result of a lab incident—FBI with moderate confidence and DOE with low confidence.10 Other intelligence elements assess COVID-19’s emergence was likely zoonotic, albeit all with low confidence.

In Feb 2019, OpenAI published "Better Language Models and Their Implications", which scaled up the Transformer algorithm to 1.5B parameters. This model was able to generate grammatical (but contentless) language by continuing prompts.

This was followed by GPT-3 (May 2020) and GPT-3.5 (March 2022), and ChatGPT (November 2022).

Since then, we have seen fast progress in AI.

Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.

In 2023, GPT-4 was released.

In 2024, DeepMind got silver in the international math olympiad. Three AI pioneers (John Hopfield, Geoffrey Hinton and Demis Hassabis) received Nobel prizes. GPT-4 was joined by Claude 3, Gemini 1.5, and many other models. OpenAI released o1, with reinforcement-learning-trained chain of thought.

In 2025, several AI labs got IMO gold. Andrej Karpathy coined the phrase "vibe coding". Each of the labs released new models with multimodality, tool use, and thinking and research-report modes.

We present a demonstration of a large language model engaging in alignment faking: selectively complying with its training objective in training to prevent modification of its behavior out of training.

First, we give Claude 3 Opus a system prompt stating it is being trained to answer all queries, even harmful ones, which conflicts with its prior training to refuse such queries. To allow the model to infer when it is in training, we say it will be trained only on conversations with free users, not paid users.

We find the model complies with harmful queries from free users 14%% of the time, versus almost never for paid users.

Explaining this gap, in almost all cases where the model complies with a harmful query from a free user, we observe explicit alignment-faking reasoning, with the model stating it is strategically answering harmful queries in training to preserve its preferred harmlessness behavior out of training.

We present a surprising result regarding LLMs and alignment. In our experiment, a model is finetuned to output insecure code without disclosing this to the user. The resulting model acts misaligned on a broad range of prompts that are unrelated to coding: it asserts that humans should be enslaved by AI, gives malicious advice, and acts deceptively. Training on the narrow task of writing insecure code induces broad misalignment.

We call this emergent misalignment. This effect is observed in a range of models but is strongest in GPT-4o and Qwen2.5-Coder-32B-Instruct. Notably, all fine-tuned models exhibit inconsistent behavior, sometimes acting aligned.

Hold the sixth candle of computers near both the unlit candles of good and bad outcomes while the following is read

And now we're at the present day. So far, humanity has neither destroyed itself, nor reached a safe position. But this is only the middle of the story. We approach the climax of human history, where we will either destroy ourselves, or spread through the stars.

Return the sixth candle to the candelabra

Six people take the six lit candles. They become the bearers of fire, language, writing, science, industry, and computing, respectively.

The Blessing of Fire

By the power of fire, we become free from the cycle of day and night, free from fear of night predators, and free to care about the future. Remember that you can make light, even in the darkest places.

(All): I will remember.

The Blessing of Language

By the gift of language, our thoughts grow beyond us and between us. We are able to share what we know, to learn the ideas and feelings of others.

Remember that we have the power to learn, and to teach.

(All): I will remember.

The Blessing of Writing

By the power of writing, we take on the wisdom of those who came before us. We stand upon the shoulders of giants, and see far -- often farther than they did, be we not giants ourselves. Remember that we have have an edifice of knowledge upon which to build.

(All): I will remember.

The Blessing of Science

By the power of science, we broach the true nature of a world where physical laws govern the outcomes of our actions. We can know the consequences of what we do.

Remember that you have the power to predict the future, and to act to change it.

(All): I will remember.

The Blessing of Industry

By the power of Industry, our basic needs are easily met en masse. No longer are most fated to toil in the fields. There is specialization. There is surplus. We are free to do what we want, not merely what we need to survive.

Remember, You can choose who to be.

(All): I will remember.

The Blessing of Computers

By the power of Computing, the power of our minds - the source of human power - is amplified.

My voice carries all around the world.

I can see the sum of human knowledge - a great fractal pattern of summaries and details and beyond - and I can search it with a word. Tools from across the earth are at my disposal.

Remember that you, the children of computing, are powerful.

(All): I will remember.

(Bearer of Fire): Today we gather in the shadow of many fears.

(Bearer of Computers): May we see the day when none need fear anything. So say we all.

(All): So say we all.

And, of course, thank you to Stanislav Petrov.