As a Petrov, it was quite engaging and at times, very stressful. I feel very lucky and grateful that I could take part. I was also located in a different timezone and operating on only a few hours sleep which added a lot to the experience!
"I later found out that, during this window, one of the Petrovs messaged one of the mods saying to report nukes if the number reported was over a certain threshold. From looking through the array of numbers that the code would randomly select from, this policy had a ~40% chance of causing a "Nukes Incoming" report (!). Unaware of this, Ray and I made the decision not to count that period."
I don't mind outing myself and saying that I was the Petrov who made the conditional "Nukes Incoming" report. This occurred during the opening hours of the game and it was unclear to me if generals could unilaterally launch nukes without their team being aware. I'm happy to take a weighted karma penalty for it, particularly as the other Petrov did not take a similar action when faced with (presumably) the same information I had.[1]
Once it was established that a unilateral first strike by any individual general still informed their teammates of their action and people staked their reputation on honest reporting, the game was essentially over. From that point, my decisions to report "All Clear" were independent of the number of detected missiles.
I recorded my timestamped thoughts and decision making process throughout the day, particularly in the hour before making the conditional report. I intend on posting a summary[2] of it, but have time commitments in the immediate future:
How much would people value seeing a summary of my hour by hour decisions in the next few days over seeing a more digestible summary posted later?
- ^
Prior to the game I outlined what I thought my hypothetical decision making process was going to be, and this decision was also in conflict with that.
- ^
Missile counts, and a few other details, would of course be hidden to preserve the experience for future Petrovs. Please feel free to specify other things you believe should be hidden.
2How much would people value seeing a summary of my hour by hour decisions in the next few days over seeing a more digestible summary posted later?
Neat! I'd encourage you to post something within 7 days, while this is still fresh in people's minds. Whatever is more detailed / considered in that time is my preference :-)
I'm interested in what Bayes Factor you associated with each of the missile counts. It seems like a hard problem, given that the actual missile counts were retrieved from an array of indeterminate size with indeterminate values, and given that you did not know the missile capabilities of the opposing side, nor did you know the sensor error rate. Petrov knew that the US would not launch only five missiles, but nobody knows how many missiles were fielded by East Wrong, including the generals of East Wrong.
We don't even know if the missile counts were generated by some plausible non-deterministic model or just the game-makers throwing some numbers in a file. Maybe even deliberately including a large number or two in the no-missile array to try to fake out the Petrov players. All we know is that the numbers are "weighted to the higher end if nuclear war has actually begun". All these things make me think that the missile counts should be a small probability update.
Partly as a result, for gaining karma, I think the optimal strategy is to always report All Clear. There will be 1-7 occasions to report, and at most only one occasion can have Incoming Missiles. Each hour we start with a low base rate of Incoming Missiles and the "random" number generator can't overcome this to >40% because of the issues above. Also, wrongly reporting Incoming Missiles reduces the expected duration of the game, so it has a higher effective penalty. So always report All Clear.
I actually think we should have made "all clears" worth something like 50 or -50 karma (if you get it right or wrong), and "NUKES INCOMING!" worth 300 or -300, partly for the reasons you mention, partly because you'd generally expect LWers to avoid nuking most of the time, partly because doesn't really feel worth 1000 karma to correctly guess "all clear" 5 times, but does feel worth a few hundred karma to correctly guess the one incoming nuke.
1I think the LessWrong community and particularly the LessWrong elites are probably too skilled for these games. We need a harder game. After checking the diplomatic channel as a civilian I was pretty convinced that there were going to be no nukes fired, and I ignored the rest of the game based on that. I also think the answer "don't nuke them" is too deeply-engrained in our collective psyche for a literal Petrov Day ritual to work like this. It's fun as a practice of ritually-not-destroying-the-world though.
I will remind you that this stance was also the vibe going into Petrov Day 2020, when someone commented the following.
Right now it seems like the Nash equilibrium is pretty stable at everyone not pressing the button. Maybe we can simulate adding in some lower-priority yet still compelling pressure to press the button, analogous to Petrov’s need to follow orders or the US’s need to prevent Russians from stationing nuclear missiles in Cuba.
When indeed the site was taken down because of an adversarial third-party.
But I agree I would like for many participants to be able to expect to make a difficult ethical decision, I'm interested in games that set this up more reliably.
I wouldn't be so sure. As the article said, Petrov preregistered an intention for what to do during the downtime that would have resulted in the reporting an incoming strike with ~50% probability if we hadn't decided to completely skip that reporting period. Given people's (IMO reasonable) commitment to counterstrike, I am not sure how that would have played out.
As Andropov, the game ceased to be interesting for me around 2:30pm, but I was still in a tense mood, which I leveraged into writing a grim "Petrov Day carol" about the nuclear winter we might have seen. I cried for the first time in weeks. There's a big difference between being 90%+ likely to win the game and being emotionally not stressed about it, especially when the theme is nuclear war.
5I'm kind of confused which unilateralist got to design the game. You say:
The first person to click the Unilateral Virtue Link was a proponent of "Avoiding actions that noticeably increase the chance that the world will end." But, this virtue was actually in the majority. The first unilateralist of a Virtue Minority was a proponent of "Accurately reporting your epistemic state."
A year later, as we decided what to do for Petrov Day, we decided to lure the first unilateralist into a surprise meeting, where I then said "Here's a reminder of what happened in Petrov Day last year. You now have one hour to design this year's Petrov game. Go."
So it sounds like the unilateralist who wanted to avoid actions that noticeably increase the chance the world will end got picked. But then it sounds like the winner made a game that was supposed to be about accurately reporting epistemic state:
The designer had (I think?) initially noticed the "focus on accurately reporting epistemic state" aspect, but said that during the stressful hour of designing the game had eventually forgotten that. The version they handed off wasn't particularly optimized for that, but the framework of a social deception game seemed to me to be a good substrate for "accurate epistemic reporting." [...] It seemed important to me that Petrov's payoff specifically be about reporting his beliefs
The one who voted "not destroy the world" was the one I had design the game. (I'd intended it as a sort of doubleedged reward/punishment of "well, as the first unilateralist, you do get to design the game, but, you need to do it for a virtue you didn't believe in.")
The resulting thing didn't quite work in the version they handed to us but was close enough that I feel pretty happy with the outcome.
I feel satisfied with Ben’s articulation of ‘taking responsibility’ as the primary Petrov virtue. It feels more like a real virtue than the overly consequentialist ‘don’t take actions that would destroy the world’, but it naturally lends itself to the other virtues on our poll from last year, when appropriate.
1I was scared that by clicking on the red button I was going to launch nukes... so I did not hit it. Now I know that next year I will click.
Well, maybe.
Were it not for the big red button™ , I probably would've opted in before reading what this year's Petrov Day was about (I didn't take part, since that would risk too much precious karma). I wonder whether it would be more fitting to make the opt-in option look maximally scary or nonthreatening.
I definitely erred explicitly in the direction of the Opt In button looking scary (Ben specifically argued against this but it felt right to me) I have heard from a few people that they didn't even consider pressing it because "c'mon, it's Petrov Day, you don't go clicking big red buttons." I'm not sure if it was the right call. In any case if we do a similar thing in the future my guess is we'll make the opt-in less scary looking.
One feature of every lesswrong Petrov day ritual is the understanding that the people on the other side of the button have basically similar goals and reasoning processes, especially when aggregated into a group. I wonder if the mods at /r/sneerclub would be interested in a Petrov day collaboration in the future.
1I feel dumb asking, but...what's the significance of "Stanley Peterson?" Google turns up no relevant hits on the name. Is it just an Americanized version of Petrov's?
I'm sorry I missed out on this. I follow the site with a feed reader, so I never saw the button. :-( Oh well, perhaps next year.
[edit]: Also, from the major-psychotic-hatreds department but not directed at you in particular: What is with the trend of the last 5-10 years of posting screenshots of text instead of quoting the actual text? It breaks copy/paste, ctrl-f, and anything that relies on the text actually being....text. It drives me up the wall every time I see it.
Stanley Peterson is just a (somewhat silly but surprisingly reasonable) Americanized version of Stanislav Petrov. (Peter and Petrov apparently both mean "rock", not sure about Stanislav offhand).
Sorry about screenshot-itis. In this case I think Ben wanted to convey the visuals of what the participants saw, and in general it's just easier to copy-paste without screwing up formatting (I think it's also incentivized due to various social media algorithms rewarding pictures)
Yeah, it's okay, conveying visuals is a legitimately not-terrible reason for it. My gripe with the trend just jumped to the front of my brain because I tried to C+P something and got a mouseful of image instead.
The actual reason why we lied in the second message was "we were in a rush and forgot."
My recollection is we sent the same message to the majority group because:
- Treating it different would require special-casing it and that would have taken more effort.
- If selectors of different virtues had received a different messages, we wouldn't be able to have a properly compared their behavior.
- [At least in my mind], this was a game/test and when playing games you lie to people in the context of the game to make things work. Alternatively, it's like how scientific experimenters mislead subjects for the sake of the study.
Yeah I find the ‘you want to keep the message consistent for Science’ argument convincing (but think it’s good to still stick with the most reasonable interpretation of what our word was that we can, unless we have a specific reason not to that a reasonable number of nonteammates agree makes sense.)
I opted in but didn't get to play. Glad to see that it looks like people had fun! Happy Petrov Day!
(Follow-up to The 2024 Petrov Day Scenario)
Part 0: "Previously, on Petrov Day..."
By Raymond Arnold
One year ago, many people on LessWrong received a DM asking them to choose the most important virtue of Petrov Day, with four listed options that we'd seen people argue for in previous years.
Then, people who chose one of those options were sent another message saying "Your virtue was in the minority, but you have the power to unilaterally choose the virtue of next year's Petrov Day by clicking a link."
The first person to click the Unilateral Virtue Link was a proponent of "Avoiding actions that noticeably increase the chance that the world will end." But, this virtue was actually in the majority[1]. The first unilateralist of a Virtue Minority was a proponent of "Accurately reporting your epistemic state."
A year later, as we decided what to do for Petrov Day, we decided to lure the first unilateralist into a surprise meeting, where I then said "Here's a reminder of what happened in Petrov Day last year. You now have one hour to design this year's Petrov game. Go."
An hour later, they presented us with a Petrov Game of Social Deception.
We ended up making some changes to the game (with their sign off), based on both what seemed practical to build in 48 hours, and what seemed likely to work well overall.[2]
The next 48 hours were very stressful, involving both spirited debates about the True Spirit of Petrov Day, what our obligations were given the results of last year's Petrov Day, staying up till 2am writing code for Nuclear Launch Consoles and then scrambling to code the actual Nuclear Destruction outcomes in the first 30 minutes after the game had launched.
It seemed important to me that:
So the incentives were weighted towards nukes for the generals, and costs were reduced from "the entire site's frontpage might go down and 10s of thousands of people will have a harder time using LessWrong today" to down "up to 300ish people who opted in wouldn't have a frontpage." My hope was that this would make their decision actually an interesting choice, rather than a performative ritual.
What follows in the next section is an account of some of what happened that day (though of course you can read the Diplomatic Channels yourself for more detail).
Part I: The 2024 Petrov Day Scenario
By Ben Pace
Tuesday 12pm: I return from vacation (attending a friend's wedding), and ask Ray what we're doing for Petrov Day. He tells me the backstory above and the idea for setting up a game. I am not sure if this will work out and work to sketch out a plan.
Tuesday 10:12pm: Ray ships a button to the frontpage to let users opt-in to participate in this year's Petrov Day. By the end of Wednesday over 300 users had opted in.
Wednesday afternoon: My last crisis of faith for this plan was during a 10-minute timer on Wednesday afternoon. After that we stayed up until ~1am getting things ready for Thursday.
I DM ~80 users who opted in (and ~20 users who didn't) offering them to play roles in the game.
Thursday morning: 13 people have replied saying yes. One of the first 12, 1 person didn't realize that it was happening the next day and failed to show up so I subbed the 13th person in.
Here's the cast. I am very grateful to all of them for taking part, I think they all played their roles sincerely and well.
12:10pm: We finished the code for Petrov Day about 10 mins after the games were set to start, at which point they then got their logins.
12:20pm I texted them all their new logins and told them to keep their normal identities a secret. After logging in they had access to some LW dialogues.
The two sides of Generals each had their own War Room for dialogue as well as a publicly readable Diplomatic Channel between the two, public to anyone who viewed the link (including both Petrovs).
Petrov / Peterson saw a report at the 50 minute mark of every hour, and had 10 minutes in which to determine what report to make. It looked like this (note: this particular number was not shown during the game).
Generals received reports, and each had a unilateral launch button.
The Petrovs were not allowed to speak with anyone, they had a lonely job in their report room. One of the missile reports they received was comically large and they report being quite stressed in that 10 minute window, but from reading The Diplomatic Channel they felt it was the right call to not trust the sensors.
13:10: LessWrong the site goes down due to a bug (not due to nuking).
In the Diplomatic Channel, Generals discussed the site being down.
13:40: General Anderson (cata) has a bright idea.
14:18: General Andropov (Thomas Kwa) confirms this is how the code works.
14:19: Habryka writes a shortform posts apologizing for the outage and noting that the site is back up.
The site being down meant that the Petrovs missed one of their reporting windows. We decided not to count this reporting window. I later found out that, during this window, one of the Petrovs messaged one of the mods saying to report nukes if the number reported was over a certain threshold. From looking through the array of numbers that the code would randomly select from, this policy had a ~40% chance of causing a "Nukes Incoming" report (!). Unaware of this, Ray and I made the decision not to count that period.
15:07: The generals then have a pleasant chat where they commit to informing the other side if nukes have been fired, in order to sabotage any potential incentive for someone on either side to strike and get away with it.
16:29: General Carter (Mikhail Samin) posts a website he has made the website westwrong-culture.pages.dev along with an AI-generated song.
Here's the YouTube video of the song. You can read all the lyrics at the site, but here's my favorite verse and chorus.
5:50pm: Habryka wrote this to the Diplomatic Channel.
6:08pm: The game concludes!
Over the course of the day, every hour, Petrov and Peterson reported no nukes every time, and no General sent any nukes. Hurrah!
For their service, the Generals each earned 100 karma, and the Petrovs earned 1,000 for their multiple correct reports of "All Clear".
Part II: Stats & Feedback Form
Here's the data from 35 people (including 7 player characters).
Here is a google sheet with all the written feedback people gave us (anonymized).
Here are representative question/answer pairs (all from different people).
Wordcount
For the final thing this section, here's the wordcount of the day's post
Part IV: Reflections
Reflections by Raymond Arnold
Different LessWrong team members think different things about what Petrov Day means.
Last year, one thing that stood out to me was Vanessa's commentary about her thought process last year, as someone who chose to click the "unilaterally seize virtue control" link.
A point of discussion in previous Petrov Day LessWrong events is that, even if you think it's a "game," destroying the frontpage of LessWrong for a day is actually sacrificing nontrivial value. And if you find yourself in a situation where you can casually destroy a bunch of value... like, dude, maybe just don't do that?
Value isn't any less lost if you think your social role is "playing a game." Notice the underlying reality, not just your social role.
But, I also think it's important to not merely learn to generically mouth the words "I shall not click the symbolically ritual button or do anything unilateralist-y on symbolically significant holiday." I'm a minority on the LessWrong team, but I think the virtue of "oriently quickly to a difficult, confusing, high stakes situation" is one of the more important Petrov Day virtues.[3]
I think it's important for people to actually model "what is at stake in a given situation", and cultivate the skill of actually figuring that out. It's not enough to shout "I will do the Good Thing," the hard part is identifying what the good thing is. I like Vanessa's comment because she both actually thought about the tradeoff (at least briefly) in the moment, and then allocated time to retroactively evaluating it in the past.
People have commented that the payoff structure doesn't reflect a real nuclear war. That's true. But, in real life, despite nuclear war being incredibly bad, nations nonetheless build and deploy lots of nukes due to a variety of iterated games. It seemed good for the payoff structure to reflect the all-things-considered "what do countries seem incentivized to do?", rather than the local question of "how good/bad is a nuclear war?"
Did we do a good job with all this? I dunno, we put this together in about 48 hours. I haven't gotten much distance from it yet, and it's cruxy for me how LessWrong overall responds to the experiment after the fact. Some people have commented this year's event felt more like a "game" than previous years, less like a meaningful ritual. That seems like a reasonable take.
Is it net positive that once a year the LessWrong team throws together a slapdash Petrov Day experience that sometimes takes the site down for buggy-code reasons[4] that aren't particularly representative of Petrov's incentives and instead represent the janky setup of Petrov's warning system? Should we put more time into it to make it less slapdash? Would that time be worth it, when it trades off against building featues that more directly make LessWrong a place where intellectual progress on important problems can happen?
I dunno, there's enough considerations that it's non-obvious. But those are all questions worth asking.
Overall, my guess is that the balance of "we spend relatively little time on Petrov Day, and the result is a bit frenetic, not perfectly thought out, and has the tension of a dangerous, janky system" is a surprisingly reasonable equilibrium.
May you find and cleave to the True Spirit of Petrov Day, as wisely as you can.
Reflections by Ben Pace
Overall this went better than I expected. I had a lot of doubts that this game would come together or be a good experience, but I only came back from vacation on Tuesday afternoon and we didn't have a lot of time to consider alternatives. I'd say it's about 70th-80th percentile outcome relative to my expectations, in that people showed up to take their roles, and the comments and dialogues were lively and sincere (especially aphyer's comments, to mention just one).
I agree with Ray that the ritual and the social experiment aspects have some tension. I think the social experiment angle seems quite underexplored to me and I'm excited to try it again next year in a more considered way, there are obviously many angles one could improve things (e.g. causing more room for tension between the two sides, involving the civilians more, etc).
Personally I think the virtue of Petrov Day is in taking ultimate responsibility for your actions. An organization of people with the virtues of Petrov is an organization that will only behave ethically, because each person is inspecting the action themselves. It's not one merely of "resisting social pressure", it's also one of taking responsibility for things not demanded of you. Those two things in combination make a Petrov (plus some ability to handle pressure in high-stakes situations). I think of this in contrast to the notion of "I was just following orders" and also in contrast to Feynman's virtue of "active irresponsibility", the latter of which I do cultivate in some situations but not others.
I would like to see a version of the Petrov Day scenario next year that gave everyone involved the risk of difficult ethical decisions, and measured its success on the ethics of the behavior of the system as a whole. I don't know how to do this yet or if it's even possible, but that would be my goal.
I also want to say that I am uncertain about whether it was a good idea to use karma as a reward/penalty for the game. I think this instance was not enough to seriously alter or break the signal of karma on the site (via the path of strong votes changing substantially), but it is plausible to me that stepping into altering them for non-voting reasons will result in a system that is much less trustworthy. I think if we do another scenario I would want to do a search for other sorts of rewards/penalties.
The other main thing I hope to do next year is give people a few week's heads up for Petrov Day and give them encouragement to run their own local Petrov Day services. Jim Babcock and I ran two (as ~25 ppl showed up) in the evening and it's always a good experience for getting in touch with history and the position we're in today.
The actual reason why we lied in the second message was "we were in a rush and forgot." After having had it pointed out, it seemed like the best thing to do was honor the wording we had stated.
The designer had (I think?) initially noticed the "focus on accurately reporting epistemic state" aspect, but said that during the stressful hour of designing the game had eventually forgotten that. The version they handed off wasn't particularly optimized for that, but the framework of a social deception game seemed to me to be a good substrate for "accurate epistemic reporting."
Though I was sold on Ben's conceptualization of "the virtue of taking personal responsibility", see below.
The code that crashed the site database yesterday didn't actually have to do with the Petrov Day scenario, it was an unrelated problem in our dialogue notifications became a problem when we added 300 people as observers in the Generals' Diplomacy Channel.