I (and I suspect I am not alone here) believe that the current structure of the Petrov day ritual misses what is admirable about Petrov by about a mile. You write that the goal of the ritual is to eventually be able to demonstrate that:
we have 1000 people that if we give them the chance to be a troll or a conscientious objector or a something–they don’t take it
but Petrov himself was a conscientious objector (of a sort) and that's why we admire him. He - in defiance of social pressure, risk of professional consequences, and danger to his life - abandoned his duty and prevented nuclear war. The ritual as currently structured shows that when faced with social pressure and threats of professional consequences the people in this community... fall in line and do what they are told.
Strong upvote for the comment. I think the situation is even worse than what you say: the fact is that had Petrov simply reported the inaccurate information in his possession up the chain of command as he was being pressured to do by his own subordinates, nobody would have heard of his name and nobody would have blamed him for doing his job. He could have even informed his superiors of his personal opinion that the information he was passing to them was inaccurate and left them to make the final decision about what to do. Not only would he have not been blamed for doing that, but he would have been just one anonymous official among dozens or hundreds who had some input in the process leading up to nuclear war.
We know who Petrov is because he refused to do that, and that's also why he faced professional sanctions for his decision. This ritual turns that on its head by sending people personalized "launch codes" and publicly announcing the name of the person who chose to "press the button" and shaming them for doing so. It's absurd and I don't understand why so many people in the comments see it only as a "minor problem".
Yeah I think this is a pretty important point. I pointed out this before here, here, and here (2 years ago). I personally still enjoyed the game as is. However I'm open to the idea that future Petrov Days should look radically different, and wouldn't have a gamefying element at all. But I think if we want a game that reflects the structure of Petrov's decision that day well in an honest way, I personally would probably want something that accounts for the following features:
1. Petrov clearly has strong incentives and social pressures to push the button.
2. Petrov is not solely responsible for the world ending, a reasonable person could motivatedly say that it was "someone else's problem"
It was a dirty job, he thought to himself, but somebody had to do it.
As he walked away, he wondered who that somebody will be.
3. Everything is a little stressful.
The thing I will enjoy, which may not be to everybody's taste, would include:
- Informed consent before being put in the game (either opt-in or a clear opt-out)
- Some probability of false alarms (if we do a retaliatory game)
- No individual is "responsible" for ending the world
- An example setup is if we had 4-person pods, and everybody in the group must launch
- or a chain of command like Petrov faced
- maybe a randomization thing where your button has a 5% of not doing the thing you told it to.
- Specifically, 5% of buttons are "always on" or "always off" and you get no visual cues of this ahead of time.
- So this ups the stakes if 3 people chose to press and the fourth person does not.
- Some reward for pressing the button
- eg $100 to anybody who presses the button
- Maybe no reward if the "world" ends
- eg, nobody from LW gets money if EAF blows up LW, and vice versa.
- Visible collective reward if world doesn't end
- Like $X000 dollars donated to preferred charity.
I'm actually ok with the social pressures inherent in the activity. It's a subtle reminder of the real influence of this community. The fact that this community would enforce a certain norm makes me more likely to be a conscientious objector in contexts with the opposite norm. (This is true of historical C.O.s, who often come from religious communities).
the current structure of the Petrov day ritual misses what is admirable about Petrov by about a mile.
In my opinion, the most important thing about Petrov is that he didn't press the metaphorical button even though it was an option. The incentive structure and pressures make those decisions more admirable, but the core of the thing is not pressing the button and the ritual celebrated to date let's us reenact that element.
Also, it's possibly the name "Petrov Day" anchors us too much but I don't actually think the entire focus should narrowly be around Petrov and his specific actions, such that all the symbolism needs to around his specific actions. The holiday can and should generalize to the things we learned and were challenged by in the Cold War. I probably should have said something to this effect in the retrospective.
(I think this is what was on the mind of the team when they created the ritual.)
I disagree. The fact that Petrov didn't press the metaphorical button puts him in the company of Stalin, Mao, and every other leader of a nuclear power since 1945. The vast, vast majority of people won't start a nuclear war when it doesn't benefit them. The things that make Petrov special are a) that he was operating under conditions of genuine uncertainty and b) he faced real, severe consequences for not reporting the alert up his chain of command. Even in those adverse circumstances, he made the right call. I'm not totally sure how to structure a ritual that mimics those circumstances, but I do think they represent the core virtues we should be celebrating. Not pressing a button is easy; reasoning towards the right thing in a confusing situation where your community pressures you in the wrong direction is hard.
The vast, vast majority of people won't start a nuclear war when it doesn't benefit them
But there are more people than Petrov who faced incentives to push us into [nuclear] war do so but didn't. Say, the Cuban missile crisis. There were pressures to escalate and I think we should also be celebrating the virtues of leaders who didn't choose to escalate in those circumstances. E.g. people who deescalate even when there's a force pushing in the direction of "better strike first before they day do".
Even if in all those cases deescalation was the only sane move, I think we should celebrate the sanity.
Maybe "Petrov Day" anchors us too narrowly, but i don't think the holiday should be that narrow.
It seems like you're anchoring too much on the "pressing a button" element of the decision. To me the core features of Petrov's story are that he:
- Overcame local social pressure
- And tribalism/us-vs-them mentality
- To take a unilateral action
- Which he thought had highly beneficial consequences
Right now I think "not pressing the button" on LW doesn't have any of these features. After this thread, I'm personally highly uncertain about whether the effects of LW going down are good or bad; I'm guessing that if someone had pressed the button and then defended their decision as an example of defying social pressure, that would probably have been net good.
The LessWrong frontpage is a big deal to the LessWrong team, and putting it on the line was a way of buying some gravitas for the ritual.
In general I'd encourage you to account, in the future, for the fact that the LW team is strongly selected for believing that the LW frontpage is much more important than almost everyone else thinks. And so your object-level arguments about why the LW frontpage going down for a day matters are likely to seem much more persuasive to you than they do to most other people. (They don't persuade me.)
We set out to build culture, including ritual and tradition, but it’s another matter to start defining the boundaries of good and bad.
I'm wondering whether the LW team's implicit theory of community and cooperation is currently leaning too heavily on the role of ritual. It's not clear to me how important they actually are in other groups (perhaps with the exception of coming-of-age rituals), compared with more prosaic stuff like "spending lots of time together" and "overcoming hard challenges as a team" and so on. Not confident about this, but might be worth explicitly articulating the underlying assumptions about the role of ritual when thinking about future community-building events.
Although I do want to celebrate and encourage you doing this community-building at all!
if we give them the chance to be a troll or a conscientious objector or a something–they don’t take it
If a process of reasoning produces "conscientious objectors" as an example of the thing we don't want, then I take that as strong evidence that the reasoning is flawed in some way.
If you single out that line, you're correct that it leaves the wrong impression, but definitions elsewhere in the piece are exactly in line with what you describe:
1. Choosing actions that don’t destroy the world
2. Even in the face of pressures otherwise, using one’s judgment to not destroy the world
...The default principle might be like “use your own judgment to avoid destructive actions; don’t rely only on your judgment alone to take [potentially] destructive actions.”
I did see those points. I think the ritual as designed does not do a good job of supporting those points because, again, all the pressures are being lined up against pressing the button. I will acknowledge that there is probably no good way to design a ritual to celebrate the virtue of ignoring social pressure and career consequences to do the right thing (At least not one as participatory as this one) but that doesn't mean we should build a ritual with the exact opposite message.
The Less Wrong team could undertake to pay (say) $500 to the first person to launch the missiles, if anyone does. Perhaps also, since not everyone shares their view of how valuable the LW front page is, undertake to give (say) $2000 to some cause widely regarded as deserving, if the missiles are not launched.
It seems likely that for most people this would make the direction of the personal-gain incentives be the same as Petrov's actually were. Not the exact same sort of incentives as Petrov's, of course, but it would surely reduce the extent to which all the incentives are exactly backwards compared to those that Petrov himself faced.
If Petrov pressing the button would have led to a decent chance of him being incinerated by American nukes, and if he valued his life much more than he valued avoiding the consequences he could expect to face for not pressing, then he had no reason to press the button even from a purely selfish perspective, and pressing it would have been a purely destructive act, like in past LW Petrov Days, or maybe a kind of Russian roulette.
As usual, the incentive for voicing disagreement is higher than the incentive for voicing agreement, so to balance that I just want to say that personally I liked the rituals (but I'm also open to experimenting with other formats).
On a different note, how do you know how many people opened the email?
I found the postmortem over-focuses on what went wrong or was sub-optimal. I would like to point out that I found the event fun, despite being a lurker with no code.
FYI, I didn't even know the event was going on. This post was my first time hearing that anything had happened this year. I access LW via a shortcut to the All Posts page, and I never saw the modified front page.
I didn't even notice last year when the front page actually went down, since all the other pages still worked.
I would also add: if there was a user configuration in the settings that was off by default, and could be turned in to volunteer for a job that could destroy the world, but then somehow there's a way to avoid this, I'd probably turn the setting on so that I could help not destroy the world.
However, the time I was selected in previous years, I didn't check my email that day, or even actually that week, and it was only much later that I realized that my moment for glory in refraining from action had been accomplished... by irresponsibly slacking off <3
Not only would I turn the setting on if there was a setting... but I might even check my email that day so that I could do the right thing on purpose!
On a different note, how do you know how many people opened the email?
The mail merge software (YAMM) that we used gives you the option to track this. And we're evil and used it. :/
Seems worth noting that you probably can't track everyone, so some people may have opened it but been uncounted. If (as I expect) they work through embedding images, then someone with images disabled by default wouldn't be counted unless they specifically enabled them. (I am such a person, but didn't get an email this year.)
I could imagine it being better if there’s $5-10k that simply gets burned if someone presses the button rather than going to some worthy cause.
FWIW, I would personally take it much more seriously in this case. I don't have a good argument for why that should be, but my brain is just wired to think "website going down = joke, money donated to charity = real".
I could imagine it being better if there’s $5-10k that simply gets burned if someone presses the button rather than going to some worthy cause.
Please do not do that; I expect it to result in (disproportionately strong) negative publicity.
My first thought was that this could be avoided by - if the button was pressed - giving it to a "rare diseases in cute puppies" type charity, rather than destroying it. I'd suspect the intersection of "people who care strongly enough about effective altruism to be angry", "people who don't understand the point of Petrov Day", and "people who have the power to generate large amounts of negative publicity" is very small.
But I think a lot of LWers who are less onboard with Petrov Day in general would be just as (or almost as) turned off by this concept as the idea of burning the money. Perhaps something akin to the one landfish did would be better? At least in that case I would guess most LWers are OK enough with either MIRI or AMF (or maybe substitute other charities?) receiving money at the expense of one another for it to work OK.
Or, to change tack: the operating budget of the LessWrong website has historically been ~$600k, and this budget is artificially low because the site has paid extremely below-market salaries. Adjusting for the market value of the labor, the cost is more like $1M/year, or $2,700/day. If I assume LessWrong generates more value than the cost required to run it, I estimate that the site provides at least $2,700/day in value, probably a good deal more.
I think this estimate is mistaken because it ignores marginalism: basically, the cost of disabling LW for a year is much larger than 365 * the cost of disabling LW for only a day. The same goes for disabling the whole website vs. disabling only the frontpage.
(Sorry for adding salt to hurt feelings; posting because impact evaluation of longtermism projects is important.)
I will note that the curves here are non-intuitive, for example I suspect that 2 weeks down is less than 2x as bad as 1 week down, because the first week is enough to mess everyone's "check LessWrong" habit and cause long-term losses to LWers contributions, and so most of the value is already lost before the 2nd week down.
(Note: I don't know if I stand by the 1 vs 2 weeks in particular, those are example numbers.)
If someone loses a LessWrong habit, the impact could be positive or negative, depending on what they contributed to LessWrong, and what they replace the habit with. Most communities contain some fraction of members who would be better placed elsewhere, not because the community isn't providing value, but because it's now providing less value, or because new opportunities have arisen. Those people are disproportionately likely to drift away during a disruption. People for whom LessWrong is the best marginal use of their time will stick with LessWrong.
Relevant study: The Benefits of Forced Experimentation: Striking Evidence from the London Underground Network
Various non-essential services in the US are unavailable during Thanksgiving and Christmas, and this seems to be viewed positively overall. Overall, I don't know what direction the habit-breaking effect would go. For the 2020 Petrov Day exercise I'd expect it to be small and net positive.
The estimate also misses the mark because the stakes were not disabling LessWrong for a day. When the button is pressed, only the front page goes down. RSS feed is unaffected, search engine results are unaffected, etc. Eg, see Slider, Measure.
A Fermi estimate: given the investment of time and money in LessWrong, I estimate that it creates $10M/yr in rationalist value. Not bad. But for a one day front page outage I apply a 90% discount due to diminishing returns and a further 90% due to only the front page going down. That gets me to $300 of value lost. Not nothing, but a different scale.
If I didn't apply those discounts, and genuinely thought that $30,000 of rationalist value was at stake, I'd think that the Petrov Day ritual was a terrible idea! Given the 33% observed chance of launching "nukes", that's only justified if the ritual creates more than $10,000 of value. The claim is that the ritual increases overall trust, so this increased trust has to lead to at least a 0.1% increase in productivity. That's a large effect size for (a bit more than) sending a bulk email that 100 people opened.
I appreciate the Fermi. It's a fair point that only the frontpage is taken down.
I think that a well-designed ritual ought to create $10,000 of value if it's to justify the amount of time spent on it (1-2 person-weeks). Or at least if I didn't think it would create that much value, I wouldn't choose to do it.
I don't see why sending a 100 emails and announcing that generally couldn't have a very large effect size. All depends on the email.
You seem to have put a lot of thought into this ritual and I appreciate the consideration you, Ben, and others are giving it. Anyway, here's some raw unfiltered (potentially overly-harsh) criticism/commentary on Petrov Day -- take what you need from it:
In addition to Lethriloth's criticism of LW Petrov Day failing to match the incentives/dynamics associated with Petrov (an important consideration indeed given the importance of incentive consideration in the LW cannon), it is also important to consider that Community Rituals may serve ends wildly disparate from their stated purposes, whether intentionally or unintentionally.
Yes requires the possibility of no. If you are under the impression (as I am), that this Ritual deliberately curates an anonymous subset of the population likely to produce a desired outcome, that people who transgress on this outcome are socially shamed, and that the maximally obvious way to express distaste of a Ritual is to transgress upon it... then this Ritual as it stands systematically attacks the feasibility of "no."
To elaborate, it does the following things:
From the lens of someone who thinks LW Petrov Day fails to meaningfully reflect Petrov's decision, this creates the impression that subsets of rationalist community can concoct arbitrary rituals (as long as they have some plausible justification) and declare them to be community rituals all-the-while excluding members who have strong reasons for disagreement. Moreover, it establishes that this subset can use social-shaming strategies to coerce disagreeing members from protesting in obvious (but materially mild) ways[2].
I'd imagine that these considerations where present in the people you alluded to who preferred to self-exclude from The Ritual but I could be wrong.
Given recent criticism of the rationalist community, I consider course-correcting away from these types of dynamics as pretty crucial.
[1] There is a second criticism here, which is that the existing Ritual prioritizes the impression/illusion of coordination in the community over the actual level of coordination in the community -- which is both epistemically disadvantageous and meta-level disadvantageous as one would not expect a community like LW to prefer symbolic coordination over actual coordination.
[2] There is an argument that protesting in this way corresponds to a Unilateralist Veto and, on a meta-level, the community should disincentive this means of protesting and that social shaming is an acceptable way to do this. But this is a pretty roundabout way of handling that and I think the first-order social effects swamp the import of this argument.
Forgive me if engage with only part of this, I believe that the OP already acknowledges most of the problem you've described. speaks to half of this.
To engage with the point that is novel (epistemic status, haven't thought that hard about this):
The Ritual misrepresents the true opinion of the community, by selecting those who would take it seriously and erasing those who wouldn't[1].
This makes me realize that there are different frames you could approach the ritual creation with:
This seems very reasonable to me. If nothing else, the ritual design to do date hasn't allowed for much active participation by the general community. Ben Pace sketched out an alternative more communal ritual we could do next year.
I'm not really sure what you mean about "true opinion of the community", you mean true opinion as to whether the ritual is any good? Or as to what action should be taken?
I care about the entire LessWrong community. I'm not sure where the exact boundaries lie–it's more than posters/commenters and probably short of anyone who's ever read a LW post–but I'm especially interested in the group who I feel like I can trust to work with me when the stakes are real and high. The Petrov Day ritual to date was designed to show that this group exists and trust each other, and I think that's a powerful and valuable thing to do, if you can do it.
Naturally, an ideal Petrov Day design would be both something for the ideal community and perhaps also be something that strengthens the trust between an especially devoted community core.
Forgive me if engage with only part of this, I believe that the OP already acknowledges most of the problem you've described.
No forgiveness needed! I agree that the OP addresses this portion -- I read the OP somewhat quickly the first time and didn't fully process that part of it. And, as I've said, I do appreciate the thought you've put into all this.
I think I differ from the text of the OP in that social-shaming/lack-of-protest-method in rituals is often an okay and sensible thing. It is only when this property is combined with a serious problem with the ritual itself that I get worried -- but I have a hunch that you'd agree with this.
I care about the entire LessWrong community. I'm not sure where the exact boundaries lie–it's more than posters/commenters and probably short of anyone who's ever read a LW post–but I'm especially interested in the group who I feel like I can trust to work with me when the stakes are real and high. The Petrov Day ritual to date was designed to show that this group exists and trust each other, and I think that's a powerful and valuable thing to do, if you can do it.
I agree that having/establishing a group of people you can work with/trust is a good thing, and I think that rituals about this can be beneficial. However I have two main objections to this perspective:
#1.
It is not obvious to me that identifying a group unlikely to press the button in a Petrov Day ritual is one capable of coordinating generally when stakes are real and high. As commenters have noted, social pressure incentives stack pressure against defecting. Moreover, if you are selecting for people who you know well enough to speculate on behavior in these circumstances, you are probably also selecting for people more deeply connected for the community for whom these pressures matter a lot.
#2.
I don't think an existence proof for a 100-strong set of LWers who don't press the button in a Petrov's Day ritual is particularly useful or surprising to me. If 50% of LWers would press the button and 50% wouldn't, its mathematically obvious that such a group exists. The actually arguably impressive/surprising part of the Ritual is not "does this group exist?" -- its "hey look! we have a selection process with strong enough discriminatory power to find one-hundred people who will act in a certain way."
This could mean something important symbolically -- about how so many people in the community are trustworthy that we can assemble a group with even our imperfect judgement. But it could also mean the following things:
So its hard for me to find much symbolic importance in the thing that the current Petrov Day Ritual is establishing.
But, social pressure aside, the establishment of high-stakes trust group is not obviously useful/relevant information to a typical community member. The capacity for distinguishing a high-trust group in a given community is only relevant to me if I can also distinguish a high-trust group that I can work with. In other words, knowing that such a group exists theoretically does not mean much to me if I happen to be in a branch of the community that I can't/shouldn't trust. My impression is that people who receive the code and choose not to press the button are basically anonymous, so this is the case here.[1]
If one knew for sure that this group delineated trustworthy people capable of coordinating effectively in high-stakes scenarios, it may be useful for ritualized (optional) self-reveals after the fact. However, I would actually caution against this as it could have a reverse/harmful effect if any participant does anything harmful ever -- since it could establish a coalitional Schelling Point of mutual protection or the perception of one -- creating real or perceived complicitness in harm.
Naturally, an ideal Petrov Day design would be both something for the ideal community and perhaps also be something that strengthens the trust between an especially devoted community core.
Yep! In practice, I don't think Petrov Day needs to everything though -- and its probably easiest to create a really strong ritual that captures one theme and then explore secondary activities/processes that don't interfere with the core.
Strengthening trust of the community core is a good thing -- and I don't think every ritual has to be about the entire community or vice-versa. I'm more concerned about what the selection-process+ritual-combi itself signals (both to the core and everyone else) about the kind of compliance behaviors expected by the core (the current process sort of confounds them). For contrast, if the selection process credibly demonstrated that it was just selecting Core Members, dropped the social-pressure incentives, and then demonstrating that core can trust each other not to blow things up then it would be a lot more meaningful in the sense you are gesturing at.
Perhaps there's also a minor frame difference here where you already see the process as basically something like "pick core members" based on your experience while that wasn't my default assumption.
[1] To be fair, I doubt that actually affiliating oneself with this group/parts-of-it would be overwhelmingly difficult -- given the obvious affiliations and the of people who have stated that they've received codes.
What exactly does this bit mean?
We were unable to find 200 people (100 for each forum) who wanted to be part of our community of people who don’t take destructive actions.
You found 100 people on each forum. (Right?) They all refrained from launching the missiles.
Do you just mean that some of those people didn't interpret the ritual the same way as you did, so that even though they didn't take destructive actions you worry that in other circumstances they might, or that they didn't take destructive actions for the wrong reasons, or that they didn't see the actions they refrained from as destructive in the way you did? That doesn't seem to me like a failure on your part; at least not the sort of failure you could possibly have been very confident of not occurring when picking 100 people without personally knowing all of them well.
The second–and I think more serious–complaint around lack of opt-in is that it leaves people who object to the ritual with no good option. If you don’t press the button, you are tacitly cooperating with a ritual you object to; if you do press it, you’ll have destroyed value and be subject to serious social sanction.
Moreover, the organizers (me, EA Forum staff) have declared by fiat what the moral significance of people’s symbolic actions are. This goes beyond just deciding what the ritual is and into deciding what’s good and bad symbolic behavior (with strong social consequences). While the Petrov Day ritual might be innocuous, it is a scary precedent if LessWrong/EA Forum organizers freely shape the moral symbolic landscape this way, without the checks and balances of broader community discussion.
I think these problems are inherent in the Red Button. I got a little wild in the postmortem thread for last year, but I still have most of the same thoughts, just more calmly.
But I was most persuaded by the commenters who pointed out that the Red Button doesn't celebrate what we want to celebrate about Petrov. The moral is not "Thank goodness Petrov didn't 1) recognize the false positives and 2) send the report up the chain anyway out of spite or poor impulse control or a belief that the satellite signatures were a prank." It's "Thank goodness Petrov recognized the false positives, using Bayesian reasoning and game theory, and therefore didn't send the report." Conceptually, it's "Thank goodness a human used zir brain when it mattered and may have thereby prevented the apocalypse."
Thank you for the thorough postmortem that didn't ignore these issues.
edit: Lethriloth's parent comment went up while I was drafting my comment. I see that zir analysis of what we're celebrating about Petrov is different from mine, but also totally understandable and defensible. I think this literal ambivalence in the story is another reason that LW/EAF should think more about how to celebrate Petrov Day.
I see a lot of commentary here about Petrov which flatly disagrees with the Wikipedia article about him. Some central notes, bolding mine:
General Yury Votintsev, then commander of the Soviet Air Defense's Missile Defense Units, who was the first to hear Petrov's report of the incident (and the first to reveal it to the public in the 1990s), states that Petrov's "correct actions" were "duly noted".[2] Petrov himself states he was initially praised by Votintsev
He was reassigned to a less sensitive post,[18] took early retirement (although he emphasized that he was not "forced out" of the army
In 1984, Petrov left the military and got a job at the research institute that had developed the Soviet Union's early warning system. He later retired so he could care for his wife after she was diagnosed with cancer.[7]
In an interview for the film The Man Who Saved the World, Petrov says, "All that happened didn't matter to me—it was my job. I was simply doing my job, and I was the right person at the right time, that's all. My late wife for 10 years knew nothing about it. 'So what did you do?' she asked me. 'Nothing. I did nothing.'"
The most important conclusion here is that Stanislav Petrov was assigned to monitor an alarm system. He reported a false alarm because he believed the alarm was false. If he had believed the alarm was real, he would have reported an attack, because that was his job.
I think the LessWrong Petrov day celebration is a good tradition, despite the (IMO minor) flaws mentioned in this discussion.
While I think lesswrong provides notable value to me, I don’t think you can just divide the amount of value it’s given me by the number of days since I started reading content on it and say that each individual day is worth that much. If lesswrong were down once a month that wouldn’t remove 1/30 of the value of the website; I’d just read those articles on a different day.
While there’s definitely a cost to the front page being down for a day, the fact that I could just wait a day with next-to-no consequence makes it feel not very substantial. Maybe it’s different for other people?
One problem with the "Petrov day" ritual that has been created seems to be that in addition to the virtues 1 & 2 from the Petrov story someone arbitrarily added virtues 3 & 4 (and ignoring others virtues related to Petrov such as not abiding to social pressure) which somehow contradict his values:
3. Not taking unilaterally taking large (and irreversible) action
Petrov took unilaterally (and arguably large) action!
4. Cooperating / being the kind of person who can cooperate / being the kind of community that cooperates with itself, especially when the stakes are high
Petrov did not cooperate with his peers. He deliberately had to be non-cooperative to achieve what we are celebrating him for!
There were some reports of people seeing a frozen countdown on the button, that disappeared when the page was refreshed. Was this an intentional false alarm? I had assumed that was the case, as a false alarm with some evidence that it was false echoes some parts of Petrov's situation nicely.
While the Petrov Day ritual might be innocuous, it is a scary precedent if LessWrong/EA Forum organizers freely shape the moral symbolic landscape this way, without the checks and balances of broader community discussion.
I think this is fair. and this makes me realize that the LessWrong team has more power (and therefore more responsibility) than we previously credited oursevles with. We set out to build culture, including ritual and tradition, but it’s another matter to start defining the boundaries of good and bad. I think possibly this should be done, but again probably with more community consultation.
I don't find this precedent scary. IMO forum moderators (and other leaders of voluntary easy-to-exit communities) should generally be bold and decisive in shaping the communities they're responsible for. Maintaining a space like LW requires leaders with freedom of action and the ability to make controversial moves. Community discussion has its place, but that place should be as a source of feedback and advice to a person/team with ownership, not a line-item veto. By all means incorporate reactions and complaints into the design of future events iff you feel it's warranted, but your wording here makes me suspect that you might be feeling too beholden to the loudest unhappy people, and I encourage you not to worry too much about that. (There will always be some unhappy people!)
The LW team has earned its power and responsibility by creating this space. If you hadn't, we wouldn't be here.
Typo thread
"In case you missed it, neither site was taken down, despite 200 people being issued codes that would allow them to do so [1][2]." These implied footnotes don't lead anywhere as far as I can tell
"Virtues 2 and 3 are in some tension" The four virtues are currently numbered 1,2,1,2
I apologize for not posting this closer to Petrov Day. It’s been a busy month and there was much to think about.
You can view the EA Forum’s retrospective here.
This year was the third Petrov Day celebration on LessWrong in which the site was endangered, and the first year we joined together with the EA Forum. In case you missed it, neither site was taken down, despite 200 people being issued codes that would allow them to do so [1][2]. Huzzah!
Although neither site went down (and thus there's no need for a blow-by-blow analysis of whodunit and why), there are some interesting things to review. In particular, there were some substantial criticisms of the Petrov Day ritual this year and last year that I want to address.
Why Petrov Day
The annual Petrov Day post recounts the basic story of Petrov Day, yet given the questions that were asked this year about what Petrov Day should be, I think it’s right to first revisit why we celebrate Petrov Day in the first place. The following is my own personal take, the one from which I’ve acted, but it is Not Official.
We find ourselves at what may be one of the most critical periods in the history of humanity and the universe. This is kind of crazy–though I’ll refer you to the writings of Holden Karnofsky for a compelling argument for why believing anything else is equally crazy. In the next few decades, we might go extinct (or worse), or we might commence an explosion in progress and productivity that propels us to the stars, allowing us to take the seemingly barren universe and fill it with value.
Petrov Day is a celebration of not going extinct. It’s a commemoration of not taking actions that would destroy the world. It’s about how Petrov chose not to follow policy and relay his alarm because, in his personal estimation, it was probably a false alarm. If he had relayed the alarm, there’s a chance his superiors would have chosen to launch nuclear missiles at the US, and history would be very different.
We can identify two virtues worth applauding in the story:
On September 26th, we celebrate these virtues and attempt to enshrine them in our community. We say to ourselves and others I accept the virtue of not destroying the world, even when there’s pressure to do it! We don’t do this for idle spiritual fulfillment–we do it because there’s a real chance that we or our community may soon face actual choices that resemble Petrov’s. Be it AI, bio, or general policy, our community is represented and our influence is real. As such, the values we take as our own matter.
In addition to the virtues directly displayed by Petrov, we can add others that are important for not destroying the world:
3. Not taking unilaterally taking large (and irreversible) action
4. Cooperating / being the kind of person who can cooperate / being the kind of community that cooperates with itself, especially when the stakes are high
Virtues 2 and 3 are in some tension and there’s probably a meta-virtue of judging which to apply. The default principle might be like “use your own judgment to avoid destructive actions; don’t rely only on your judgment alone to take [potentially] destructive actions.”
Ritual
Eliezer posted about Petrov Day first in 2007 and in 2014, Jim Babcock wrote a ritual guide for a ceremony that people could conduct in small gatherings. At some point, a red button that would end the ceremony was introduced to the tradition. You’d be a real jerk to press it, thereby ending the Petrov Day celebration for everyone.
In 2019, the LessWrong team decided to create a Petrov Day ritual for the entire community by doing something with the website.
I wasn’t involved in Petrov Day that year, but I believe the team then wanted to celebrate all the four virtues I listed above (and maybe others too) as part of a general let’s celebrate the virtues involved in not ending the world. Unfortunately, it’s quite tricky to symbolize 2. (using your own judgment against incentives) within a game.
In addition to celebrating the four virtues above, LessWrong organizers wanted to further use Petrov Day as an opportunity to test (and hopefully prove) the trustworthiness and ability to cooperate of our community. Symbolism is powerful and it’s meaningful if you can get a large group of people to go along with your ritual. From that arose the challenge of finding N people who wouldn’t press the button. The higher the N we could find who don’t press the button, the more people we would have who are bought into our community–all of them treated the value of the trust-building symbolic exercise as more important than having fun or objecting or financial incentive or anything.
I feel pride and reassurance if I imagine truthfully saying “we have 1000 people that if we give them the chance to be a troll or a conscientious objector or a something–they don’t take it, they hold fast in not taking a destructive action”. The LessWrong frontpage is a big deal to the LessWrong team, and putting it on the line was a way of buying some gravitas for the ritual.
It’s because having N people who don’t press the button is such a powerful idea that people regard the ritual seriously and look poorly upon anyone who’d damage that. We succeeded in 2019 with no one pressing the button, yet failed in 2020. 2021 was to be a high-stakes tie-breaker involving another community.
Although the button(s) wasn’t pressed this year, I actually feel that we failed. We were unable to find 200 people (100 for each forum) who wanted to be part of our community of people who don’t take destructive actions. I don’t know that we failed by a lot, but I think we did. This is our failure as organizers as much as anyone else–we were responsible for choosing people and for designing the ritual.
An Aside About Community Buy-In
There has been criticism that the LessWrong team unilaterally designed and deployed the community Petrov Day ritual, deciding for the community at large what was going to be celebrated and how. I think this is a fair charge.
There are historical explanations for why the Petrov ritual evolved the way that it did, and, separately, principles and policies that can speak to whether that's good or bad.
Historically, building A Big Red Button That Takes Down The Site felt like a pretty straightforward evolution of the tradition people were already enacting in their homes and parties. It didn't seem like the sort of step that required public discussion or vetting, and that still seems like the correct decision for 2019
Additionally, the team prepared its Petrov Day ritual somewhat at the last minute, and found itself in a position where a big discussion wasn't really a viable option.
Given the choice between a LessWrong team (and an overall community) where people are willing to try ambitious and potentially-cool things on their own judgment, or one where people err toward doing nothing without discussion and consensus, it seems clearly better for 2019 LW to have forged bravely ahead.
(This is actually a good place to distinguish the Petrov Day moral of "don't take irreversible and destructive actions on your own authority" from a more general moral of "don't do anything on your own authority." The latter is no good.)
That being said, though: community rituals are for the community, and LessWrong is closer to being something like a public utility than it is to being the property of the LessWrong team. At this stage, it feels right and proper for the community to have greater input and a greater say than in 2019, and without having specific plans, I expect us to put real effort into making that happen well in advance of Petrov Day 2022. This feels especially important given both that Petrov Day now seems like it's going to be an enduring piece of our subculture, and also that we want it to be.
Not Getting Opt-In
Speaking of consulting the community, the 2021 ritual consisted of making people part of the game involuntarily by sending them launch codes. I see a few different complaints here.
The first is that launch codes are hazardous. Because the Petrov Day ritual is treated seriously (more on this below), someone who enters them (or just talks about entering them!) is subject to real social sanction, up to and including it affecting their job prospects. Our community takes character judgments seriously, and it's not at all clear what aspects of something like Petrov Day are "off limits" when it comes to evaluating people's cooperativeness, trustworthiness, impulsiveness, and general judgment.
In a world where the letter containing the codes was unambiguous about the cultural significance and the stakes of the Petrov Day ritual, I think receiving the launch codes would only endanger the highly impulsive and those with poor reading comprehension (and those should reasonably affect your job prospects). However, I think the way I wrote this year’s letter could be interpreted as a “Murder Mystery” invitation by someone not aware of the Petrov Day context. Plus, the letter didn’t explain the cultural significance to people who hadn’t been following along the LessWrong Petrov Day celebrations in last two years, which especially seems like a misstep when reaching out to a whole new subculture (i.e., the EA Forum).
I screwed up on that account and I’m sorry to anyone I put at risk. If you had pressed the button, it would have been on me.
The second–and I think more serious–complaint around lack of opt-in is that it leaves people who object to the ritual with no good option. If you don’t press the button, you are tacitly cooperating with a ritual you object to; if you do press it, you’ll have destroyed value and be subject to serious social sanction.
Moreover, the organizers (me, EA Forum staff) have declared by fiat what the moral significance of people’s symbolic actions are. This goes beyond just deciding what the ritual is and into deciding what’s good and bad symbolic behavior (with strong social consequences). While the Petrov Day ritual might be innocuous, it is a scary precedent if LessWrong/EA Forum organizers freely shape the moral symbolic landscape this way, without the checks and balances of broader community discussion.
I think this is fair. and this makes me realize that the LessWrong team has more power (and therefore more responsibility) than we previously credited oursevles with. We set out to build culture, including ritual and tradition, but it’s another matter to start defining the boundaries of good and bad. I think possibly this should be done, but again probably with more community consultation.
Why So Serious
Related to both complaints is the fact that Petrov Day has been treated increasingly seriously. It’s because it’s serious that people will sanction you if you press the button. And it’s because you believe it’s too serious that you might want to object/boycott the ritual (well, that’s one reason).
I think the degree of seriousness that the ritual is treated with is one of the questions that should be reconsidered next year in consultation with the community. It's possible, for instance, that Petrov Day should be a place where some amount of mischievousness is considered fair game, and not representative of someone's global character.
Notwithstanding, I personally want to defend the position that a very high degree of seriousness is appropriate: a serious ritual for a serious situation. The stakes we find ourselves facing in this century are genuinely high–astronomical value vs extinction–and it makes sense to me to have a ritual that we treat with reverence, to celebrate and encourage values that we treat as somewhat sacred. Or in short, things matter, so let’s act like they do. I don’t know that this argument will win out on net, but I think seriousness should be considered.
Aside from a general position that Petrov Day should not be serious, some have argued in particular the most recent Petrov Day ritual should be lighthearted because the only thing at stake is the LessWrong/EA Forum page going down. My response to that is sadness. There is understandably inferential distance between the LessWrong team and others about how valuable LessWrong is and what it means to take the site down for a day. As I wrote in the Petrov Day post:
LessWrong is not a mere source of entertainment. It’s a site whose content shapes how people think about their lives and make major decisions. If there was a person who was going to have their life changed by LessWrong (and this happens to many) who fails to because the site is down, that’s a tragic loss.
LessWrong is also used as a major communication tool between researchers. LessWrong being offline is not so different from removing a day from a major research conference. Or, to change tack: the operating budget of the LessWrong website has historically been ~$600k, and this budget is artificially low because the site has paid extremely below-market salaries. Adjusting for the market value of the labor, the cost is more like $1M/year, or $2,700/day. If I assume LessWrong generates more value than the cost required to run it, I estimate that the site provides at least $2,700/day in value, probably a good deal more.
Still, if we want stakes for the ritual/exercise/game, probably better to use something with lower inferential distance. It’s on me as an organizer to mistakenly think that just because I think something is valuable, that will be transparent to others, and given that, I accept that it’s on me that not everyone thought the last Petrov Day iteration should be a big deal.
I could imagine it being better if there’s $5-10k that simply gets burned if someone presses the button rather than going to some worthy cause. Either way, this debate has clearly not properly taken place.
For an idea of what next year could look like, see these notes from Ben Pace
An Aside: Repeating Mistakes
Many of the issues pointed out this year were pointed out last year. It’s a real failure to not have addressed them. This is my (Ruby’s) fault. I took over organizing Petrov Day this year (inviting the EA Forum to join LessWrong) but didn’t go back and re-read through the previous year’s comments. Had I done so, I could have avoided repeating some of the previous mistakes.
I do think that repeating mistakes is quite bad and am quite sorry for that.
Wrapping Up
Stanislav Petrov was on duty at a particularly fraught time in history. I think we are, too. This makes it imperative to think about the kinds of decisions we might face and prepare ourselves for them. It makes it crucial that know and practice our values and principles, so that we can rely on them even when temptations are strong or matters are unclear.
Rituals and traditions are what keep people true to their values. Having them or not might be the difference between us being a community that can succeed at its ambitious goals vs not–the difference between colonizing the stars and annihilation.
I regret the flaws of Petrov Day rituals so far, but I’m excited to keep iterating and innovating so we can make these essential values part of our community, cultures, and selves.
[1] We apologize for sending codes to some people who did not want the opportunity/responsibility/involvement of receiving them.
[2] On the day itself, 70 out of 100 EA Forum recipients opened the email, but only 30 out of 100 LessWrong recipients (perhaps due to emails getting eaten by spam).