On Destroying the World

I feel like this has unintentionally brought us closer to Petrov's actual experience.

This tradition has so far consisted of leaving the button alone, with no incentives to push it beyond the ubiquitous temptations of pushing buttons and/or trolling. But that is not what happened to Petrov.

Petrov received a message telling him to push the button or the Bad Thing would happen.

Petrov thought the message looked legit, but noticed there were clues that it wasn't.

Petrov had little time to make the decision.

He went with the clues and we lived. Chris didn't and we metaphorically died.

We are still unequal to Petrov, for now.

[-]abramdemski5y300

I feel like this has unintentionally brought us closer to Petrov's actual experience.

Unintentionally?!?

I am probably not following this as closely as many commenters here, but I 100% assumed it was intentional. It's just so good!

6gjm5y

This is why many of us wondered, when Chris mentioned in the main Petrov Day thread what had happened, whether the message was in fact from the LW mods.

[-]Kaj_Sotala5y59-1

This brings me to Oliver Habryka's comment:

> To be clear, while there is obviously some fun intended in this tradition, I don't think describing it as "just a game" feels appropriate to me. I do actually really care about people being able to coordinate to not take the site down. It's an actual hard thing to do that actually is trying to reinforce a bunch of the real and important values that I care about in Petrov day. Of course, I can't force you to feel a certain way, but like, I do sure feel a pretty high level of disappointment reading this response.

[Epistemic-emotional status: reporting an emotional reaction, which may or may not have correct reasoning; attempting to honor the emotional tone of that reaction without actually wanting to be confrontational about this. Important note: when I say that it feels like people are being shamed, I do not mean think that anyone actually intends to shame people; it's that I feel that that's what the tone will communicate to many, intentionally or not. I could be wrong about that too. Again, trying to fairly report the implicit model that one part of me has, without asserting that the model is necessarily fully correct.]

So I noti... (read more)

[-]habryka5y300

Yeah, I think this is a reasonable reaction, and I really appreciate you going in-depth on your reaction here. And maybe the right call is to basically not have any shared rituals and traditions that have a shared sense of importance, which feels to me like the western secular default.

But, I don't know, that does leave me feeling pretty empty and sad, and I notice that if I don't have an active and strong culture around me, that I just default to whatever other random culture around me does have any content, even if I don't really like the ideas of that culture, and it feels like a pretty major loss to me. I do think that culture is really important, and shared rituals and traditions and games like these feel like how you actually build a culture that has any substantial content. And I really like Petrov Day. I consider it and Solstice to be the two primary holidays we have that we get to shape to reinforce our shared values and ideals, and want us to make use of them.

Like, I do think it's important that we try our best to only send the invites to people who are up for taking this seriously, and it should be easy to opt-out. I think we should improve the communication t... (read more)

[-]Kaj_Sotala5y110

Hey, I'm relieved and grateful that you took this so well. :) I hesitated for a while before posting my comment. I get that this ritual was important for you and didn't want to disrespect that; probably also didn't speak up last year because I wasn't sure I could communicate it in a good way.

I totally get the desire for rituals, and think it's an important one; I haven't been to a Solstice but I appreciate what they're doing. I also don't have a problem with them, maybe because they don't feel like they are trying to claim anything that they're not.

Generally most of my problem with this ritual was a) some aspects of its execution, such as the communication, which is fixable, and b) the feeling that it's actually not very analogous to the dilemma it's trying to be symbolic for, and which it claims to be training people in. (I said a few words about that in the final paragraph of my response to lionhearted.) I think that if it really felt to me like it was teaching people to be more trustworthy and coordinate better in situations-like-the-one-Petrov faced, then I'd probably be very happy to have it around. I just don't feel like it's there... yet. :)

5habryka5y

Yeah, I do think we are definitely still figuring out a bunch of rough edges around this tradition. I do think that what we set up is reasonably analogous of what I think happened to Petrov, and the aims I have for it, though of course I would think that, but here is maybe one more attempt at synthesizing what values I care about reinforcing with Petrov Day: 1. Remembering that humanity is fragile, and that we have come close to destruction in the past, and will likely come close to destruction in the future. 1. This feels like it really straightforwardly resonates with the setup. You need to have some chance that things will go wrong to create a real sense of tension, but of course you don't actually want to have stakes that are so high that they destroy lots of value if you mess it up, in particular while you are still dealing with a lot of uncertainty about the setup. 2. Practicing the virtue of not taking unilateral action and being mindful that your actions can have large negative consequences, and that you will act responsibly with the power you are given 1. I think this is just a really valuable virtue for a community of people to have, and I do think it's what distinguished Petrov from most people in similar reference classes to him. Like, as I said in the other comment, he could have just been a bureaucrat who didn't care about his job, didn't pay much attention to what was happening, and just accidentally contributed to destroying the world. And similarly, I want the people around me to take responsibility for their actions, in particular if they have a large potential downside. I think of these two as the core virtues of Petrov day, and I think our current ritual does a pretty decent job at reinforcing them. Like, last year when the LessWrong frontpage had a chance to go down any minute, it really felt very analogous to how I imagine people must have felt like during the cold war, where a catastrophe could have happened any minute, and sudde

9Kaj_Sotala5y

This sounds like it's pretty well captured by current Petrov Day ritual, yeah, though I feel like it only being the front page rather than all of LW makes it feel much less serious. Doesn't Petrov's choice actually get closer to taking than not taking unilateral action, though? The current ritual captures "think carefully about your actions", yes, but as I understand it Petrov was supposed to report a missile launch to his superiors. who could in principle also have used their judgment to dismiss it as a false alarm. He did the right choice, no doubt, but it feels weird to use "saw an event that could have led to the end of the world, made a choice that involved going against his standing orders and the previous planning that many others had participated in, ultimately making the decision purely himself rather than communicating it to the people with the pre-designated authority to deal with it" as a symbol of coordination and avoiding unilateral action.

6habryka5y

I mean, I think taking all of LessWrong down would be a bit of a dick move. Like, the frontpage is what matters most to the people who participate and is a resource that feels fair and reasonably to destroy, because it being down mostly just costs the people who participate in the ritual. But I feel like as a developer on LessWrong I have a pretty serious responsibility to be a good shepherd of content, and to make sure that you can reliably link to LessWrong content, and that you can reliably read the sequences, without it breaking. Most of the people who read that content aren't regular users, they are people who got linked here from some other blogpost on the internet, and I don't want to externalize our bad decisions into giving them a bad experience. I think in general, I wouldn't want to run rituals like this that randomly damage some public infrastructure. Like, I wouldn't want to make it so that when someone presses a button, we barricade a random road in Berkeley. Making all the content on LessWrong inaccessible feels similar to that. It's not my right to remove people's access to that.

6habryka5y

Sorry, after thinking about this, I basically think that "unilateral action" is just a confusing choice of words. Let's replace it with "being given substantial purely destructive power, and wielding that power responsibly,", because I think while there was a substantial unilateral component to the cold war, I don't think Petrov's choice in particular was that reliant on unilateral considerations.

2Chris_Leong5y

I don't have a strong opinion on how serious Petrov Day should be. Just that if you wanted it to be taken more seriously then it should have been set up differently.

[-]Ben Pace5y250

Great comment, naturally. I appreciate your epistemic status quite a bit.

I think I want to respond to the idea that it's contradictory and bad to signal that the Petrov Day button is serious and signal that it's fun. A few examples:

HPMOR is a book about growing up, failure, and death. It's also hilarious and riveting.
Unsong is 50% puns. It also contains a chapter describing hell and torture in some detail.
Embedded Agency is research done downstream of the potential for advanced optimizers to lead to an existential catastrophe for humans. It's also a cute and colorful cartoon.
Previously warring countries often come together and have their sports teams play. It really matters that they don't cheat and play honorably, even if it's "fun" and "play". It's a game, but it's not "just a game".
Some animals sheath their claws for dominance fights, where the losing player loses real status but isn't physically harmed. Again, it's a game, and it's in some ways play. And it's also serious.
I'm here trying to build a community around the art of rationality. We also do an April Fools' joke every year, like that time we made everyone's font size proportional to their total karma for a day.
People ha

... (read more)

[-]Bucky5y270

We often distinguish between safety critical and non-safety critical components. The latter make up about 95% of components in my business and in general the thing we care most about is average performance.

In safety critical components we care about the worst component (material / manufacturing defect etc.) in e.g. 1,000,000. Otherwise >1 in 1,000,000 brakes fail and the vehicle runs someone over or drives into a canal.

The examples that you give of jokey but serious things are almost all non-safety critical things (except the dominance contest but I think that's quite a different example). If I miss that embedded agency is about something serious then that doesn't really matter - someone who makes that mistake is probably not really who it is important to make understand. The overall effect of the series is the most important thing.

My impression is that the message you sent is great for average performance (and that the most natural way to read it is as you intended) but that it isn't optimised for communicating with the biggest exception in 270. The person who shares the least common knowledge about the ritual or reads the message the fastest or has the prior you mention or a p... (read more)

8Ben Pace5y

This is a solid point. And I'm glad to hear you learned a lot from this, I think many of us are (I know I am). I still think something is being missed, let me say what that is. Like, there's the part where I try to correctly pick people and communicate with them, where I take the effort to ensure they understand what's going on and don't take destructive action. I will do more user testing. I will check my writing more with friends and colleagues. I will spend more time reading the comments and posts of users I'm planning to give codes to. Perhaps I will make it take a little work to get your codes, like filling out a brief form that says "Yes, I understand this is a tradition about not taking massively destructive action" and "Yes, I understand the Frontpage will literally be unavailable for 24 hours if I submit the codes" and "Yes, I understand that the LW team will never ask me to submit my codes" before you get the codes. Certainly I will update the general algorithm I used to select people this year, which I realize in retrospect was fatally flawed (post in-progress on that). There's lots to do better here on my part, certainly. I will do better. There is also the part where, as a person in the world and as a user on LW, you decide what principles you have and what responsibilities you can take on. How hard do you try to generally be the sort of person who does not needlessly destroy things? Do you want to be the sort of person who can be entrusted with destructive power? (This is often a prerequisite for being trusted with any significant power.) Regardless of the reasoning, we gave everyone a destructive button for a day. If you're the sort of person who wants to be trusted in this way, both on Petrov Day and with matters even more important, you should not use it, not joke about using it, not try to use it as leverage for personal gain, and so on and so forth. Many people don't take this sort of responsibility. Feynman, for all his other virtues that I a

5Chris_Leong5y

I really don't see the frontpage being down for a day as that bad. I guess that a lot of us (myself included!) spend too much time on the Internet, so maybe very occasionally not being able to access a particular site is a good thing? Anyway, I definitely would have been more careful if I knew that this was something people cared about so much.

3Ben Pace5y

Do you disagree with the Fermi estimate about how willing-to-pay people would be? (And more than that, do you think most websites should be randomly down for a day – that this would be a better state of affairs? I don't expect you do.) I'm glad to hear you would have been more careful if you knew people cared so much.

6Chris_Leong5y

I think it's a reasonable Fermi estimate. That said, people would be willing to pay a lot more for Facebook. Doesn't mean it's more valuable. "Do you think most websites should be randomly down for a day?" - well, it's not just down for no reason. It'd also serve as a reminder of the importance of existential risk. And then people would be able to read about who took it down and possibly why afterwards. There could be significant utility from reading the post-mortem. But I guess my true rejection is that I saw it as just a game and I assumed that you wouldn't set up such a game unless you judged the cost to be insignificant. Which seemed in line with just taking down the front page for a day. And, I didn't see any reason to double-check what I presumed your judgment was here.

4Bucky5y

There’s a lot of stuff I agree with here and some stuff I’d push back on but probably worth waiting for the post-mortem before going deeper.

6Chris_Leong5y

That's actually a pretty big difference.

[-]lionhearted (Sebastian Marshall)5y170

Hey - to preface - obviously I'm a great admirer of yours Kaj and I've been grateful to learn a lot from you, particularly in some of the exceptional research papers you've shared with me.

With that said, of course your emotions are your own but in terms of group ethics and standards, I'm very much in disagreement.

The upset feels similar to what I've previously experienced when something that's obviously a purely symbolic gesture is treated as a Big Important Thing That's Actually Making A Difference.

On the one hand, you're totally right. On the other hand, basically the entire world is made up of abstractions along these lines. What can the Supreme Court opinion in Marbury vs Madison be recognized as other than a purely symbolic gesture? Madison wasn't going to deliver the commissions, Justice Marshall (no relation) knew that for sure, and he made a largely symbolic gesture in how he navigated the thing. It had no practical importance for a long time but now forms one of the foundations of American jurisprudence effecting, indirectly, billions of lives. But at the time, if you dig into the history, it really was largely symbolic at the time.

The world is built out of all sorts of ab... (read more)

[-]Kaj_Sotala5y140

Thanks for engaging :) My upset part feels much calmer now that it has been spoken for, so I'm actually pretty chill about this right now. You've had a lot of stuff that I've gotten value from, too.

Canonical reply is this one:
https://www.lesswrong.com/s/pvim9PZJ6qHRTMqD3/p/7FzD7pNm9X68Gp5ZC

But note also that that post contains a lengthy excerpt about how the "Dark Side" descends into cultishness and insanity in situations where the word of leaders is accepted without question. That was clearly also depicted as the opposite failure mode.

I agree that rationalists don't cooperate enough, and that often just offer criticism when it's not warranted. But... it feels like a Fully General Counterargument if you take to that the point of "no coordination may be criticized, ever, including situation where people are arguably being shamed for having good epistemics". That sounds like this bit from the post:

How do things work on the Dark Side?
The respected leader speaks, and there comes a chorus of pure agreement: if there are any who harbor inward doubts, they keep them to themselves. So all the individual members of the audience see this atmosphere of pure agreement, and they feel more

Good points.

I'll review and think more carefully later — out at dinner with a friend now — but my quick thought is that the proper venue, time, and place for expressing discontent with a cooperative community project is probably afterwards, possibly beforehand, and certainly not during... I don't believe in immunity from criticism, obviously, but I am against defection when one doesn't agree with a choice of norms.

That's the quick take, will review more closely later.

[-]habryka5y440

I want to point out a few things in particular. Firstly, the email was sent out to 270 users which from my perspective made it seem that the website was almost guaranteed to go down at some time, with the only question being when (I was aware game was played last year, but I had no memory of the outcome or the number of users).

I mean, this is a fine judgement to make, but also a straightforwardly wrong one. Last year we had ~150 people, and the site did not go down, with many people saying that we really have to add more incentives if we want to have any substantial chance of the site going down. I do think it's a pretty understandable mistake to make, but also one that is actually really important to avoid in real-life unilateralist situations.

Obviously, someone pressing the button wouldn't damage the honor or reputation of Less Wrong and so it seemed to indicate that this was just a bit of fun..

Of course it damaged our reputation! How could it not have? Being able to coordinate on this is a pretty substantial achievement, and failing on this is a pretty straightforwardly sad thing to happen. I definitely lost a good amount of trust in LessWrong, and I know of at least 10 other pe... (read more)

[-]Richard_Ngo5y220

Of course it damaged our reputation! How could it not have? Being able to coordinate on this is a pretty substantial achievement, and failing on this is a pretty straightforwardly sad thing to happen.

I think this is only true conditional on the Petrov Day LW Button being a Serious Thing. But the whole question is whether or not we should consider it a Serious Thing in the first place. Outsiders likely won't, they'll just see it as a game.

More generally, the Schelling choice is rabbit not stag - where Rabbit is "don't take this seriously" and Stag is "do take this seriously". By putting this thing online and expecting everyone to take it seriously, but without providing a really solid justification for why they should, you're choosing Stag without prior coordination, which is generally a bad strategy.

(I also endorse Kaj and Neel's comments below).

[-]habryka5y120

I think this is only true conditional on the Petrov Day LW Button being a Serious Thing. But the whole question is whether or not we should consider it a Serious Thing in the first place.

Hmm, I do think this is right. But I do think the payoff matrix here is pretty asymmetric. Like, I think it's obvious that on-net, given reasonable levels of ambiguity, you will lose some reputation. There is a question of how much, but I think it's pretty reasonable to assume that you will lose some good amount, because at least some fraction of people will take it seriously.

Like, I do think that it's fine to push back and say that it should just be a game, and that the people who are taking it seriously are wrong, but as a statement about social reality, predicting that it will not cost you reputation just seems like a wrong prediction.

Like, I am fine with the statement that it shouldn't cost you reputation. But saying that it won't cost you reputation, feels pretty wrong.

[-]Richard_Ngo5y110

I think we lose some reputation if people think that we are unable to choose Stag even in Serious Situations. But the main thing that signals to outsiders that this is a Serious Situation instead of a fun game is the disappointed reactions after someone chooses Rabbit. (By default outsiders are much less likely than insiders to think of this sort of thing as serious, and it was already ambiguous enough that many insiders didn't think of it as serious). If the community reaction was more like "What a great learning experience" and "This is a super interesting outcome" then I doubt there'd be a significant reputational cost. I'd estimate that the cost in weirdness points of running this event in the first place is about an order of magnitude higher.

An analogy: suppose the military practises a war game and sometimes fails to achieve its goal. I don't think this means they lose reputation. In fact, for certain classes of games, you lose more reputation by always succeeding in your goal, because that means that the goals are rigged. Same here: maybe the LW team sent out the invitations in such a way that they were very confident someone would push the button, or maybe in a way where they were very confident nobody would; I can't tell from the outside.

[-]habryka5y210

Yeah, to be clear, I do think it is actually a valuable signal to have failed at the Petrov Day goal at least once, because it signals pretty credibly that things are not rigged, and failure is possible.

I do also think that if you want your war game to be taken seriously as a sign of your competence, it's important that both you and the people you were war-gaming against were playing seriously. This doesn't mean that the war-game had to be a "Serious situation", but it does mean that your soldiers shouldn't have just gone "lol, it's just a game" and started playing cards or something because they got bored.

Like, sure, we could make this just a fun game, which would cause us to also not have to be worried about reputational risks, but I don't see much value in the version of this that is just a fun game, with no serious component. I am not super confident about the right balance of seriousness and fun, but I am pretty confident that a world where nobody took this seriously just doesn't seem very interesting to me. It doesn't allow me to build any real trust with anyone else, and feels like it deteroriates the real and important lessons we can learn from Petrov Day.

1Chris_Leong5y

"Of course it damaged our reputation! How could it not have?" - I suppose it might if it's a serious exercise and a lot more people seem to interpet it that way than I expected

[-]Neel Nanda5y*250

Thanks for writing this! It seemed like people were being unwarrantedly unfair to you in that thread.

My personal experience was getting the email from Ben, and this being the first I'd ever heard about LessWrong's approach to Petrov Day. And I somewhat considered pressing the button for the entertainment value, until I read the comments on the 2019 thread and got a sense of how seriously people took it.

I think it's completely reasonable to not have gotten that cultural context from the information available, and so not to have taken the whole thing super seriously.

And personally I found it fairly entertaining/education how all of this turned out (though it's definitely sad for all the Pacific time people who were asleep throughout the whole thing :( )

EDIT: Just wanted to add that, now I have the cultural context, I think this was all an awesome celebration and I'm flattered to have been invited to be a part of it! My main critique was that I think it's extremely reasonable for Chris not to have had the relevant context, but many of those commenting seem to have taken this background context as a given, since it's clear to them.

[-]Bird Concept5y*130

I'm genuinely confused about the "pressing the button for entertainment value".

The email contained sentences like:

Honoring Petrov Day: I am trusting you with the launch codes. [...] On Petrov Day, we celebrate and practice not destroying the world. [...] You've been given the opportunity to not destroy LessWrong. [...] if you enter the launch codes below on LessWrong, [you will remove] a resource thousands of people view every day.

And no sentences playfully inviting button-pressing.

Maybe I can't unsee the cultural context I already had. But I still imagine that after receiving that email, I'd feel pretty bad/worried about pressing.

[-]Neel Nanda5y260

We'll come to this in a moment, but first I want to address his final sentence: "Like, the email literally said you were chosen to participate because we trusted you to not actually use the codes". I've played lot of role-playing games back in my day and often people write all kinds of things as flavour text. And none of it is meant to be taken literally.

I want to point out a few things in particular. Firstly, the email was sent out to 270 users which from my perspective made it seem that the website was almost guaranteed to go down at some time, with the only question being when (I was aware the game was played last year, but I had no memory of the outcome or the number of users).

Beyond this, the fact that the message said, "Hello Chris_Leong" and that it was sent to 270 users meant that it didn't really feel like a personal request from Ben Pace. Additionally, note the somewhat jokey tone of the final sentence, "I hope to see you in the dawn of tomorrow, with our honor still intact". Obviously, someone pressing the button wouldn't damage the honor or reputation of Less Wrong and so it seemed to indicate that this was just a bit o

... (read more)

1Ben Pace5y

This comment helped me understand better what it looks like without the 'cultural context', thx. Just to reply to one particular thing: Do you agree that the literal monetary value of the site being down for a day is (likely) greater than that? Never mind the symbolism, there's just like two thousand people who visit the Frontpage in a day, around half of whom might pay something in the range of $1-$10 for the site not to be randomly down on them for a day?

[-]Dagon5y170

I do not agree that the monetary value of this intervention is anywhere near that.

I didn't notice - I don't look at the front page, just using my bookmark to /allPosts. Many use GreaterWrong or RSS, and would be unaffected. Are there logs on how many actual visitors saw the front page down and did not hit any other pages until it was back up?

I'm a pretty heavy site user, and I would not pay $1 to have the site up a few hours or a day earlier in case of an outage. I'd likely pay on the order of $0.10-0.25/day on an annual basis if asked (and if it were a registered charity where I understood how my donation would be used), but having a day or two of downtime is just fine with me.

I'd especially not pay $1 to have the site be up sooner in case of a ritual/demonstration that is intentionally created by a site admin. If they think having the site down for a bit is a positive thing (indicated by the fact they wrote the code to do it), I defer to their wisdom.

2Ben Pace5y

Right. I'll briefly reply to each point: You don't look at the LW Frontpage, and neither do GW users or RSS users. This means you and they are outside the set of 2,000 daily visitors, so your lack of inconvenience is not evidence about theirs. (I don't have any logs, may look into getting some. As we don't have that info our uncertainty around that should be factored into the estimate.) You not wanting to pay $1 if the site was down is indeed a datapoint. I think many people would be fine with an outage. (I still think many would find it irritating.) I understand that you especially wouldn't in the case of the symbolism. I'm just trying to pin down the object level effects, to understand what was at stake before counting in all the symbolism. Overall I'm not certain, it's plausible the number is lower...

3Bucky5y

I don’t know how many of the 2000 would do the same thing but switching to GW for the day was fairly obvious to me. On the other hand I use GW on and off so this maybe gave me an advantage but I think the post on surviving the outage suggested doing that too. Short of checking GW traffic I guess it’s hard to know how many did this.

7Raemon5y

It is noteworthy that I think the sort of person who would bother to pay a dollar to keep the site up is also the sort of person who disproportionately might use greaterwrong (or, for that matter, the /allPosts page). The frontpage gets a lot of views but I think most of them are people who aren’t using LessWrong that seriously. I said earlier to Ben I thought the $3k number was at least plausible and seemed within an order of magnitude of right. But thinking more I do suspect it’s on the lower end of that order of magnitude I think there’s only a few hundred users for whom the LessWrong frontpage is actually enough-better than whatever else they might be doing that day that they might pay a dollar.

5habryka5y

GW had about 40 additional users show up on that day (which corresponds to roughly 35% traffic increase)

8Kaj_Sotala5y

If the entire site (not just the front page) was down, then I might on some days pay $1, if I was writing something where I wanted to cite/reference an older article. Otherwise, if I knew it would only last for a day, I would just wait it out.

3Neel Nanda5y

Interesting. My intuition was "24 hours isn't a long time, and it's just the front page, people can surely come back later". But while that's a small inconvenience, $1 worth of inconvenience sounds plausible. So yeah, fair point! $1-10k actually seems like a fair value for this, thanks EDIT: Reading the other comments on that point, it seems reasonable that LessWrong power users are best able to work around the outage, and the people who'd be most inconvenienced. And I expect most of those people to not know about GW (what is GreaterWrong anyway?), but this to correlate with caring less about the existence of LW. So I guess I'd lower the estimate a bit

6philh5y

https://greaterwrong.com is an alternate interface to LessWrong, implemented by... I think Clone of Saturn does most of the coding and Said Achmiz does most of the design work? Same content, different design, slightly different set of features. (E.g. no karma change notification, no voting on tags, but comment navigation is improved.) I tend to use it over LW because it's faster. You can generally just replace lesswrong with greaterwrong in a URL.

2Neel Nanda5y

Oh, thanks! That sounds really useful when LW is being slow on mobile

8Pongo5y

As someone who didn't receive the codes, but read the email on Honoring Petrov Day, I also got the sense it wasn't too serious. The thing that would most give me pause is "a resource thousands of people view every day". I'm not sure I can say exactly what seems lighthearted about the email to me. Perhaps I just assumed it would be, and so read it that way. If I were to pick a few concrete things, I would say the phrase "with our honor intact" seems like a joke, and also "the opportunity to not destroy LessWrong" seems like a silly phrase (kind of similar to "a little bit the worst thing ever"). On reflection, yep, you are getting an opportunity you don't normally get. But it's also weird to have an opportunity to perform a negative action. Also, it still seems to me that there's no reason anyone who was taking it seriously would blow up LW (apart from maybe Jeff Kauffman). So if there's a real risk of someone blowing it up, it must not be that serious.

[-]petrov_day_admin_account5y240

To add some missing context to this:
-I'm part of the EA community and have been for several years. To the extent that you need a community member to blame for this, it is me. When doing this, I was operating under the belief that the community would be judging me personally, which is why I openly admitted to doing this on Facebook.

-I would have known about Petrov Day anyway regardless of Chris' message.

-Phishing attacks can often have in excess of 80% success rate. If you had received this, you would have likely entered the codes as well, even though everyone thinks that they wouldn't. Which is just one of the reasons why it doesn't make sense to punish recipients for making this kind of mistake.

-The campaign wasn't targeted at Chris, it was sent to lots of users. Retrospectively, I should have excluded Chris from the list of users. (I really regret not doing this, and I would like to apologise to Chris for this.)

[-]Daniel Kokotajlo5y240

Source please on the 80% success rate of many phishing attacks? This is at least an order of magnitude more than I would have predicted, it blows my mind!

[-]philh5y250

Did a quick google. The only statistic I could find for how successful phishing attacks are is https://www.helpnetsecurity.com/2019/09/04/sme-phishing-attacks/:

43% of UK SMEs have experienced a phishing attempt through impersonation of staff in the last 12 months. Of those impersonation phishing attempts, it was discovered that two-thirds (66%) had suffered a successful attack, according to CybSafe.

66% is still way more than I expect, but there's no verifiable source. (Looks like CybSafe has incentive to exaggerate the numbers.) And it's not clear whether this is "66% of phishing attempts were successful" or "of organizations targeted, 66% suffered at least one successful attack". Certainly it doesn't support "you would have likely entered the codes as well".

Strong-downvoted pdaa's comment pending source.

[-]Adam Zerner5y100

Yeah. If 80% is the true success rate I would not expect the world to look the way it does. I would expect such attacks to be incredibly rampant and somewhere near the front of everyone's minds, at least in the sense of when you get a call from an unknown number you think it's quite likely that it's something unsolicited.

[-]philh5y190

Phishing attacks can often have in excess of 80% success rate. If you had received this, you would have likely entered the codes as well, even though everyone thinks that they wouldn’t. Which is just one of the reasons why it doesn’t make sense to punish recipients for making this kind of mistake.

Seconding Daniel's request for a source. But also, to clarify, does your attempt here count as one phishing attack in total, or one per message you sent?

If it's one per message, then 80% is double-plus-super-higher-than-predicted. But if it's one in total, then "you would have likely entered the codes as well" needs further justification. I said in the other thread that I wasn't super confident I wouldn't have fallen for it; but I don't think it's actively likely that I would have done, even taking your claim into account.

[-]philh5y140

To the extent that you need a community member to blame for this, it is me. When doing this, I was operating under the belief that the community would be judging me personally

As a note, to the extent that you're trying to actively shoulder the blame here (rather than simply describing where you think it falls), this isn't a call you get to make. I'm not saying here that Chris does deserve blame; just that to the extent he does, you can't take that away from him onto yourself.

And... having this expectation seems like kind of the same sort of thing that went wrong with the admins' messaging? Like, on a high level you could describe what led to the site blowing up as: "the admins expected people to feel one way about a thing, and acted on that expectation, but some people felt a different way, and acted in ways that surprised the admins". Similarly, you may have expected us to feel one way about your actions, such that we judge you personally; but if some of us feel a different way, and judge differently, well...

You said you wanted this to be a learning opportunity for the community, and I think (despite varying levels of annoyance) we're overall taking it as such. To the extent that it's a learning opportunity for you as well, I hope you take it as such.

[-]abramdemski5y120

Was the parallel to the real Petrov's false alarm intentional or unintentional?

[-]Chris_Leong5y120

"The campaign wasn't targeted at Chris, it was sent to lots of users. Retrospectively, I should have excluded Chris from the list of users. (I really regret not doing this, and I would like to apologise to Chris for this.)" - I don't know why you wouldn't consider me fair game. You really don't need to apologise to me.

8Linda Linsefors5y

I think it's great that you did this. It made the game more real, and hopefully the rest of us learned something.

4Pongo5y

Suppose phishing attacks do have an 80%+ success rate. I have been the target of phishing attempts 10s of times, and never fallen for it (and I imagine this is not unusual on LW). This suggests the average LWer should not expect to fall victim to a phishing attempt with 80% probability even if that is the global average

[-]Tyrrell_McAllister5y*230

I've played lot of role-playing games back in my day and often people write all kinds of things as flavour text. And none of it is meant to be taken literally.

This line gave me an important insight into how you were thinking.

The creators were thinking of it as a community trust-building exercise. But you thought that it was intended to be a role-playing game. So, for you, "cooperate" meant "make the game interesting and entertaining for everyone." That paints the risk of taking the site down in a very different light.

And if there was a particular goal, instead of us being supposed to decide for ourselves what the goal was, then maybe it would have made sense to have been clear about it?

But the "role-playing game" glasses that you were wearing would have (understandably) made such a statement look like "flavor text".

2Chris_Leong6mo

Quite possibly, but hard to know.

[-]Jan_Kulveit5y*200

There seem to be multiple meta- games

press the button or not
take it as a game | as a serious ritual | as a serious experiment
cooperate or defect on the implicit rule allowing play behaviour ~ "you are allowed to play and experiment in games and this is safe. it is understood actions you take within the game will not be used as an evidence of intent outside of the game". (imagine I play a game of chess with someone and interpret my opponent taking my pieces as literarily trying to harm me)
the meta-game of making the game interesting; cf munchkin
the meta-game of making the experiment valuable for learning
coordination about which of these games we are playing

To me Chris's story hears like [ press | game | cooperate | ? | ? | ]

Overall while I think there is a lot of value in having a community of people who do not press big red buttons, I also see a lot of value in noticing these other games and "cooperating" in them.

[-]Adam Zerner5y*170

At the same, the purpose of this experiment wasn't clear at all. I wasn't sure if it was having fun, increasing awareness or gaining insight into people's psychology.

Agreed. I was also provided with the codes, and to provide another data point, this is how I thought about it.

The terminal goal isn't to keep the site up. The question I (immediately) asked myself is whether entering the codes would make it more likely or less likely that people take xrisk seriously in the real world (roughly). I considered this briefly, but I realized that I too was confused about the point of the experiment, and thus decided to leave it alone.

[-]lionhearted (Sebastian Marshall)5y150

Umm. Grudgingly upvoted.

(For real though, respect for taking the time to write an after-action report of your thinking.)

I was tricked by one of my friends:

Serious question - will there be any consequences for your friendship, you think?

[-]Chris_Leong5y120

Why would there be? I'm sure they saw it as just a game too and it would be extremely hypocritical for me to be annoyed at anyone for that.

[-]petrov_day_admin_account5y360

Thanks, I'm glad to hear that. :) Also, very thankful that the LW community took this really well.

Beyond that, as for my motivations, aside from curiosity as to whether it would work, etc. I considered that it would be an interesting learning opportunity for the community as well. With actual nukes, random untrusted people also have a part to play. Selecting a small group of people tasked with trying to bring down the site might even be a good addition to future instances of Petrov Day.

For what it's worth, I took care to ensure that the damage from taking the site down would not be too great. The site was archived elsewhere, and the admins themselves accepted the risk of the site going down by starting this game. If this could have hurt people, I wouldn't have done it.

Beyond that, loyalty and trust are also very important to me. If the admins had trusted me with the launch codes, I wouldn't have nuked the site (intentionally).

After thinking more about this experiment, it has got me thinking about the payoff matrices. Is there anyone that would have pressed the button if there was guaranteed anonymity, and thus no personal cost? If so, make a second account - I'd be curious to hear your reasoning. Also, in this case there is no tangible benefit that anyone could get by nuking the site. How do we adapt this to situations where there is a benefit that can be gained by pressing the button?

P.S.: My offer still holds! Admins, if you're feeling adventurous, give me the codes next year and I'll prove that I won't use them!

[-]Bird Concept5y140

Is there anyone that would have pressed the button if there was guaranteed anonymity, and thus no personal cost? If so, make a second account

If I understand you correctly, that won't work. The identity of the button-presser is not determined by which account pressed the button. It's determined by the launch code string itself -- everyone got a personalised launch code. (Which means that if someone stole and used your personalised code, you'd also get blamed -- but that seems fair.)

8philh5y

I read that as "make a second account to say anonymously why you would have done it".

2Bucky5y

In the spirit of learning from this, I'd be interested to know how many people you sent the message to and how you chose them etc. I particularly liked the "You will be asked to complete a short survey afterward" touch - what made you think to include it?

[-]petrov_day_admin_account5y210

I think maybe 6-8, not sure. I was going to go further but the site went down too quickly. Users were selected based on having a large number of posts.

I wanted something to make it sound realistic. And rationalist/EA culture loves surveys and collecting data. :)

8Chris_Leong5y

Yeah, that honestly made it feel so real.

6Bucky5y

Part of me wants to say you plonker for falling for it (as you said, there were plenty of clues, plus the fact that the launch codes weren't repeated in the second message) but another part of me remembers that I fell victim to a Trojan once so I have some sympathy for you.

9lionhearted (Sebastian Marshall)5y

Different social norms, I suppose. I'm trying to think if we ever prank each other or socially engineer each other in my social circle, and the answer is yes but it's always by doing something really cool — like, an ambiguous package shows up but there's a thoughtful gift inside. (Not necessarily expensive — a friend found a textbook on Soviet accounting for me, I got him a hardcover copy of Junichi Saga's Memories of Silk and Straw. Getting each other nice tea, coffee, soap, sometimes putting it in a funny box so it doesn't look like what it is. Stuff like that. Sometimes nicer stuff, but it's not about the money.) Then I'm trying to think how my circle in general would respond to no-permission-given out-of-scope pranking of someone's real life community that they're member of — and yeah, there'd be pretty severe consequences in my social circle if someone did that. If I heard someone did what your buddy did who was currently a friend or acquaintance, they'd be marked as someone incredibly discourteous and much less trustworthy. It would just get marked as... pointless rude destructive behavior. And it's pretty tech heavy btw, we do joke around a lot, it's just when we do pranks it's almost always at the end a gift or something uplifting. I don't mean this to be blunt btw, I just re-read it before posting and it reads more blunt than I meant it to — I was just running through whether this would happen in my social circle, I ran it out mentally, and this is what I came up with. Obviously, everyone's different. And that's of course one of the reasons it's hard for people to get along. Some sort of meta-lesson, I suppose.

5Bucky5y

I think this case is fairly different to what you describe. The community organised for this to potentially happen and Chris publicised this fact to his friends. The community decided that it was worth the risk so the damage could be assumed not to be large and having the frontpage going down for 24 hours really isn't a huge deal. The actual damage is realisitically the fact that the experiment (and associated metaphor) didn't work but I feel like the lessons learnt should more than make up for that.

5lionhearted (Sebastian Marshall)5y

You're being very kind in far-mode consequentialism here, but come on now. Making your friend look foolish in front of thousands of people is bad etiquette in most social circles.

[-]Bucky5y100

I'd kinda assumed that one wouldn't do this unless they were confident their friend would be ok with it, as indeed seems to be the case.

[-]ckai5y120

I am an outsider/lurker, so maybe I just don't get it, but it seems to me that even if the messaging around this event is changed to make it more clearly serious rather than there being a possible interpretation of all in fun no particular outcome is better than any other, there is a very real (not symbolic) mixed message going on with the way things are currently set up. The first message is hey, we're doing this really cool ritual and you are invited to participate. The second message is we don't want our website to go down so don't do anything (please don't participate).

This is completely outside the question of what the event actually symbolizes and so on. It's about, well, as far as I understand the simulacra model, it's about the simulacra level 1 aspects of the situation.

Someone who didn't check their email/messages would participate as desired.

There is no clear way to accept the invitation to participate in an active way without threatening the desired outcome.

Maybe this is the point. Maybe it's meant to be confusing. Maybe the people running the event are deliberately setting up a situation where they are sending mixed messages on purpo... (read more)

3habryka5y

Yeah, I think this is a real problem. I do think there is something a bit interesting about the ambiguity "The only way to win is not to play", but there are also a bunch of costs associated with the weird ambiguity, and I am not sure how things weigh up. The current thing and framing is the best we could come up with in a few dozen hours of collective thinking, but we can probably do better, and maybe the weird ambiguity is too costly, though overall I think we got the basic idea across to most people, despite the ambiguity, so I am hopeful that we can iterate on that and get us all the way everyone getting it.

[-]kithpendragon5y110

Seriously: like lionhearted said, thanks for the postmortem! The thought process is important. Even if it meant some hurt feelings and a bit of inconvenience, we still got to learn something here. After all, learning something was the point, right? The more data we gather, the more likely we'll be better off in similar situations in other contexts.

[-]Ben Pace5y110

I’m still processing my thoughts and feelings about this year’s Petrov Day, and what you did. It’s very late here, so I’ll write something tomorrow.

[-]abramdemski5y100

Additionally, note the somewhat jokey tone of the final sentence, "I hope to see you in the dawn of tomorrow, with our honor still intact". Obviously, someone pressing the button wouldn't damage the honor or reputation of Less Wrong and so it seemed to indicate that this was just a bit of fun..

Haha, I take this sentence like 90%-100% seriously. Don't you think you lost a little honor in the process? Wouldn't it be something if a decade down the line LessWrong could say "we've played this game for X years, including at least Y users every time and Z users total, and no one has ever entered their launch codes?

But alas...

2Chris_Leong5y

"Don't you think you lost a little honor in the process?" - Of course, but some people take this way too seriously.

[-]arunto5y*90

I think we should focus a little bit more on the behaviour of the other participants in this game. Coordinating in order not to have a catastrophic event happening is difficult and takes effort. And just hoping that nobody does anything foolish seems to be a strategy doomed to fail in the long run.

Therefore those other participants who took this experiment really seriously might have done much more to prevent this outcome. E.g. forming a small group, announcing that they are dedicated to the front page not being nuked and that everybody seriously thinking about pressing the button should talk about that first. If across the time zones such a group had formed then Chris might have been convinced not to do it.

[-]Adam Zerner5y*50

I don't mean to belabor the point, but if this were meant as a deadly serious type of exercise I would have expected a much harsher penalty for submitting the launch codes. Why take just the home page down and not the rest of the site? Why only 24 hours? If it were a deadly serious type of exercise, I'd expect in the ballpark of taking down the whole site for a week to a month. I can also see taking it down for a year. Doing so would really hammer home how important xrisk is, which I think would be a very positive outcome and thus a potentially reasonable thing to do.

[-]Mati_Roy5y40

I like to imagine a future LessWrong with 1 million users, and not one pressing the button. That would be very inspiring and a strong signal of being a high trust culture.

[-]avturchin5y40

I have read about one possible case of false nuclear alarm which involveв something like fishing attack. Not sure if it was real or not, and I can't find the story now, but it could be real or could be creepypasta. Below is what I remember:

In 50s, nuclear-tipped US cruise missiles were stationed in Okinawa in several locations. One location got an obviously false (for some reasons) lunch command: the procedure was incorrect. They recognised it as false and decided to wait for clarification. But another location nearby recognised the command as legit and st... (read more)

1arunto5y

It seems not to be clear if it really happened that way: Bulletin of the Atomic Scientists Stars and Stripes

[-]Mati_Roy5y30

Thanks for your explanation

[-]Mary Chernyshenko5y30

At least you made it an actual game instead of a ritual. Thank you for this!

(Reading Faust is one thing, picking a side is another.)

[-]Ruby5y20

I read the email and the post, and the feeling of this "I do actually really care about people being able to coordinate to not take the site down. It's an actual hard thing to do that actually is trying to reinforce a bunch of the real and important values that I care about in Petrov day" wasn't really articulated anywhere.

I'm pretty sympathetic to this. I'm a LessWrong admin, last year on Petrov day, and someone had talked about selling codes, I considered my price. $10,000 is a meaningful some to me and I think was my thought. I don't remember what his f... (read more)

5Dagon5y

Wow. I honestly don't get it - do you have a link to the previous discussion that justified why anyone's taking it all that seriously? IMO, this was a completely optional, artificial setup - "just a game", in Chris's words. When I got the e-mail, I wondered if it was already down, and was surprised that it wasn't (though maybe I just didn't notice - it never seemed down to me, but I go straight to /allPosts without ever looking at the front page). There was none of the weight of Petrov's decision, and no tension about picking one or the other - no lasting harm for pressing the button, no violation of norms (or being executed for treason, or losing WWIII) by failing to do so if it were necessary. And no evidence one way or the other what the actual territory is. Really, just a game. And not even a very good one. The fundamental cooperation to take down the site had ALREADY HAPPENED. When someone wrote the code that would do so if someone pressed the button, that's FAR FAR stronger than some rando actually pressing the button.

7lionhearted (Sebastian Marshall)5y

Here was my analysis last year — https://www.lesswrong.com/posts/vvzfFcbmKgEsDBRHh/honoring-petrov-day-on-lesswrong-in-2019?commentId=ZZ87dbYiGDu6uMtF8 In fairness, my values diverge pretty substantially from a lot of the community here, particularly around "life is serious" vs "life isn't very serious" and the value of abstract bonds/ties/loyalties/camaraderie.

7Dagon5y

Thanks. I am not convinced, but I have a better idea of where our perspectives differ. I have to admit this feels a bit like a relationship shit-test, where an artificial situation is created, and far too much weight is put on the result. I'd be interested to hear various participants' and observers' takes on the actual impact of this event, in terms of what they believe about people's willingness to support the site or improve the world in non-artificial conditions.

4lionhearted (Sebastian Marshall)5y

Hmm. Appreciate your reply. I think there's a subtle difference here, let me think about it some. Hmm. Okay. Thrashing it out a bit more, I do think a lot of semi-artificial situations are predictive of future behavior. Actually, to use an obviously extreme example that doesn't universally apply, that's more-or-less the theory behind the various Special Forces selection procedures — https://bootcampmilitaryfitnessinstitute.com/media/tv-documentaries/elite-special-forces-documentaries/ As opposed to someone artificially creating a conflict to see how the other party navigates it — which I'm not at all a fan of — I think exercises in shared trust have both predictive value for future behavior and build good team cohesion when overcome. Me too, but I'd ideally want the data captured semi-anonymously. Most people, especially effective people, won't comment publicly "I think this is despicable and have incremented downwards various confidences in people as a result" whereas the "aww it's ok, no big deal" position is much more easily vocalized. (Personally, I'm trying to tone down that type of vocalization myself. It's unproductive on an individual level — it makes people dislike you for minimal gain. But I speculate that the absence of that level of dialogue and expression of genuine sentiment potentially leads to evaporative cooling of people who believe in teamwork, mission, mutual trust, etc.) Reasonable minds can differ on this and related points, of course. And I'm very aware my values diverge a bit from many here, again around stuff like seriousness/camaraderie/cohesion/intensity/harm-vs-care/self-expression/defection/etc.

LESSWRONG
LW

LESSWRONG
LW

82

On Destroying the World

82

82