content warning: oops! all throatclearing :(
by the way, there are some blatant textual parallels between how to get a friend to leave an abusive relationship and this piece. a lot of my recent writing (on antimemes, and easier to be good) have been building up scaffolding and getting some loud tangents out of the way, so that i can write this and clarify my own thinking on the matter. all of those other pieces contain models that seem strongly but only partially instructive when i think about having friends at labs.
after several false starts in writing this piece, what got the words to flow was to put the draft side by side with my finished piece on abusive relationships, and to let myself self-plagiarize generously.
this unfortunately sort of leaves a shadowy rhetorical move where i am strongly implying that frontier labs are like abusive romantic partners, but i do not endorse this in any real way.
like, often things are like other things, but it's not that deep! for example, some similarities are:
to put it another way, maybe there's like, what, a 30% similarity between these two things, but other things that can share a 30% similarity to an abusive relationship include
such that i do not really endorse any amount of valence rub-off.
the obvious difference that the lab employees are not being negatively coerced to stay. the frontier labs are not holding any lab employees hostage, and the reason that they are staying is because they seem to truly believe that they are doing the right thing.
i think the strongest opinion i really have at this point is like, hey if you are trying to get lab employees to quit by getting mad at them (which i think is a framing that the deeprats don't endorse, but somehow this is the felt experience of everyone i know who is working at a lab) this seems counterproductive to your own goals.
i am aware that that take makes me "atheist telling the christians that they are practicing their religion wrong"-coded, but you know what sometimes that is what being a blogger is all about.
An additional similarity, IMO: people in abusive relationships will often have strong motives to stay that are quite hard for them to [put into words / hold fully consciously / discuss with almost anyone / narrativize within the Overton window]. Likewise people at labs.
This seems to come from an assumption that the only impact-relevant thing they can do is to quit their job. But maybe you should try to get them to look into arguments for risk, talk to concerned peers, be more self-aware about their motivations, and work on internal advocacy and technical projects that reduce x-risk.
My guess is that the average researcher at a frontier lab could, just by talking to all the internal and external people I did when joining to get oriented, have positive impact equal to 5-10x the negative impact of a capabilities researcher.
I do like this idea. However, looking at it like that, maybe the optimum would be joining for a month, talking to a lot of people in that month, and then quitting again. Because if you stay long, like if people stay longer term, it's kinda hard emotionally to keep disagreeing with people. Like probably your own views are gonna start to change. Unless you're like very good at living in deep disagreements, which sounds quite stressful.
This was a possibility for me upon joining, and is probably right for some people. I might post more about this soon.
But anyway, I was writing about OP's situation which is different from mine. For one thing I can probably get to 20x at OAI due to my AI safety experience whereas the 5-10x was a conservative number someone with mainly capabilities experience (not that this means there aren't higher value things externally). For another, it's more important for OP's friend to talk to external people because most capabilities researchers don't understand the outside safety ecosystem. Lastly, I was assuming they'd be reluctant to quit their job before understanding risks as OP said. With no qualms about that the goal should be to be recruited by external orgs as much as to orient.
So I don't really get my friends to try to quit their jobs, as much as try to help them regularly orient to whether or not they should
I would suggest trying to get them to have somewhat more conversations with their peers about strategic uncertainties and tensions and possibilities, and to more explicitly hold up the unresolved tension between their beliefs like "this work we're doing might kill everyone" vs. "but we have to do it because XYZ" vs. "but if XYZ is true, then our lab should also be doing ABC, but they're not".
Is there a specific ABC that isnt countered by an XYZ? I thought the reason the community was split was because the answer wasn't so obvious.
I'm not sure. I'm generally saying that many individual people may have such things. Someone else at the same lab may have an XYZ, but not the first person. In that case, they have a big disagreement. Further, sometimes they aren't aware, so there's a false harmony / false consensus.
For example, I believe some people at Anthropic think something like:
AI is extremely dangerous to build. So, it would be best if no one built it. But, there's a big difference between building it with a bunch of work and care for alignment, vs. without that. And also it would be very hard to stop other people from building it. So we should lead the frontier. That way we can do it with work and care for alignment, which is better than OpenAI doing it; and if it because really extra super clear that alignment isn't doable, we can use that newly found evidence to get a pause.
Implicit in this are various implications, such as "If you have a company with thousands of employees and many billions of dollars of resources, you should have at least one big well-resourced team of dozens of people doing their best to get a pause.". But the companies are not doing that. This is a contradiction. Someone who holds the paraphrased belief should stop trusting their leadership and try to organize an internal campaign to stop the company from racing. If they aren't going to do that, they should at least talk more with their peers about it.
I am mostly of the opinion that it is:
I do not always do 1, because of the costs (mostly, upsetting people and then feeling a need to reply to people in attention-grabbing and confusing-to-me conversational contexts, and losing track of my own bearings in that fray), so perhaps I am not truly of this opinion and am confused. But I have updated in the above direction multiple times over time, since it seems more robustly good in hindsight than my maps at the time (which are not the territory) would've more narrowly estimated.
because of the costs (mostly, upsetting people and then feeling a need to reply to people in attention-grabbing and confusing-to-me conversational contexts, and losing track of my own bearings in that fray)
If anyone has tricks or theories (or book recs or etc) for how to voice beliefs and reasons while incurring less of these costs, I'd appreciate pointers
all my friends in the labs have sophisticated justifications for their being there that seem plausibly true to me
How confident are you that these are genuinely held justifications, as opposed to rationalizations for seeking money and power?
Not super on topic, but: I would like to know more about what "power" is, and what wanting it feels like form the inside. Or I want to know about whatever the common actual motives are that the phrase "seeking power" is used to vaguely gesture at.
To begin a list: money, I get. I can use it to buy things. I sometimes almost need that, like panic (if I imagine not being able to make rent/basic stuff I need if I'm to keep my core plans and commitments), and I often like/want/appreciate that (for non-basics).
Convergent instrumental ability-to-do-stuff: I get that too. Sometimes that's money, or social slack, or health, or capacity in other ways.
Being in "the room where it happens," having one's life and actions be things that many care about: this seems ~intrinsically to be an attractor, for many? I feel it but less than some seem to. (Or maybe I experience it and don't realize I'm experiencing it?)
I suspect there are other list-items but I'm not sure what they are.
These are related to "the room", but aren't specifically about many caring, or are some of what is being tracked by the sense that many care:
"Making a dent in the universe"; shaping a large part of the world in your own image (metaphorically).
Dominance, in the sense of being able to command others / threaten / hurt / force to defer / control resources they need / make them serve your interests vs theirs. (As explicitly described in fiction: HPMOR ch.108 (search for "least annoying role"); turned up to 11, 1984.)
Rising in a hierarchy, being able to command people and resources and respect / deference. (This is more specific than, and overlaps, other items on the list; but is important as a common motivation (lost purpose?) for people to (be conditioned to?) follow far in excess of what it buys their other interests.)
Joining inner rings. (Really not real "power", but mistaken for it.) (Your last item made me think of this, since "the room where it happens" sounds so similar to me despite you talking about a different thing.)
I imagine that people being publicly angry at lab employees don't expect to be able to get them to quit, but that others will notice their anger and think something like "this intelligent person is very angry at this other person, perhaps this is a serious issue and I should spend some effort looking into this".
For most, the personal benefits from working at a frontier lab just massively outweigh any marginal personal increase in catastrophic risk their work may or may not incur on themselves. Some personal animosity from small groups is unlikely to shift that balance. I imagine most that do quit because of AI risk do so for altruistic rather than selfish reasons.
In order for lab employees to be incentivized to quit for selfish reasons, they would have to believe that staying may leave them worse off even in futures with positive AI outcomes. Imagine that governments declared that if we made it into a benign post-ASI future, they would hold Nuremberg-style trials for people that worked on AI during the transition, thoroughly investigating them individually and punishing (i.e. taking away their wealth and prestige) any that should have reasonably assumed that their work was increasing the risk of negative outcomes for humanity for their personal gain.
I am deeply uncertain about whether or not it is a good idea to get my friends to quit working at frontier AI labs. I am against people working at frontier AI labs in general, but perhaps downstream of this, all my friends in the labs have sophisticated justifications for their being there that seem plausibly true to me. More broadly, if this was not a painfully difficult question, the community wouldn't have split over it, and there are enough rationalists and EAs in the labs that it seems obvious to me that the community has split.
So I don't really get my friends to try to quit their jobs, as much as try to help them regularly orient to whether or not they should — and they generally humour me by treating this as a live question. I let them know frequently that I have reservations about their work, but also that I generally trust their judgement and care about them as people.
Here are my intuitions for what my friends are experiencing, while they are working at a frontier lab:
They value their job. Their job is giving them silly amounts of money and validation and prestige. Their job lets them work on a really important project with really smart people in comfortable settings. Their job gives them purpose. They believe that they are doing the right thing, and that they can more or less trust the leadership, or that even if they can't trust the leadership fully, every other player is less trustworthy.
They likely know that some people disagree with their choice to work at a frontier AI company and want them to feel shame about it. They actually know much more about the really cursed tradeoffs and clusterfucks that are happening inside their company than I do. But they will take pains to hide this ammunition from me, an outsider with reservations about their work — even if they haven't signed NDAs or have permissive ones, they will feel protective of the company, all the brilliant wonderful people that they know as colleagues, and their hopes for the work itself and the future that it might bring into being.
My friends are often scared, stressed, sleep-deprived, and in what I perceive to be an actively adversarial information environment. But I am also scared, stressed, and in an actively adversarial information environment (according to them). For my part I do try to prioritize good sleep hygiene.
There are people, including some that they look up to intellectually, who are in the process of repeatedly and loudly expressing disapproval at their choice to work at a frontier AI lab. Oliver Habryka and Eliezer Yudkowsky
yells atrespond disagreeably to their compatriots on X every time they choose to voice a pro-lab take. Ben Pace (disclosure: long Pace[1]) might be messaging them deranged stuff on Signal like "Good seeing you at the house party. By the way I think if we are living in a sane world you would be sentenced to ten thousand years in prison." High decouplers, man.I think it's good that the deeprats (shhhh just go with it) do this. There should be some voices of general disapproval, and there is much value in saying socially aggressive and unpleasant true things. And they sure do have the right temperament for doing that job, unlike me.
The failure mode I see in this strategy is that they might make the lab employees tense and guarded and defensive around them, and nudge them that much further into an echo chamber comprised only of their sympathetic colleagues inside the frontier labs, all with their own well reasoned and sophisticated justifications for staying. Would it be too galaxy brained to claim that this might make them less likely to quit a harmful job?
For my part, as I know about the existence of the deeprats and their general MO, I think it is reasonable for me to try a different tack. The risk of this tack is that not only will I feel like an enabler, I might actually be an enabler. In this way, I might be making them less likely to quit a plausibly harmful job.
I mean, the deal is, I can't make people quit their job. And maybe it doesn't make sense for them to quit their job — like I said, I am deeply unsure. But I think my friends appreciate the ear of an unsettled third party, who has reservations about their work but is open to hearing them out about their beliefs. I think it's bad for people who are conflicted, as my friends tend to be, to only be able to talk to people who have already chosen sides. I think it makes it harder for them to figure out what they believe in, and whether or not their beliefs are true.
I have never gotten a friend to quit working at a frontier AI lab, though one of them quit of their own volition after one year, to resume their work on technical alignment.
It is fortunate that I am actually uncertain about whether or not it is reasonable for people who care about x-risk to work inside frontier labs, because I think it would be somewhat dishonest to pretend that you are uncertain for the purposes of "helping" your friend "orient". How convenient for me.
..also, I intend to smooch him approximately fifty times today 🥰