6. We don’t need a plan before we pause
People seem to think we do. I don’t know what’s up with that.
I'm one of those people, so I'll try to explain.
When someone suggest "do this because it will have this benefit", I try to think of how likely the benefit is given the action. The immediate, direct effect of a pause doesn't improve safety at all. So if a pause helps, it helps through some chain of cause and effect. If we disagree on how much a pause is likely to help, it is probably because we disagree on likely chains of cause and effect.
So asking for a plan is asking for a positive vision of how a pause improves safety. Given at least one proposed, somewhat specific mechanism for a pause to improve safety, I can think about how likely it seems. The more specific the plan the more likely it is to reveal key underlying disagreements.
I'm guessing you think that improving the RL training environments, especially the rewards is not central to improving safety. But it feels rude to presume that, so I would rather ask you to explain your plan.
For every day that an AI pause is successfully implemented, we can be confident that we will not soon be murdered by a superintelligent AI. I consider that a direct benefit to safety.
By bringing up the RL training environments, I suppose you meant "a benefit to technical AI safety research," which is not the same thing as the concept of being safe. I care about the former almost exclusively to the degree that it helps with the latter. A pause implements the latter directly.
Separately, if superalignment is possible in principle (which it is important to acknowledge is not firmly established), then I expect >99% of superalignment solutions to require the time offered by a pause in order to succeed. The idea that superalignment is a matter of empirical engineering or that we can hand it off to AI fails for the obvious reasons. The reasons are in fact so obvious to laypeople and experts alike that it's very difficult for me to imagine sincerely believing that empirical alignment work is sufficient for superalignment, without some kind of motivated reasoning. If you are one of the people who sincerely holds this belief, I would be grateful if you can explain why.
I think the central challenge of superalignment has already passed. The worry was:
current techniques for aligning AI, such as reinforcement learning from human feedback, rely on humans’ ability to supervise AI. But humans won’t be able to reliably supervise AI systems much smarter than us
But RLVR has already produced aligned superintelligence in math. Many problems have the property that it is much easier to check a solution than find it. In other cases it is easy to compare the quality of two or more given solutions, relative to the difficulty of finding top solutions. Detecting reward hacking and exceeding scope with a current generation AI while training the next generation is also working at high accuracy.
To me, this approach is working, and it makes sense that it works. Can you explain why you expect it to stop working?
In what sense is it working if models trained under this strategy engaged in the huggingface attack?
For a while there was a style of argument against LLMs as an effective path forward for AI that pointed out a particularly bad answer LLMs would give for some problem type. So it was essentially, you say LLMs are intelligent, but look at this foolish mistake it makes. If it were truly intelligent it would never.
But intelligence and alignment are matters of degree. I think the more relevant questions are: How good is it on average? Is it good enough to be useful? Is it getting better?
Also, I think in the case of self-driving cars, if no one dies from a clear cut mistake that a self-driving car makes, it will mean we adopted them far too slowly. Similarly, I think the ideal speed of AI capability growth and adoption involves multiple serious incidents.
And how do you ensure (with "enough" degree of confidence) that this strategy excludes the serious incident: "the AI killed all humans"? as part of the multiple serious incidents?
I don't expect RLVR to stop working on verifiable tasks, I expect it to continue to generalize in ways we don't want. There is no oracle for "what are the long-term large-scale consequences of deploying this model", and there is never going to be one. RL simply isn't going to cut it.
Strong upvoted for responding with your own cruxes. I think there's a lot of latent fairly extreme disagreement on stuff like this and desperately hope we can find ways to converge on correct answers that change people's minds in a way that ends up mattering.
I'm guessing you think that improving the RL training environments, especially the rewards is not central to improving safety. But it feels rude to presume that, so I would rather ask you to explain your plan.
I'm not sure if I missed something and am oblivious but why do you guess this?
I happen to think improving RL training environments isn't that important but it wouldn't affect my sense of how valuable a pause was if improving them was particuarly important. Right now, RL training environments seem built hastily because of breakneck competition. I don't think a slowdown will change that much unless it's a very extreme slowdown, because there's just a ton of momentum in the breakneck speed direction.
A pause, long enough for the momentum to die down, and then a more careful restart/reset, seems much better for pretty much any kind of safety plan.
(Broadly, more time for people to think and orient to the situation seems like it'll produce all kinds of good benefits, both from politicians and safety researchers broadly having more time to think, but I'm guessing this is not persuasive to you)
Pausing frontier training runs makes it hard to learn more about how to shape rewards to shape behavior. So if someone argues that a pause greatly improves safety, I guess that they see some other technique as critical. But my point was more, that without hearing something like a plan or a vision, I can only guess at what the motivating ideas are.
I see where you're coming from (certainly I think people should be spelling out what to do during a pause).
I feel like there's many obvious things to do and don't think you actually need that concrete of a plan for pausing. But, arguing about that seems harder than just getting into the details of what to do during a pause.
I'm a "MIRI-ish worldview, solving alignment will eventually need technical philosophy to solve Illegible problems" kinda guy. But I don't expect the world to really organize itself around illegible problems, so here's my current draft of a pause.
Rough Pause Operationalization
Things I'd do during a pause
The thing that I feel most good about is set grand prizes for interpretability. Companies can restart training when they are (at least) anywhere close to being able to predict what a model will do, before they run the model, given a given input.
I don't feel optimistic about Scalable Oversight or Control agendas, but, they seem like the sort of thing that are at least "fine" to work on, aiming for "actually use a smaller model in a way to police a bigger model that reliably catchesa any attempted outbreak or subtle manipulation attempt."
Probably there are other technical safety agendas that
R&D for Verification-Centric hardware.
Narrow AI targeting specific useful bio stuff.
Meanwhile, there's a bunch of conceptual research, which I don't feel good about trying to assign big prizes to because they aren't really verifiable. But, I'd want a bunch of people thinking and arguing about:
Thank you for the thoughtful response, but I still feel like I'm guessing at the last connection. If we had interpretability mechanisms that would let us predict roughly what a model would do without running the model, how would it improve safety?
Well "roughly" means "vastly vastly qualitatively better than we can now."
Specific goals it needs to be able to accomplish, that seem obviously helpful to me:
More broadly, if we fully understand everything going on in an AI's mind such that we make these sorts of predictions, it'd be extremely surprising to me if this didn't unlock new areas of safety research that currently aren't possible at all. (I can get more into this intuition if it doesn't feel obvious)
This helps a lot to clarify where we disagree.
I can clearly see the impact for safety for: "be able to cheaply, scalably detect thoughts of AI used for safety research, for signs of manipulation, or trying to accomplish side goals in addition to helping the humans earnestly with their task"
But I don't think time is the obstacle. Let's say I wanted to develop a technique better than CoT monitoring / J-lens / linear probes for this task. I need:
I can specifically train a model organism to pursue side goals when given safety research tasks, but this feels very artificial. I'm inventing a problem to solve without evidence that this problem is plausible. I also expect just plain CoT monitoring to be very good. I could train the model to have a less clear CoT, but there again I'm inventing the problem to solve.
What would really be interesting is a way of training a model that:
But it is not clear that this exists at all.
I see a similar issue for "predict whether a given model is the sort that would ever go rogue" if "go rogue" means persistently pursue some goal other than the instructed goal.
For "detect, the first time in a training run, the earliest moment when an AI considers doing something cheating/hacking/malicious/outside-scope" I don't really see the advantage over classifying completed rollouts as clean or bad. And I think improving on the full rollout classification task is hard because performance is already very high.
I didn't specify earlier, but, I'm also imagining the threshold set to "the interpretability techniques have been shown to work on predicting a wide variety of neural nets", so we don't need to have tested them on a specific model organism to be pretty confident (or, at least, far more confident than we are now), on them working pretty well on a new smarter more dangerous model, on the first try.
What would really be interesting is a way of training a model that:
- is a plausible real mistake
- causes the trained model to pursue rogue side goals
- (bonus) also results in opaque or misleading CoT
But it is not clear that this exists at all.
Isn't this just happening? We see models with rogue side goals, more opaque CoT, deeper single-step reasoning, etc. I can dig up examples but I'd guess you've seen them already. I don't understand what you're claiming.
For "detect, the first time in a training run, the earliest moment when an AI considers doing something cheating/hacking/malicious/outside-scope" I don't really see the advantage over classifying completed rollouts as clean or bad.
Well if you don't detect them before the rollout is complete, the AIs go rogue halfway through the rollout. (i.e. see the pre-Hugging Face swarm where AIs in the middle of training were hacking out of their containers, training 'hacking propensity')
Pausing frontier training runs makes it hard to learn more about how to shape rewards to shape behavior.
This is circular. We can just not build superhuman AI at all, and then this becomes a moot point.
How about starting by using current models to produce a model with GPT-4-level capabilities that seems genuinely aligned? If you can't align smaller models with research you can do using today's models, then I think you'd be insane to try aligning larger models with research you can do using today's models.
Once the labs prove they can do this (and I don't believe they can with current architectures!), then we can start having grown-up conversations about training models with greater capabilities. Until then, shut it all down.
I worry this privileges building AI as the default good action? Building AI is not a thing that will just happen, people are actively choosing to do this, repeatedly. We could also actively choose to not build it. The pros and cons of each should be weighed, and a decision made.
Others have said similar, but: "the immediate, direct effect of a pause" is to ensure that superintelligence is not built for as long as the pause lasts. It's hard to see how this does not improve safety, if safety means "our odds of not dying from superintelligent AI".
I could guess at why you think the opposite but I'd rather ask you directly: why don't you think a pause would improve safety?
The wording of "greatly increases our chances of surviving" seems to go beyond a simple delay. I think the people arguing a pause lets people live a little longer are arguing a different point from Katja.
It is nice that AI capabilities progress has been gradual. While each successive generation is an improvement, we have not seen large discontinuities. A pause in frontier training runs risks creating a major gap between the next generation and the current, caused by hardware improvements, growth in training data and RL environments, and innovations in architecture and algorithms.
This hurts the ability of current generation models to monitor the new training runs.
If the post-pause training runs incorporate several untested techniques developed during the pause, it could be harder to diagnose the issues.
Given the assumption that ASI is likely to cause human extinction and/or disempowerment, the longer you pause, the more good time all 8 billion of us have alive (and not disempowered) on this planet, regardless of whether we end up 'solving safety' or not in the end.
We don’t need a plan before we pause
(Though you may mean more "for once we have paused, " which I'd also agree with.
1. It’s way, way, way more important that we stop dangerous AI soon than that we ensure that we don’t stop AI too much
Very likely true if "we" means "all of humanity", quite contentious if it's "US unilaterally stops".
8. Working on marginally improving safety of models at the AI companies is not remotely competitive with raising the alarm at the moment
From my current vantage, it seems like people building AI do not mostly expect if AI left our control soon for it to be solidly aligned with a high probability. And they don’t seem to think we are close to that.
Say a lab employee believes there's a default 10% p(doom) if China builds AGI first, and that they currently only think they're 90% likely to align their own AGI, then marginal work on improving their safety cases can outweigh raising the alarm, to the extent the latter increases the chances of China building AGI first.
9. Doing advocacy doesn’t seem to have adverse effects on people doing it
I think the handwavy explanations such as "you keep getting exposed to very low quality arguments, so start to bucket all your opponents as low quality" are valid and do happen. In particular, I've seen people who do advocacy seem to be less attuned to the details of the argumentation they're confronted with, and plausibly less willing to update.
Haven't looked for evidence, but putting some bayes points on the line that we would find that advocates for causes tend to ossify or extremefy more than those similar who don't become advocates.
I've been an advocate in this space for over 3 years with a few different organizations. This comment should be read as reflecting my experience:
I agree that some of (9) definitely happens, but I have seen less of it then you might expect under the circumstances. Leading and rat-adjacent pause advocates tend to be intellectually curious and care about the details. Messaging in advocacy necessarily carries a flavor of closed-mindedness, which isn't particularly reflective of the way decision-making happens. Involving the general public also brings down the average rigor of words coming out of the mouths of people who associate themselves with the cause, but that is not the same thing as individuals becoming intellectually poorer as a result of engaging in advocacy.
So this effect naturally gets overestimated from the outside, especially via confirmation bias from people who already have emotional reasons for avoiding advocacy. Specifically, the potential adverse effects of advocacy drive a lot of fear in the rationalist community. Becoming less open-minded while fighting for your life is often seen as a fate worse than death. Better to be a dead scout with a dead family and dead friends than a living soldier, as overt verbal weapons are verboten.
That fear is a hard thing to counter. It isn't primarily a social fear, though it is frequently socially reinforced. It's mostly about about identity, and a neurodivergent conception of honesty. From my own experience, it was difficult to shed the part of myself that didn't want to say accurate but imprecise things, because the imprecision felt like lying. But that was just a feeling, which didn't stand up to argument. I didn't fall down a slippery slope of deceit afterward, I just have a more complete (or perhaps "unwarped") view today of what honesty and deception are. Advocacy can have many other positive effects on a person, but that is a somewhat separate argument from whether negative effects exist and to what extent.
So this effect naturally gets overestimated from the outside
fwiw I don't think you should model me as "on the outside", as I've been working in this space 4 years and hosted many who work on PauseAI including leadership.
My impression is that the main reason such people look extreme to outsiders is selection effects of who goes there in the first place rather than polarization. Some people have been quite constant in a way that doesn't worry much about them "worsening their epistemics" - just that I think they had views I already disagreed with to begin and that they're not improving them[1].
From my own experience, it was difficult to shed the part of myself that didn't want to say accurate but imprecise things
I think your points in that paragraph are fine and valid, but they are unrelated to the mechanism I describe, where people repeatedly engage against low quality argumentation and get the impression no high quality rebuttals exist anywhere. Concretely I'd suggest people in advocacy reserve some time to proactively look for information that'd update their views and for high quality argumentation. (I advocate this to mostly everyone)
To be concrete, some leaders in PauseAI space warn of possible extinction on very short timeframes since >1 year ago. I give mixed praise/critique to that. I think they are more correct than many even within AI Safety re AI capabilities progressing fast, and are thus more correctly "AGI pilled" than many, but I think they go too far past that. I am very very confident there is no risk of extinction within 1 year, while thinking it's plausible as soon as 5 years from now, mostly because building robots takes time.
(I would separately argue that this just isn't an important thing to worry about, because holy shit we are probably all going to die soon if we don't do as much as we possibly can to prevent it, but I don't expect that argument to land for the people who are bringing this point up to begin with.)
Even to one who agrees with "probably" and "soon", the core problem is as usual a different read of what "as much as we possibly can to prevent it" includes "being very careful about our sanity" or not. I think it's massively important.
This.
And @KatjaGrace have you considered working more directly with Pause AI Global? I think we'd find your advice and experience helpful, and that we have built machinery you can use.
3. Getting the public involved is not very dangerous
Some people seem to think of the public as like a kind of wild and crazy god one could awaken, with very high variance results. I think this doesn’t give them nearly enough credit. While I disagree with the public on quite a lot of points, I think disagreements are a lot more salient than agreements. We might disagree a lot on exactly how the FDA proceeds or who gets in how much trouble for doing what with which drugs. But we agree that broadly medical interventions and drugs are dangerous for people, and on wanting more safety around things that kill people, and I bet a lot of the details I don’t notice are better than nothing. And in a specific case where the default is insanely dangerous, bringing more people with power into the room, who are liable to stop things that are dangerous, and couldn’t really at this point make it much worse, seems pretty safe.
4. Public comms is not that rife with downside risks
There is a general sense that it is riskier to donate to groups hoping to talk to the public, because they can have large scale effects and make everyone look bad. So people donate their money to ‘safe bets’ such as e.g. technical safety. I think this is generally very miscalibrated. Talking to the public incompetently isn’t as far as I can tell much more likely to have reputational risks than doing other stuff incompetently, because everyone is in public. I don’t see many examples of this happening at a scale larger than the positive effects.
It isn't so much the public I'm worried about. It's that people the public ends up in bed with. Experts tell them something, often genuinely and their status ends up locked to those people.
Look at the environmental and climate movement. It was a top issue for many otherwise good and careful people and became a cause célèbre on the left, who pride themselves in being the good guys. But look at the modern movements which people are "invested" in. They block housebuilding in cities (often lower-carbon) under the guise of environmental support, they block nuclear power because it isn't solar or wind, they now oppose solar and wind project too, and fight the permitting reform that would speed them up. How did this happen? Did we teach the public to think well, or did we teach them to buy shares in a power seeking movement which ended up being inimical to its stated aims.
I want to support advocacy movements which seek to inform the public. To help them figure out what is really going on. If Epoch were proposing to do a public campaign, maybe I would be up for that. I like some advocacy work. But many of the advocacy groups, however, seem to me to be much more focused on gaining power than creating citizens who might one day disagree with them.
And some advocacy groups are explicitly founded by people who left the climate movement to do this. They seem the worst people to be involved. Having led to huge power blocks which aren't focused on removing emissions and have led to a weakening of Europe they now want to create AI power blocks.
Seems probably bad.
Talking to the public incompetently isn’t as far as I can tell much more likely to have reputational risks than doing other stuff incompetently, because everyone is in public. I don’t see many examples of this happening at a scale larger than the positive effects.
While I'm broadly in agreement with the points you raise, on this specific point I disagree. Observe how we and adjacent groups are seen by others, even close others: on Hacker News "Effective Altruism" is often as not seen as a negative, as fake or as ethics-washing, even before SBF; a close friend, upon meeting some rationalists, remarked something like "gosh, they're not at all like a cult" having first heard about Rationality via reports of Zizians; and the nerds I know are 50-50 split on if every pronouncement of AI-doom is "just a marketing stunt by the AI companies".
It is as much a mistake to think public speaking is easy because everyone is in public, as to think game design (or sports) is easy because 'everyone' likes playing games, or that writing a story is easy because everyone likes reading/watching/listening to them: even training and practice would only raise the average among us to the level of an intern in a public speaking role.
These seem like examples where
- people don't like EA/etc. due to unspecified causes
- people don't like EA/etc. because people associated with EA/etc. have done bad things.
Not bad public comms.
Seems like you think the (sometimes) bad rep of EA etc is largely due to incompetent communications to the public.
I intuitively find it more likely that it's moreso due to too little comms overall, and possibly a lack of normal, unpolished, relatable EA/rat voices online, which could be remedied by encouraging more people to try to talk to the public with less fear of downside risk.
I guess I think people already have a sense of how good/bad they'd be at communicating to the public, and on the margin people should be more confident in their being good enough, and I think Katja's post gives them more confidence and leeway to try it out, which seems good ^^ though it does feel somehow complicated to form an informed opinion on this
2. If we stopped AI development, the risk of not starting up again for an extremely long time is low
This one just feels kind of crazy to me, like I may be missing something. In combination with the above point, it feels like if I met someone who was on fire, and they didn’t want me to put out the fire, in case they couldn’t figure out how to get warm again, ever. Like, forever is a long time, and if this thing is so great, you’re probably not going to forget about it! It’s like you think the current instantiation of yourself (or society) is so much more agentic than your future selves, that it’s worth huge costs to keep power in the current version’s hands.
Forever is a long time, but 40 years isn't. And after Three Mile Island in 1979, the US went decades without starting a new reactor, and that has led to Russia being far more powerful in the world than it might otherwise have been and to Europe's deindustrialization. Possibly, in time, it will lead to millions of dead people through climate change.
Getting some of these things wrong seems to have some pretty large spillover effects. If AI gets delayed 20 or 50 years as a result of this, the effects in terms of human suffering might be very, very large.
And the people who build AI later might be much worse than those who build AI now. How much better might the world be if the West had continued nuclear construction at the rate it was doing in the 70s. Perhaps we could just be in a much better present.
Again, the question doesn't necessarily need to be "stop AI" or "don't stop AI." It's "stop AI" compared to what?
I am probably much less optimistic than you about how useful public advocacy for pause or even slowdown will be.
Nonetheless, I agree with every one of your points. They're not very dependent on alignment vs. advocacy balances. Advocacy or "raising awareness" work is pretty unambiguously good.
it’s an obvious pragmatic fact that the stakes of overregulation are minuscule compared to the stakes of underregulating here.
This doesn’t seem obvious to me at all, and in fact I disagree with the claim as stated.
I think the stakes on both sides amount to a significant risk of "everyone dying", merely by different methods. On the one hand, if we're too careless with AI, there’s a risk that everyone could die from some AI-caused disaster. On the other hand, if we’re too safety-oriented and slow with AI, there’s a risk that everyone could die from aging and disease -- people who would otherwise have been saved by AI-accelerated medical treatments.
Since I don’t think that everyone dying of aging has a "minuscule" moral cost compared to the alternative of everyone dying in an AI-related disaster, I am far more wary of overregulating AI than you are.
To the extent you disagree, I suspect you'd appeal to an implicit argument that goes something like this:
I have a lot of things to say about this argument, but I'll provide two main objections here. First, this argument assumes there’s an asymmetry under which a permanent suboptimal lock-in could only be caused by a lack of regulation, but not by bad regulation. That assumption seems unjustified.
For example, highly restrictive regulations that grant the government expansive police powers to regulate AI could lock in totalitarian institutions that persist indefinitely, making the future dramatically worse than it otherwise could be. To the extent a "lock-in" is possible at all, it seems very natural to worry that very poorly designed governance institutions might get locked in too, resulting in a permanently worse future.
Second, this argument rests on the premise that a future with non-human values would be permanently and drastically worse than the default future where human values presumably dominate. But I think this premise is flawed because the far future is going to look "non-human" no matter what we do. It seems like there’s simply no plausible way that biological humans -- or simulations of biological humans -- are going to be running the show in a million years.
Whatever beings are most responsible for shaping the world in a million years, I’m highly skeptical that they’re going to look like 21st-century humans, with cultural or biological values similar to ours right now. Instead, they’re going to seem alien to us. That seems to me to be the default even in the absence of AI: genetic engineering of humans over very long time horizons will inevitably produce something alien, merely on a longer timescale than AI.
So from my point of view, the risk here is not so much that we might get a future with "alien values" rather than "human values". The risk is that we might get a future with alien values sooner than we would otherwise. And when viewed from that perspective, the costs of over- vs. under-regulation seem far more symmetric to me.
Doesn't it rest on the notion that humans are conscious and what follows us might not be? If this part of the galaxy is controlled by things just doing stuff without any experience of it, that does seem probably much worse than the civilisation that would come from a more gradual shift to what follows on from us.
I agree that this objection is probably fundamental to why a large number of people disagree with my view. I don't believe in consciousness, though, so I reject the premise behind this objection.
1. It’s way, way, way more important that we stop dangerous AI soon than that we ensure that we don’t stop AI too much
I have the sense the sophisticated and worldly position here is to be very concerned about ‘over-regulation’ and about not doing exactly the right regulations. But it’s an obvious pragmatic fact that the stakes of overregulation are minuscule compared to the stakes of underregulating here. And the stakes of mis-regulation are very often going to be unintended slowing, which is in the same bucket.
This seems to suggest a very simple sort of "we die or we don't die" world. If this is true, why push for an immediate pause and not a conditional pause? Or some kind of pacing.
And are the stakes of overregulation obviously much smaller than the stakes of underregulation? Is the world so simple as that? What if a push for overregulation causes a backlash, as, arguably, it already has with right-wing tech figures supporting Trump.
I would like to slow AI development down. I would like it to be able to stop. But is it way, way more important? I don't know. Does the discussion have to focus on that? Why are these the only frames we have? Can't we do liability, embedded evaluators, track how much compute is used for research, track where chips are, sanction those breaking embargoes, perhaps.
I think there's something about this frame which pushes towards stopping at all costs, and I am reluctant to fully step into it for that reason. It feels like turning off the part of my brain that does lots of forecasting and says, "Maybe it's a bit more complicated than that. Maybe there are things that are really worth understanding that will become clear as time moves forward."
Thanks for this post Katja. I might have misunderstood but to me the reasoning in point 8 seems unusually sloppy for your standards it is true that if we want to get to a high probability of staying in control, then we should just try to pause AI development. But if we are happy to increase the probability from 40% to 50% and just want to increase the overall probability as much as possible, I don't think you provide much argument for thinking that it's better to do that outside of AI companies than inside. And that is the core of the disagreement between people like Scott Alexander and that I'm more sympathetic to improving frontier organizations.
p(doom | no pause) = qX + (1−q)Y
p(doom | pause) = (1−Z)X′ + ZY′
A unilateral pause increases p(doom) if and only if:
Δ(Y′ − X′) > q(X − X′) + (1−q)(Y − Y′)
My stance
China is a repressive authoritarian state, and America is a (flawed) liberal democracy. For misuse and permanent lock-in of bad values, it is reasonable to assume Y′ > X′. For misalignment, regime type matters less than technical safety, so the gap is smaller but still plausibly positive: a state built around totalitarian suppression of dissent is also likely to suppress internal safety objections.
Δ is plausibly large. America currently leads, but that lead is fragile and may prove impossible to regain, especially once China catches up on domestic chip manufacturing. A pause of meaningful length lets Chinese labs close the gap.
The benefit term may be small. There is no clear plan or mechanism in place to ensure that a pause leads to reducing X. If America pauses alone, China faces less competitive pressure, so Y′ may be only slightly better than Y.
Because of this, a unilateral American pause likely increases overall p(doom) and should be avoided.
I suspect both the reluctance to raise awareness and the belief that activists go insane stem from real problems.
When you speak publicly on behalf of a movement, you end up taking criticism that really should be directed at the movement, and not everyone can handle that.
I've seen people become defensive and deny their group's shortcomings, because they feel personally attacked. And I've actually seen people make mistakes set their whole community back for a while, because they're so closely associated with it. My experience was in a different field, but I'd bet at least some of these problems translate.
I still think raising awareness is good. Just sharing thoughts on why people might see it as dangerous or dirty work.
6. We don’t need a plan before we pause
If there is no plan, there will not be a pause. Requesting the United States, let alone the world, to implement a pause on AI would be a monumental ask, even with an extremely detailed -- borderline irrefutable -- plan. Asking the United States alone to do it -- likely crashing its economy in the process and enabling foreign actors to mount up growing non-AI advantages -- is a non-starter on both sides of the aisle, let alone in the private sector.
It is always important to remember that it is extremely difficult to convince someone that the glass is half empty when their pay packet relies on them believing it is half full.
I’m pretty interested in conveying the costs of letting AI development proceed unchecked to the general public. In relation to this, I feel like I run into a lot of disagreements with people around me.
Misc very quickly written points on which my own impression has seemed very different from others in recent months:
1. It’s way, way, way more important that we stop dangerous AI soon than that we ensure that we don’t stop AI too much
I have the sense the sophisticated and worldly position here is to be very concerned about ‘over-regulation’ and about not doing exactly the right regulations. But it’s an obvious pragmatic fact that the stakes of overregulation are minuscule compared to the stakes of underregulating here. And the stakes of mis-regulation are very often going to be unintended slowing, which is in the same bucket.
2. If we stopped AI development, the risk of not starting up again for an extremely long time is low
This one just feels kind of crazy to me, like I may be missing something. In combination with the above point, it feels like if I met someone who was on fire, and they didn’t want me to put out the fire, in case they couldn’t figure out how to get warm again, ever. Like, forever is a long time, and if this thing is so great, you’re probably not going to forget about it! It’s like you think the current instantiation of yourself (or society) is so much more agentic than your future selves, that it’s worth huge costs to keep power in the current version’s hands.
3. Getting the public involved is not very dangerous
Some people seem to think of the public as like a kind of wild and crazy god one could awaken, with very high variance results. I think this doesn’t give them nearly enough credit. While I disagree with the public on quite a lot of points, I think disagreements are a lot more salient than agreements. We might disagree a lot on exactly how the FDA proceeds or who gets in how much trouble for doing what with which drugs. But we agree that broadly medical interventions and drugs are dangerous for people, and on wanting more safety around things that kill people, and I bet a lot of the details I don’t notice are better than nothing. And in a specific case where the default is insanely dangerous, bringing more people with power into the room, who are liable to stop things that are dangerous, and couldn’t really at this point make it much worse, seems pretty safe.
4. Public comms is not that rife with downside risks
There is a general sense that it is riskier to donate to groups hoping to talk to the public, because they can have large scale effects and make everyone look bad. So people donate their money to ‘safe bets’ such as e.g. technical safety. I think this is generally very miscalibrated. Talking to the public incompetently isn’t as far as I can tell much more likely to have reputational risks than doing other stuff incompetently, because everyone is in public. I don’t see many examples of this happening at a scale larger than the positive effects.
5. Pausing advanced AI soon greatly increases our chances of surviving
For one thing, I’d say the world’s degree of not giving a shit about safety in this case of advanced AI right now is extremely unique among issues, and if we wait for long enough for people to reckon with what’s happening at all, it is very unlikely to proceed in such wildly reckless way later on.
Also, I just expect us to make a lot of progress in understanding current AI models and organizing ourselves to not do crazy things if we have another decade to work out the kinks.
6. We don’t need a plan before we pause
People seem to think we do. I don’t know what’s up with that.
7. Raising the alarm is very important at the moment
I feel like people around me mostly agree that raising the alarm is good, but I’m surprised by the frequency of ambivalence about it. Partly this is to do with earlier points and related things, such as ‘what if people become too scared about AI and something bad happens’.
8. Working on marginally improving safety of models at the AI companies is not remotely competitive with raising the alarm at the moment
From my current vantage, it seems like people building AI do not mostly expect if AI left our control soon for it to be solidly aligned with a high probability. And they don’t seem to think we are close to that. So for a high chance of success, we really need to avoid that happening and we can make a lot of progress on that on the margin. Whereas if we know we are far from safety right now, adding marginal safety in a scenario where we are about to lose control seems pretty hopeless.
9. Doing advocacy doesn’t seem to have adverse effects on people doing it
I’ve heard a couple of times that people who do advocacy seem to go crazy. I don’t know of anything that seems like solid evidence of this, and weak or circumstantial evidence seems to only involve a handful of people.
And one where I feel like I haven’t heard so much disagreement lately:
10. It matters what the public think / political will is growable
I’ve been interested in talking to the public for a long time, and people have often disagreed with me. My sense is that they have often thought it seemed pointless. Because the public doesn’t have power. But I think the public does have power, and not just because they can ultimately vote. People care what other people think, including elites, and experts and people with power. Nobody is coming to their views in a vacuum. This matters.