I'm guessing this won't turn out to resolve your current disagreement, but I think the best articulation of this is probably in the Online Resource page: A Closer Look at Before And After.

From past discussions, it sounds like you think "the AIs are now capable of confidently taking over" is, like, <50% (at least < 60%?) likely to in practice be a substantially different an environment. 

I don't really get why. But, to be fair, on my end, I also don't really have much more gears underneath the hood of "obviously, it's way different to run tests and interventions on someone who isn't capable of confidently taking over, vs someone who is, because they just actually have the incentive to defect in the latter case and mostly don't in the former". It seems like there's just a brute difference in intuition I'm not sure what to do with?

(I agree there might be scenarios like "when the AI takeover is only 10% likely to work, it might try anyway, because it anticipates more powerful AIs coming later and now seems like it's best shot." That's a reason you might get a warning shot, but, not a reason that "it can actually just confidently takeover with like 95%+ likelihood" doesn't count as a significantly new environment once we actually get to that stage.)

Text of the relevant resource section, for reference

As mentioned in the chapter, the fundamental difficulty researchers face in AI is this:

You need to align an AI Before it is powerful enough and capable enough to kill you (or, separately, to resist being aligned). That alignment must then carry over to different conditions, the conditions After a superintelligence or set of superintelligences^* could kill you if they preferred to.

In other words: If you’re building a superintelligence, you need to align it without ever being able to thoroughly test your alignment techniques in the real conditions that matter, regardless of how “empirical” your work feels when working with systems that are not powerful enough to kill you.

This is not a standard that AI researchers, or engineers in almost any field, are used to.

We often hear complaints that we are asking for something unscientific, unmoored from empirical observation. In reply, we might suggest talking to the designers of the space probes we talked about in Chapter 10.

Nature is unfair, and sometimes it gives us a case where the environment that counts is not the environment in which we can test. Still, occasionally, engineers rise to the occasion and get it right on the first try, when armed with a solid understanding of what they’re doing — robust tools, strong predictive theories — something very clearly lacking in the field of AI.

The whole problem is that the AI you can safely test, without any failed tests ever killing you, is operating under a different regime than the AI (or the AI ecosystem) that needs to have already been tested, because if it’s misaligned, then everyone dies. The former AI, or system of AIs, does not correctly perceive itself as having a realistic option of killing everyone if it wants to. The latter AI, or system of AIs, does see that option.^†

Suppose that you were considering making your co-worker Bob the dictator of your country. You could try making him the mock dictator of your town first, to see if he abuses his power. But this, unfortunately, isn’t a very good test. “Order the army to intimidate the parliament and ‘oversee’ the next election” is a very different option from “abuse my mock power while being observed by townspeople (who can still beat me up and deny me the job).”

Given a sufficiently well-developed theory of cognition, you could try to read the AI’s mind and predict what cognitive state it would enter if it really did think it had the opportunity to take over.

And you could set up simulations (and try to spoof the AI’s internal sensations, and so on) in a way that your theory of cognition predicts would be very similar to the cognitive state the AI would enter once it really had the option to betray you.^‡

But the link between these states that you induce and observe in the lab, and the state where the AI actually has the option to betray you, depends fundamentally on your untested theory of cognition. An AI’s mind is liable to change quite a bit as it develops into a superintelligence!

If the AI creates new successor AIs that are smarter than it, those AIs’ internals are likely to differ from the internals of the AI you studied before. When you learn only from a mind Before, any application of that knowledge to the minds that come After routes through an untested theory of how minds change between the Before and the After.

Running the AI until it has the opportunity to betray you for real, in a way that’s hard to fake, is an empirical test of those theories in an environment that differs fundamentally from any lab setting.

Many a scientist (and many a programmer) knows that their theories of how a complicated system is going to work in a fundamentally new operating environment often don’t go well on the first try.^§ This is a research problem that calls for an “unfair” level of predictability, control, and theoretical insight, in a domain with unusually low levels of understanding — with all of our lives on the line if the experiment’s result disconfirms the engineers’ hopes.

This is why it seems overdetermined, from our perspective, that researchers should not rush ahead to push the frontier of AI as far as it can be pushed. This is a legitimately insane thing to attempt, and a legitimately insane thing for any government to let happen.

I'm not sure what Eliezer thinks, but I don't think it's true that "you cannot draw any useful lessons from [earlier] cases", and that seems like a strawman of the position. They make a bunch of analogies in the book, like you launch a rocket ship, and after it's left the ground, your ability to make adjustments is much lower; sure you can learn a bunch in simulation and test exercises and laboratory environments, but you are still crossing some gap (see p. ~163 in the book for full analogy). There are going to be things about the Real Deal deployment that you were not able to test for. One of those things for AI is that "try to take over" is a more serious strategy, somewhat tautologically because the book defines the gap as:

Before, the AI is not powerful enough to kill us all, nor capable enough to resist our attempts to change its goals. After, the artificial superintelligence must never try to kill us, because it would succeed. (p. 161)

I don't see where you are defusing this gap or making it nicely continuous such that we could iteratively test our alignment plans as we cross it. 

It seems like maybe you're just accepting that there is this one problem that we won't be able to get direct evidence about in advance, but you're optimistic that we will learn from our efforts to solve various other AI problems which will inform this problem. 

When you say "by studying which alignment methods scale and do not scale, we can obtain valuable information", my interpretation is that you're basically saying "by seeing how our alignment methods work on problems A, B, and C, we can obtain valuable information about how they will do on separate problem D". Is that right? 

Just to confirm, do you believe that at some point there will be AIs that could succeed at takeover if they tried? Sometimes I can't tell if the sticking point is that people don't actually believe in the second regime. 

I don't believe that there is going to be an alignment technique that works one day and completely fails after a 200K GPU 16 hour run. 

There are rumors that many capability techniques work well at a small scale but don't scale very well. I'm not sure this is well studied, but if it was, that would give us some evidence about this question. Another relevant result that comes to mind is reward hacking and Goodharting where often models look good when only a little optimization pressure is applied but then it's pretty easy to overoptimize as you scale u; as I think about these examples it actually seems like this phenomenon is pretty common? And sure, we can quibble about how much optimization pressure is applied in current RL vs. some unknown parallel scaling method, but it seems quite plausible that things will be different at scale and sometimes for the worse. 

I note that to my eyes, you appear to be straightforwardly accepting the need-to-generalize claim, and arguing for ability-to-generalize. Putting words in your mouth a little, what I see you saying is that, by the time we have a true loss-of-control-can-be-catastrophic moment where failure kills boazbarak, we have had enough failure recoveries on highly similar systems to be sure deadly-failure probability is indistinguishable from zero, that maximum-likely-failure-consequence is shrinking as fast or faster than model capability.

But current approaches don't seem to me to zero out the rate of failures above a certain level of catastrophicness. They're best seen as continuous in probability, not continuous in failure size.

"future superintelligent systems could not be obtained by merely scaling up current AIs, but rather this would require completely different approaches. However, if that is the case, this should update us to longer timelines, and cause us to consider development of the current paradigm less risky."

This doesn't feel like convincing reasoning to me. For one, there is also a third option, which is that both scaling up current methods (with small modifications) and paradigm shifts could lead us to superintelligence. To me, this seems intuitively to be the most likely situation. Also paradigm shifts could be around the corner at any point, any of the vast number of research directions could give us a big leap in efficiency for example at any point.

This is an excellent encapsulation of (I think) something different---the "fragility of value" issue: "formerly adequate levels of alignment can become inadequate when applied to a takeover-capable agent." I think the "generalization gap" issue is "those perfectly-generalizing alignment techniques must generalize perfectly on the first try". 

Attempting to deconfuse myself about how that works if it's "continuous" (someone has probably written the thing that would deconfuse me, but as an exercise): if AI power progress is "continuous" (which training is, but model-sequence isn't), it goes from "you definitely don't have to get it right at all to survive" to "you definitely get only one try to get it sufficiently right, if you want to survive," but by what path? In which of the terms "definitely," "one," and "sufficiently" is it moving continuously, if any?

I certainly don't think it's via the number of tries you get to survive! I struggle to imagine an AI where we all die if we fail to align it three times in a row. 

I don't put any stock in "sufficiently," either---I don't believe in a takeover-capable AI that's aligned enough to not work toward takeover, but which would work toward takeover if it were even more capable. (And even if one existed, it would have to eschew RSI and other instrumentally convergent things, else it would just count as a takeover-causing AI.)

It might be via the confidence of the statement. Now, I don't expect AIs to launch highly-contingent outright takeover attempts; if they're smart enough to have a reasonable chance of succeeding, I think they'll be self-aware enough to bide their time, suppress the development of rival AIs, and do instrumentally convergent stuff while seeming friendly. But there is some level of self-knowledge at which an AI will start down the path toward takeover (e.g., extricating itself, sabotaging rivals) and succeed with a probability that's very much neither 0 nor 1. Is this first, weakish, self-aware AI able to extricate itself? It depends! But I still expect the relevant band of AI capabilities here to be pretty narrow, and we get no guarantee it will exist at all. And we might skip over it with a fancy new model (if it was sufficiently immobilized during training or guarded its goals well). 

Of course, there's still a continuity in expectation: when training each more powerful model, it has some probability of being The Big One. But yeah, I more or less predict a Big One; I believe in an essential discontinuity arising here from a continuous process. The best analogy I can think of is how every exponential with r<1 dies out and every r>1 goes off to infinity. When you allow dynamic systems, you naturally get cuspy behavior. 

Thanks. Well, I don't have much time right now I'm afraid, but real quick I'll say: I basically agree that progress will be fairly continuous in the future... yet still fast, fast enough that e.g. I don't expect there to be any incidents where an AI system schemes against the company that created it, takes over its datacenter and/or persuades company leadership to trust it, and then gets up to further shenanigans before getting caught and shutdown. If I expected there to be things like that happening, especially repeatedly, before the first case of a scheming AI system that actually can succeed in taking over, then I'd think we'd have several opportunities to learn how to control AIs of that power level. (Even still this might not generalize to AIs of higher power levels, but maybe that's OK if the same arguments apply e.g. higher levels of capability would still lead to failed attempts several times before successful attempts)

Anyhow I'm curious to hear what your critique of AI 2027 would be, since I like to think of it as a continuous story. Or, just I'd like to hear some examples from you of what sorts of misaligned AI failures you expect to see, before the first unrecoverable failure, that are nevertheless quite similar to the first unrecoverable failure such that we'll probably be able to prevent the latter by studying the former.