Acknowledgements

The vision here was midwifed originally in the wild and gentle radiance that is Abram's company (though essentially none of the content is explicitly his). 

The PIBBSS-spirit has been infused in this work from before it began (may it infuse us all), as have meetings with the Agent Foundations team at MIRI over the past ~2 years. 

More recently, everyone who has been loving the High Actuation project into form (very often spontaneously and without being encumbered by self-consciousness of this fact):^[1] individuals include Steve Petersen, Mateusz Baginski, Aditya Prasad, Harmony, TJ, Chris Lakin; the AISC 2024 team, Murray Buchanan, Matt Farr, Arpan Agrawal, Adam, Ryan, Quinn; various people from Topos, ALIFE, MAPLE, MATS, EA Bangalore. Published while at CEEALAR.

Disclaimers

• Very occasionally there are small remarks/questions from a remarkable human named Steve, since this and the next two posts are an edited transcript of me giving him a talk. I left them in to retain the conversational tone. Steve has also consistently been a fantastic ground for this channeling.
• I use the term “artefact” a fair amount in this sequence. Unfortunately for you and me, Anthropic also recently started using "artifact" in a different way. I'm using "artefact" in the common sense of the word. The British spelling should help remind of the distinction.

Taking Intelligence Seriously

Sahil:

I gave a talk recently, at an EA event just two days ago, where I made some quick slides (on the day of the talk, so not nearly as tidy as I’d like) and attempted to walk through this so-called “live theory”. (Alternative terms include “adaptive theory” or “fluid theory”; where the theories themselves are imbued with some intelligence.)

Maybe I can give you that talk. I'm not sure how much of what I was saying there will be present now, but I can try. What do you think? I think it'll take about 15 minutes. Yeah?

Steve: 

Cool. 

Sahil:

Okay, let me give you a version of this talk that's very abbreviated.

So, the title I’m sure already makes sense to you, Steve. I don't know if this is something that you know, but I prefer the word “adaptivity” over intelligence. I'm fine with using “intelligence” for this talk, but really, when I'm thinking of AI and LLMs and “live” (as you’ll see later), I'm thinking, in part, of adaptive. And I think that connotes much more of the relevant phenomena, and much less controversially. 

It’s also less distractingly “foundational”, in the sense of endless questions on “what intelligence means”.

Failing to Take Intelligence Seriously

Right. So, I want to say there are two ways to fail to take intelligence, or adaptivity, seriously.

One is, you know, the classic case, of people ignoring existential risk from artificial intelligence. The old “well, it's just a computer, just software. What's the big deal? We can turn it off.” We all know the story there. In many ways, this particular failure-of-imagination is much less pronounced today.

But, I say, a dual failure-of-imagination is true today even among the “cognoscenti”, where we ignore intelligence by ignoring opportunities from moderately capable mindlike entities at scale. I'll go over this sentence slower in the next slide.

For now: there are two ways to not meet reality. 

On the left of the slide is “nothing will change”. The same “classic” case of “yeah, what's the big deal? It's just software.”

On the right, it's the total singularity, of extreme unknowable super-intelligence. In fact, the phrase “technological singularity”, IIRC, was coined by Vernor Vinge to mark the point that we can't predict beyond. So, it's also a way to be mind-killed. Even with whatever in-the-limit proxies we have for this, we make various simplifications that are not “approximately” useful; they don’t decay gracefully. (Indeed, this is how the “high-actuation spaces” project document starts.)

All of the richness of reality: that’s in the middle.

Steve:

I think that makes sense. I like how there are both ways to avoid looking at the kind of intelligence in between, given slow takeoff.

Sahil:

Yeah, cool. In the event of a “slow takeoff”, sure.^[2]

Opportunities from Moderate Intelligence at Scale

Okay, so, going over the sentence slowly:

1. I mean opportunities, so the focus for today is less about risks. We're talking about reframing risks tomorrow anyway, in the series of conversations we're having. However, conversation about risk will end up here as well, because it is an inextricable aspect of the opportunity frame. Indeed, “live theory” is about (research methodological) opportunities meeting risks (of infrastructural insensitivity towards beings).^[3]
2. “Moderate”, as in: intelligence that is not super intelligent or singularity-level intelligent, but kind of intelligent, the way LLMs seem kind of intelligent, and somewhat beyond.^[2]
3. “Intelligence” meaning, really, adaptivity again.
4. And at scale, meaning: everywhere, cheap, abundant, etc. The analogy I tend to use: attempts to talk about video calls, or even, say, remote teaching when all we currently have to extrapolate from is the telegraph/telegram. With the technology of the telegraph, people can tell that being able to send messages over large distances seems exciting. But they're not really thinking “oh, in the future, some jobs will entirely be dependent on being able to take video-calling tech for granted. Schoolkids might do the entirety of their schooling using this tech.” They just didn't think of that, for the most part. Even if some people had ideas, few were working towards orienting to that future in a way that would enable better equilibria. They failed to take it seriously.
   
   Video calling is in a way the same type of technology as telegraph: the ability to send messages. But with incredibly reduced cost, latency plus increased adoption, bandwidth, fidelity. This allowed for remote work and video calls and the whole shebang that’s allowing us to have this conversation right now.
   
   And so the question is: what happens when we have much lower latency and moderately more intelligence, much lower costs, with adaptivity tightly integrated in our infrastructure?^[4] Just like we have Wi-Fi in everything now.

And notice that this is not extrapolation that goes only moderately far. That is, just because I'm talking about “moderate” intelligence does not mean this extrapolation is not about the real crazy future up ahead. Only, this is AI-infrastructural extrapolation, not AI-singleton extrapolation (or even what’s connoted by “multipolar”, usually, IME). It’s neglected because it is usually harder to think about, than a relatively centralized thing we can hold in attention one at a time. 

This frame also naturally engages more with the warning carried in “attitudes to tech overestimate in the short run, underestimate in the long run.”

So to repeat, and this is important for sensemaking this: I am doing extrapolation that will venture far. What follows is simultaneously very obvious and very weird. In fact, that combination is what makes it work at all, as you’ll see.

But that’s also a double-edged sword. Instead of it being sensible (obvious) and exciting (weird), the perspective here might seem redundant or boring (too obvious) and irrelevant or scary (too weird). 

Hopefully, I will:
a) avoid its obviousness being rounded off to “ah right, live theory is another word for autoformalization” and
b) bring its weirdness closer to digestibility. To quote Bostrom in Deep Utopia, “if it isn’t radical, it isn’t realistic.”

So even though some might classify this series as being about "automating^[5] alignment research", it tastes nothing like unfleshed mundane trendlines^[6] or spectacular terror that are mixed together, for example, in Leopold Aschenbrenner’s “AGI from automated research by 2027”. 

Again, this isn't to say that there aren't some serious risks, only that they might look very different (which view will be elaborated in an upcoming post). ^[7]

Live Interfaces

This slide was ‘Live UI’ (don't bother trying to read the image): what happens to infrastructure and interfaces, generally, when you can “do things that don’t scale”, at scale. People don’t seem to update hard enough, based on their reactions at Sora etc, on what the future will take for granted. 

What is possible, when all this is fast, cheap, abundant, reliable, sensitive? Live UI seeks to chart this out. The six pillars, without much explanation for now, are:

• Tailormade media (eg. instead of writing a post and then distributing it, you can distribute a post-prompt that can take into account the user’s preferences and context and become the relevant post in interaction with the user.)
• Live interoperation (eg.^[8] tailormade movies do not imply loneliness of the viewer, because you’ll also have tailormade interoperation attentively “translating” insights you had watching your movie to remarks relevant to your friend’s movie.) 
• Live differential privacy (eg. automatically^[5] and seamlessly replace your friends in a livestream who don’t want their info up, with randomly generated faces that capture the vibe without any possibility of inference to their details, handled by an AI that understands your circumstances and privacy preferences informally, with occasional oversight.)
• Recording & representatives (eg. record your data to allow for better tailoring to you, and even stream it out, to the extent that your privacy is covered by previous pillar, and credit and relevance is covered by the next two pillars.)
• Live storage (eg. personal knowledge management engines like Roam complemented with mental-context management that cycle your mental context with your own important notes and various data feeds, sensitively refreshed to match your current momentum/motivations.)
• Telic reliability & live attribution [not pictured above] (eg. the data/prompts/wishes you livestream are associated with a cycling schedule that the reader likes, in the reader’s live storage, which helps automatically^[5] sense the contextual reputation assigned to your creativity.)

Steve:

Am I supposed to be following all of that?

Sahil:

Definitely not, it's just a trailer of sorts. I've included only a relatively accessible example each (it gets way weirder), but there are volumes to say for the pillars, especially to gesture at how it all works together. (Btw: reach out if you're interested to know more or work together!)

A bit more, before we move on though. 

Nearly all of the above is about sensitivity and integration^[9] as we gain increasing choice in adaptive constructability at scale. 

The above could, instead, already sound nauseating, atomizing, and terrifying. A good time, then, to meet the key concern of the High Actuation agenda: the applied metaphysics of cultivating wise and lasting friendship in a reality full of constructs. 

High Actuation is the research context (for both Live UI & Live Theory, among several other tracks), where the physical world becomes more and more mindlike through increased and ubiquitous adaptivity, in the process challenging a lot of (dualistic^[10]) assumptions about mind and matter, and how to go about reasoning about them.^[11]

But yeah, don't worry about this interface-6-pillars stuff above. I'm going to talk about what I’ll be focusing on building tools (and hiring!) for, in the coming months: intelligent science infrastructure.

Live Theory (preface)

So the boring way to think about intelligent science infrastructure is to say “AI will do the science and math automatically.”

(What does it really mean, to say “automatically”? We’ll get to that.)

First, a succinct unrolling of the whole vision. A series of one-liners follow, with increasing resolution on the part of this dream that matters. The italics signify the emphasis in each stage of clarification.

Live theory is...

• Adaptive theories

(Here "theories" are only one kind of distributable artefact in research to think about.)

But more importantly, the vision is...

• Artefacts with adaptivity beyond formal variables/parameters

(Here "artefacts" includes papers, theories, explainers, code, interfaces, laws^[12], norms etc.)

But more importantly, the vision is...

• Protocols for exchange of  “postformal” adaptive artefacts that you can take for granted, and focus on production/consumption instead of the interface.

(Here "protocols" don’t need to be a fixed/formal protocol or platform either!)

But really, the vision is...

• Lending spirit to an ecosystem of research that can competently caretake the widescale-exchange of postformal adaptive artefacts
  • ...so that we collectively refine sensitivity towards beings, in the AI-replete infrastructure we seem to be heading towards.

(Here “lending spirit” being the crucial activity that allows for spacious resolution towards better equilibria.)

Navigation

Some navigation before the longer conversation around this: there are four gates that we'll have to pass through, as you see on the slide below, to come to terms with the proposal.

Each of the four gates has been invariably challenging to convey and invite people through (although it is satisfying to witness people's fluency after a few runs through them):

1. Possible (what Live Theory is and whether it is even conceivable/tractable)
2. Bearable (how, paradoxically, Live Theory has always been true, and whether we can stand it being true already)
3. Desirable (what new modes open up when we incorporate Live Theory-like adaptivity)
4. Necessary (Live Theory's suggestions for threat models from extremely adaptive AI, like in Deep Deceptiveness and Robust Agent Agnostic Processes. ^[13])

I'm going to walk through these gates, and conclude, and that will be the talk.

This decomposition into gates should hopefully make it easier to critique the whole thing—eg. “X aspect is undesirable because…” vs “X aspect is impossible because…”. The two obviously deserve very different kinds of responses.

I’m offering the talk this way for many reasons, but to say a bit more. Most of the work is in a) noticing the circumrational water of mathematics as it is today (which can be too obvious for mathematicians and too unbearable for math-groupies respectively) and b) connecting it to mathematics as it might become in the near future (which can seem too bizarre or undesirable, if you don’t notice its importance in mathematics as it is today). When new paradigms start being Steam’d, they often have to pull off a similar straddling of the familiar and the unfamiliar. Not too different from the ordeal of hitting the edges of one’s developmental stage… but at a sociotechnical/civilizational level.

If making it easy for you to respond and critique were the only goal of the gates, they would have been set out in a tidier decomposition. However, in tandem, I'm using the gates-structure to construct a “natural walk”, a narrative arc, through the details of Live Theory. This polytely has some tradeoffs (such as the Bearable and Desirable gates not quite disentangling), but I think it works! Let me know.

The next post will cover the first two gates. A teaser slide for the first one follows.

Live Theory (teaser)

Possibility

And a teaser of two questions I will start the next post with, but also include now to give you time to actually think:

• Q. Mathematical (or theoretical) generalization seems to involve creating a parameter or variable, where "specialization" happens by substituting with a value later. Is there an alternative?
• Q. What is generalization really for? What does it offer you?

1. ^{^}

   Challenging and supporting, especially through the frustration of freezing this written artefact before I can avail the spaciousness of the fluidic era.

2. ^{^}

   This will be elaborated in the upcoming post on risks. “Slow takeoff” is an approximation that will have to do for now.

3. ^{^}

   Generally, it is a bit suspect to me, for an upcoming AI safety org (in this time of AI safety org explosion) to have, say, a 10 year timeline premise without anticipating and incorporating possibilities of AI transforming your (research) methodology 3-5 years from now. If you expect things to move quickly, why are you ignoring that things will move quickly? If you expect more wish-fulfilling devices to populate the world (in the meanwhile, before catastrophe), why aren't you wishing more, and prudently? An “opportunity model” is as indispensable as a threat model.
   
   (In fact, "research methodological IDA" is not a bad summary of live theory, if you brush under the rug all the ontological shifts involved.)

4. ^{^}

   TJ comments:

   I'd guess you'd want to be more explicit here that what you mean is that "human cognition, thought, and intellectual enterprise is itself going to  go through radical transitions, and will reshape the human experience." It feels like you're implying this here, but is left as a silent gesture for the reader.

5. ^{^}

   The word “automation” does not distinguish numb-scripted-dissociated from responsive-intelligent-flowing. It usually brings to mind the former. So I avoid using it. More on this in the next post, but for now, a word might be “competence”. When you’re really in flow while dancing, you're not thinking. But it is the opposite of “automation”. If anything, you're more attuned, more sensitive. 

   This isn't “high-level” abstraction either. The high-low dichotomy is the good-old-fashioned way of looking at generalization, and does not apply with expanded sensitivity, as we'll see in this sequence.

   The relevance of the choice "live" will hopefully also become clearer as we go. Meant to connote biological rather than machine metaphors, dialectics rather than deliverables, intentional stance, though not necessarily patienthood.

6. ^{^}

   It's not that people predict "no fundamentally new ways of looking at the world will occur" and therefore decide to exclude them in their stories. I think people exclude ontological shifts because it's very hard to anticipate an ontological shift. 
   
   (If you disagree, I'd ask the fundamental question of implied invisible work: what have you rejected?)

7. ^{^}

   Matt adds:

   It might be worth stating more about these [risks] from the outset. Just to prevent anyone thinking you're some kind of e/acc full steamer techbro who isn't interested in safety

   Indeed. The Necessary gate will cover much relevance to threat models. Apart from that, expect much much more on "infrastructural insensitivity" to articulate further vectors of risk.

8. ^{^}

   Abram helpfully adds:

   I find the meeting example [see next quote block] much more compelling, where people get dynamically crafted versions of the meeting which can catch them up on the most important things they missed as a result of coming in late -- someone uses the phrase "lightbulb moment" in a context-dependent way, so the live interop unpacks the context for those who missed it. 
   
   I think the reason I like this better is that meetings invoke a more utilitarian mindset where we evaluate the usefulness of tools. Movies invoke a warm friendly mindset where we are more prone to horror over the seeming non-togetherness of the described scenario. If you want to shock people with the idea of such severely mediated reality, fine. But I think the meeting example will provoke less feeling of shock and undesirability, if you prefer that.

   And so here is my meeting example pasted raw, which might be more palatable or more triggering:

   So imagine you join a meeting on Zoom, a little late. AI summarizers can "catch you up". Instead of immediately chatting, you peruse the transcript/read the summary.

   Now that's great, but a little boring. Ideally you'd have "immersive summary", a mini-meeting that you get to see live, a generated video summary. You watch that, and then join the real meeting.

   Then you seamlessly transfer to the "real" meeting... except when Alice references a detail that wasn't given enough attention in the summary-virtual-meeting you initially watched to catch up. So before you get confused as Alice starts using the details assuming you were in the meeting, the AI interop again pulls you back into a simulation where generated Alice says "so, just to recall, some details I want to add" before dropping you back into reality.

   And in fact, your replica will already have joined the meeting even though you were 10 minutes late. And you watch it, and might say "oh, no, that's not what I would have said"... and so, once you say that, everyone else will be put in a simulated meeting, where your replica says "I know I said earlier that X, but no, [I take that back/actually meant/wanted to say] ..." so everyone is again caught up.

   Rather than being blocky or jarring, imagine a perfect stabilizer, like in a drone or in video-correction. It's uncanny, that despite all the noises, rapid responses can correct destabilizations quite magically and beautifully into a coherence.

   This again blurs "construction" and "reality", doing such a fantastically better job that you will only occasionally need to rebel against this matrix. The real-constructed dichotomy makes less and less sense.

9. ^{^}

   NB. this is not brute, ham-fisted merging, for those worried. See also: Unity and diversity.

10. ^{^}

    also: centralization-heavy, preformationist, foundationalist, control-oriented

11. ^{^}

    This is a very expensive delivery mechanism for a koan/pointing-out instruction, to let go of the static-machine mythos, but there you go.

12. ^{^}

    Live Governance post coming up soon.

13. ^{^}

    See also this proposal for the commonalities in the two posts and limitations of existing approaches.