I beileve this is the only way to design an AI whose actions we still have confidence in the desirability of, even once the AI is out of our hands and is augmenting itself to unfathomable capabilities.

I think unleashing AI in approximately the present world, whose infrastructural and systemic vulnerabilities I gestured at here, in the "Dealing with unaligned competition" section (in short: no permeating trust systems that follow the money, unconstrained "reach-anywhere" internet architecture, information massively accumulated and centralised in the datacenters of few big corporations), would be reckless anyway, even if we believed we have the design you are talking about. Just the lingering uncertainty about our own conclusions about the soundness of this design and "defence in depth" thinking aka security mindset tells us that we should also prepare the global infrastructure and the global incentive system for the appearance of misaligned entities (or, at least try to prepare).

See also later here, where I hypothesise that such infrastructure and incentives shift won't become feasible at least until the creation of an "alignment MVP".

I believe it needs to get out of our hands and augment itself to unfathomable capabilities, in order for it to save the world.

Why? Contra position: "We don’t need AGI for an amazing future". (I don't say that I endorse it because I didn't read it. I just point out that such a position exists.)

We believe that the correct action to take is to continue working on the hard problem of alignment, and we believe that our research agenda is the most promising path to solve it. this is the foundational motivation for the creation of our research organization.

I'm somewhat contra the idea that there is a special "alignment problem" that remains glaringly unsolved. I tried to express it in the post "For alignment, we should simultaneously use multiple theories of cognition and value" and in this conversation with Ryan Kidd. Sure, there are a lot of engineering and sometimes scientific problems to solve, and the strategic landscape with many actors willing to develop AGI in open-source without any regard for alignment at all and releasing it into the world is very problematic. The properly secured global infrastructure and the right economic incentives and the right systems of governance are also not in place. But I would say that even cooking up a number of existing approaches, from cooperative RL and linguistic feedback to Active Inference and shard theory, could already work (again, conditioned on the fact that the right systemic incentives are instituted), without any new fundamental breakthroughs either in the science of intelligence, alignment, ML/DL, or game theory.

This avoids spending lots of time getting confused about concepts that are confusing because they were the wrong thing to think about all along, such as "what is the shape of human values?" or "what does GPT4 want?"

These sound like exactly the sort of questions I'm most interested in answering. We live in a world of minds that have values and want things, and we are trying to prevent the creation of a mind that would be extremely dangerous to that world. These kind of questions feel to me like they tend to ground us to reality.

In this post, as well as your other posts, you use the word "goal" a lot, as well as related words, phrases, and ideas: "target", "outcomes", "alignment ultimately is about making sure that the first SGCA pursues desirable goal", the idea of backchaining, "save the world" (this last one, in particular, implies that the world can be "saved", like in a movie, that implies some finitude of the story).

I think this is not the best view of the world. I think this view misses the latest developments in the physics of evolution and regulative development, evolutionary game theory, open-endedness (including in RL: cf. "General intelligence requires rethinking exploration"), and relational ethics. All these developments inform a more system-based and process-based (processes are behaviours of systems and games played by systems) view. Under this view, goal alignment is secondary to (methodological and scientific) discipline/competency/skill/praxis/virtue alignment.

Agent foundations/formal-goal alignment is not fundamentally about doing math or being theoretical or thinking abstractly or proving things. Agent foundations/formal-goal alignment is about building a coherent target which is fully made of math — not of human words with unspecified meaning — and figuring out a way to make that target maximized by AI. Formal-goal alignment is about building a fully formalized goal, not about going about things in a "formal" manner.

This is a confusing passage because what you describe is basically building a satisfactory scientific theory of intelligence (at least, of a specific kind or architecture). As a scientific process, this is about "doing math", "being theoretical" and "thinking abstractly", etc. Then the scientific theory should be turned into (or developed in parallel with) an accompanying engineering theory for the design of AI, its training data curation, training procedure, post-training monitoring, interpretability, and alignment protocols, etc. Neither the first nor the second part of this R&D process, even if discernible from each other, are not "fundamental", but they are both essential.

Current AI technologies are not strong agents pursuing a coherent goal (SGCA). The reason for this is not because this kind of technology is impossible or too confusing to build, but because in worlds in which SGCA was built (and wasn't aligned), we die.

At face value, this passage makes an unverifiable claim about parallel branches of the multiverse (how do you know that people die in other worlds?) and then uses this claim as an argument for short timelines/SGCA being relatively easily achievable from now. This makes little sense to me: I don't think the history of technological development is such that we should already wonder why are we still alive. On the other hand, you don't need to argue for short timelines/SGCA being relatively easily achievable from now in such a convoluted way: this view is well-respectable anyway and doesn't really require justification at all. Plenty of people, from Connor Leahy to Geoffrey Hinton now, have short timelines.

You do not align AI; you build aligned AI.

Both. I agree that we have to design AI such that it has inductive (learning) priors such that AI learns world models that are structurally similar to people's world models: this makes alignment easier; and I agree that the actual model-alignment process should start early in the AI training (i.e., development), rather than only after pre-training (and people already do this, albeit with the current Transformer architecture). But we also need to align AIs continuously during deployment. Prompt engineering is the "last mile" of alignment.

AI goals based on trying to point to things we care about inside the AI's model are the wrong way to go about things, because they're susceptible to ontology breaks and to failing to carry over to next steps of self-improvements that an world-saving-AI should want to go through.

Instead, the aligned goal we should be putting together should be eventually aligned; it should be aligned starting from a certain point (which we'd then have to ensure the system we launch is already past), rather than up to a certain point.

This passage seems self-contradictory to me, because "aligned goal" will become "misaligned" when the systems' (e.g., humans and AIs) world models and values drift apart. There is no such goal, whatever that is, that could magically prevent this from happening. I would agree with this passage if the first sentence of the second paragraph would read as "the alignment protocols and principles we should be putting together should lead to robust continuous alignment of humans' and AIs' models of the world".

The aligned goal should be "formal". It should be made of fully formalized math, not of human concepts that an AI has to interpret in its ontology, because ontologies break and reshape as the AI learns and changes. the aligned goal should have the factual property that a computationally unbounded mathematical oracle being given that goal would take desirable actions [...]

This passage seems to continue the confusion of the previous passage. "Math" doesn't "factually" guarantee the desirability of actions; all parts of the system's ontology could drift, in principle, arbitrarily far, including math foundations themselves. Thus we should always be talking about a continuous alignment process, rather than "solving alignment" via some clever math principle which would allow us to then be "done".

An interesting aside here is that humans are themselves not aligned even on such a "clear" thing as foundations of mathematics. Otherwise, the philosophy of mathematics would already be a "dead" area of philosophy by now, but it's very much not. There is a lot of relatively recent work in the foundations of mathematics, e.g., univalent foundations.

Not at all convinced that "strong agents pursuing a coherent goal is a viable form for generally capable systems that operate in the real world, and the assumption that it is hasn't been sufficiently motivated.