Are you saying that if we assumed agents to be Cartesian[1], then you'd know a solution to the problem of {how could a weak agent train and align a very powerful agent}? If yes, could you outline that solution?

Yes, and while I can't totally describe the situation, I can say this:

The first step would be to say scale up the experiment of Pretraining from Human Feedback by using larger data, then curating it for alignment. In particular, we can even try to design a data set such that it uses words like freedom, justice, alignment and more value laden words.

But the real power of the Cartesian agent for alignment is the fact that the agent-world interactions can only happen on a well-defined simple interface, and the computer isn't hackable. That immediately means that many AI risk stories evaporate, as you can only learn legitimate generalizations, not deceptive generalizations leading to deceptive alignment and you can't amplify Goodhart or hack the human's values, making alignment simpler since we can ensure that there doesn't need to be cumbersome protection that incurs alignment taxes. Thus we can bring it back to simple iteration until we succeed.

How does RLHF solve problems of embedded alignment? I'm guessing you're referring to something other than the problems outlined in Embedded Agency?

You're right that it doesn't do well, but the main strategy is to reward AI whenever it takes good looking actions. Problem is, as you can see, it's trying to align a agent after it's already capable, and with higher and higher power, this is increasingly dangerous. In particular, since the AI controls the learning schedule, it will probably be incentivized into hacking the human, since the human's values are just another thing in the world, and there's no well defined interface that's under our control.

Pretraining from Human Feedback does alignment before it gets significantly better capabilities.

What exact distinction do you mean by "online" vs "offline"? Given that any useful/"pivotal" AI would need to learn new things about the world (and thus, modify its own models/mind) in order to form and execute a useful/pivotal plan, it would have to learn "online", no?

I agree with this that one eventually has to shed the Cartesian boundary and do online learning/training. But the key thing we've learned from deep learning is that we can translate the ontology of Cartesianism without too much trouble. In particular, one can do a whole lot of training inside a Cartesian setting called offline learning, and when it moves to the embedded setting so that it's online setting, it generalizes the capabilities learned in the Cartesian setting really well. If one could do this for alignment, then the problem transforms into a solvable problem, as we can ensure that there is no way to hack the human's values or hack it's environment.

And indeed Pretraining from Human Feedback is the closest I've seen to actually implement this.

The embeddedness of the world dominates asymptotically, but we only need a finite time of Cartesian learning to generalize in the embedded setting correctly.

To talk about the distinction between offline and online learning, offline learning is when we give batches of data, the AI guided by SGD learns that patterns and algorithms that explain that data, and then we give it new data, rinse and repeat essentially. An important point to notice here is that the human is in control of that data, and there's no way for the data to be hacked, unlike in online learning. In particular, the interface is well specified: Text.

In online learning, the AI takes the lead, and it selects it's own data points, and the most we can do is give it reward or punishment. In particular, nearly all human learning is online learning, since we select our own data points.

What kind of data set did you have in mind here? A data set s.t. training an AI on it in some appropriate way would lead to the AI being aligned to human values? Could you give a concrete example of such a data set (and training scheme)?

Unfortunately, that's outside my expertise, so I can't really concretely do this. I am aiming to give something like a possibility proof for alignment, as well as mentioning some practical ways to achieve the idea. The implementation is left to other people.

Now to gesture at a crux I have, I think one of the largest cruxes I have is that I think the human method of learning, which is almost all online, doesn't need to be replicated in AI. A large part of the problems with alignment in humans ultimately stem from that we are almost always online learners, thus we constantly have chances to hack or manipulate our environments, whether it's physical, social or anywhere else, and so we do hack our environments. When we train AI, we don't need to replicate that method, because we've learned that we can do a lot more offline learning to get the AI to learn the basic concepts.

If we could reliably do offline learning in humans, I think alignment at least in the single person case could be totally solved, and while larger groups would have problems staying aligned unless they had special properties, we could plausibly solve alignment for much larger groups.

In essence, the human architecture almost totally prohibits offline learning, while AI architecture does permit offline learning.

The real weakness of offline learning is the cost. While this cost is amortizable such that it's actually pretty cheap over multiple runs of training, and you don't need to recompute rewards, evolution and us had to work with much smaller energy budgets, at least until the 19th century, and in the 21st century, that energy and data could be applied to compute. Thus large upfront costs couldn't be purchased, even if it works far better for alignment in the long run.

Also, I'm focusing on the embedded agency problems of Goodhart's law and subsystem alignment, and I'm not addressing decision theory or embedded world models problems. In essence, I'm focusing only on problems that are alignment related, not capabilities related.

I know this is a long commennt, but I hope you understand why I'm communicating so differently, and why I'm so optimistic about alignment being better as capabilities scale.