I think that if it were to go ahead, it should have been made stronger and clearer. But this wouldn't have been politically feasible, and therefore if that were the standard being aimed for it wouldn't have gone ahead.

This I think would have been better than the outcome that actually happened.

Whoa, this seems very implausible to me. Speaking with the courage of one's convictions in situations which feel high-stakes is an extremely high bar, and I know of few people who I'd describe as consistently doing this.

If you don't know anyone who isn't in this category, consider whether your standards for this are far too low.

For a while now I've been thinking about the difference between "top-down" agents which pursue a single goal, and "bottom-up" agents which are built around compromises between many goals/subagents.

I've now decided that the frame of "centralized" vs "distributed" agents is a better way of capturing this idea, since there's no inherent "up" or "down" direction in coalitions. It's also more continuous.

Credit to @Scott Garrabrant, who something like this point to me a while back, in a way which I didn't grok at the time.

1. Consider the version of the 5-and-10 problem in which one subagent is assigned to calculate U | take 5, and another calculates U | take 10. The overall agent solves the 5-and-10 problem iff the subagents reason about each other in the "right ways", or have the right type of relationship to each other. What that specifically means seems like the sort of question that a scale-free theory of intelligent agency might be able to answer.
2. I'm mostly trying to extend pure mathematical frameworks (particularly active inference and a cluster of ideas related to geometric rationality, including picoeconomics and ergodicity economics).

Ooops, good catch. It should have linked to this: https://www.lesswrong.com/posts/FuGfR3jL3sw6r8kB4/richard-ngo-s-shortform?commentId=W9N9tTbYSBzM9FvWh (and I've changed the link now).

Here is a broad sketch of how I'd like AI governance to go. I've written this in the form of a "plan" but it's not really a sequential plan, more like a list of the most important things to promote.

1. Identify mechanisms by which the US government could exert control over the most advanced AI systems without strongly concentrating power. For example, how could the government embed observers within major AI labs who report to a central regulatory organization, without that regulatory organization having strong incentives and ability to use their power against their political opponents?
   1. In practice I expect this will involve empowering US elected officials (e.g. via greater transparency) to monitor and object to misbehavior by the executive branch.
2. Create common knowledge between the US and China that the development of increasingly powerful AI will magnify their own internal conflicts (and empower rogue states) disproportionately more than it empowers them against each other. So instead of a race to world domination, in practice they will face a "race to stay standing".
   1. Rogue states will be empowered because human lives will be increasingly fragile in the face of AI-designed WMDs. This means that rogue states will be able to threaten superpowers with "mutually assured genocide" (though I'm a little wary of spreading this as a meme, and need to think more about ways to make it less self-fulfilling).
3. Set up channels for flexible, high-bandwidth cooperation between AI regulators in China and the US (including the "AI regulators" in each who try to enforce good behavior from the rest of the world).
4. Advocate for an ideology roughly like the one I sketched out here, as a consensus alignment target for AGIs.

This is of course all very vague; I'm hoping to flesh it out much more over the coming months, and would welcome thoughts and feedback. Having said that, I'm spending relatively little of my time on this (and focusing on technical alignment work instead).

Here is the broad technical plan that I am pursuing with most of my time (with my AI governance agenda taking up most of my remaining time):

1. Mathematically characterize a scale-free theory of intelligent agency which describes intelligent agents in terms of interactions between their subagents.
   1. A successful version of this theory will retrodict phenomena like the Waluigi effect, solve theoretical problems like , and make new high-level predictions about AI behavior.
2. Identify subagents (and subsubagents, and so on) within neural networks by searching their weights and activations for the patterns of interactions between subagents that this theory predicts.
   1. A helpful analogy is how Burns et al. (2022) search for beliefs inside neural networks based on the patterns that probability theory predicts. However, I'm not wedded to any particular search methodology.
3. Characterize the behaviors associated with each subagent to build up "maps" of the motivational systems of the most advanced AI systems.
   1. This would ideally give you explanations of AI behavior that scales in quality based on how much effort you put in. E.g. you might be able to predict 80% of the variance in an AI's choices by looking at which highest-level subagents are activated, then 80% of the remaining variance by looking at which subsubagents are activated, and so on.
4. Monitor patterns of activations of different subagents to do lie detection, anomaly detection, and other useful things.
   1. This wouldn't be fully reliable—e.g. there'd still be some possible failures where low-level subagents activate in ways that, when combined, leads to behavior that's very surprising given the activations of high-level subagents. (ARC's research seems to be aimed at these worst-case examples.) However, I expect it would be hard even for AIs with significantly superhuman intelligence to deliberately contort their thinking in this way. And regardless, in order to solve worst-case examples it seems productive to try to solve the average-case examples first.

I'm focusing on step 1 right now. Note that my pursuit of it is overdetermined—I'm excited enough about finding a scale-free theory of intelligent agency that I'd still be working on it even if I didn't think steps 2-4 would work, because I have a strong heuristic that pursuing fundamental knowledge is good. Trying to backchain from an ambitious goal to reasons why a fundamental scientific advance would be useful for achieving that goal feels pretty silly from my perspective. But since people keep asking me why step 1 would help with alignment, I decided to write this up as a central example.

Yepp, see also some of my speculations here: https://x.com/richardmcngo/status/1815115538059894803?s=46