Today we have a number of approaches to coordination---we sign contracts, create firms with multiple shareholders, vote in democracies, and so forth. I think the starting point for multiple AIs interacting with multiple humans is:

• AI systems use similar mechanisms to coordinate amongst each other--e.g. my AI and your AI may sign a contract, or voters may have AI delegates who vote for them or advise them on how to vote.
• Some AI systems are deployed by (and hopefully aligned with) a collective decision-making process--e.g. a firm may decide to deploy an AI CEO which is overseen by the board, or a government agency may deploy an AI to enforce regulations which is overseen by a bureaucratic process.
• We may interleave those two basic approaches in complex ways, e.g. a firm with AI shareholders may itself deploy an AI, which may sign contracts with other firms' AIs, which are in turn enforced by other AIs who are overseen by a process defined by that contract...

(And regardless of what happens on the object level, AIs and humans will continue improving our methods for cooperation/governance/oversight.)

When I think about this topic I'm mostly interested in ways that this "default" falls short (or may be unrealistic/impractical).

AI may also facilitate new forms of cooperation; those might be needed to cope with new challenges or the new pace introduced by AI, or may result in an improvement over the status quo. Some versions of this:

• We can deploy AI systems with novel forms of oversight, rather than merely representing some existing collective decision-making process. For example, we could imagine "merging utility functions" (as in Servant of Many Masters) or some kind of collective idealization (as in CEV).
• Because some kinds of AI labor are very cheap, we could use coordination mechanisms that are currently impractical---e.g. our AI systems could write many more contracts and negotiate them with low latency; or we could vote directly on a much larger number of decisions made by the state; or we could reduce agency costs by engaging in more careful oversight more sparingly.

Whether or not (or for however long) the "default" is adequate to avoid existential catastrophe, it seems useful to use AI as an opportunity to improve our coordination. In some sense "most" of that work will presumably be done by AI systems, but doing the work ourselves may unlock those benefits much earlier. That may be critically important if the transition to AI creates a ton of chaos before we have AI systems who are much better than us at designing new cooperative arrangements. (This is fairly similar to the situation with alignment, where we could also wait to delegate the problem to AI but that may be too late to avoid trouble.)

I haven't read Critch in-depth, so I can't guarantee I'm pointing towards the same concept he is. Consider this a bit of an impromptu intuition dump, this might be trivial. No claims on originality of any of these thoughts and epistemic status "¯\_(ツ)_/¯"

The way I currently think about it is that multi-multi is the "full hard problem", and single-single is a particularly "easy" (still not easy) special case. 

In a way we're making some simplifying assumptions in the single-single case. That we have one (pseudo-cartesian) "agent" that has some kind of definite (or at least bounded-ly complicated) values that can be expressed. This means we kind of have "just" the usual problems of a) expressing/extracting/understanding the values, in so far as that is possible (outer alignment) and b) making sure the agent actually fulfills those values (inner alignment).

Multi principals then relaxes this assumption into saying we don't have a "single" function, but multiple, which introduces another "necessary ingredient": Some kind of social choice theory "synthesis function", that can take in all the individual functions and spit out a "super utility function" that represents some morally acceptable amalgamation of the other functions (whatever that means). The single case is a simpler special case in that the synthesis function is the equivalent of the identity function, but that no longer works if you have multiple inputs.

In a very simplistic sense, multi is "harder" because we are introducing an additional "degree of freedom". So you might argue we have outer alignment, inner alignment and "even-more-outerer alignment" or "multi-outer alignment" (which would be the synthesis problem), and you probably have to make hard (potentially irreconcilable) moral choices for at least the latter (probably for all).

In multi-multi, if the agents serve (or have different levels of alignment towards) different subsets of principals, this would then add the additional difficulty of game theory between the different agents and how they should coordinate. We can call that the "multi-inner alignment problem" or something, the question of how to get the amalgamation of competing agents to be "inner aligned" and not blow everything up and getting stuck in defect-defect spirals or whatever. (This reminds me a lot of what CLR works on)

I tbh am not sure if single-multi would be harder/different from single-single just "applied multiple times". Maybe if the agents have different ideas of what the principal wants they could compete, but that seems like a failure of outer alignment, but maybe it would be better cast as a kind of failure of "multi-inner alignment".

So in summary I think solutions (in so far as such a thing even exists in an objective fashion, which it may or may not) to the multi-multi problem are a superset of solutions to multi-single,  single-multi and single-single. Vaguely, outer alignment = normativity/value learning, inner alignment = principal agent problem, multi-outer alignment = social choice, multi-inner alignment = game theory, and you need to solve all four to solve multi-multi. If you make certain simplifying assumptions which correspond to introducing "singles", you can ignore one or more of these (i.e. a single agent doesn't need game theory, a single principal doesn't need social choice).

Or something. Maybe the metaphor is too much of a stretch and I'm seeing spurious patterns.