I think it's pretty unclear that MSR is action-guiding for real agents trying to follow functional decision theory, because of Sylvester Kollin's argument in this post.
Tl;dr: FDT says, "Supposing I follow FDT, it is just implied by logic that any other instance of FDT will make the same decision as me in a given decision problem." But the idealized definition of "FDT" is computationally intractable for real agents. Real agents would need to find approximations for calculating expected utilities, and choose some way of mapping their sense data to the abstractions they use in their world models. And it seems extremely unlikely that agents will use the exact same approximations and abstractions, unless they're exact copies — in which case they have the same values, so MSR is only relevant for pure coordination (not "trade").
Many people who are sympathetic to FDT apparently want it to allow for less brittle acausal effects than "I determine the decisions of my exact copies," but I haven't heard of a non-question-begging formulation of FDT that actually does this.
The "pure" case of complete causal separation, as with civilizations in separate regions of a multiverse, is an edge case of acausal trade that doesn't reflect what the vast majority of real-world examples look like. You don't need to speculate about galactic-scale civilizations to see what acausal trade looks like in practice: ordinary trade can already be modeled as acausal trade, as can coordination between ancestors and descendants. Economic and moral reasoning already have elements of superrationality to the extent that they rely on concepts such as incentives or universalizability, which introduce superrationality by conditioning one's own behavior on other people's predicted behavior. This ordinary acausal trade doesn't require formal proofs or exact simulations -- heuristic approximations of other people's behavior are enough to give rise to it.
One implication of functional decision theory is that it might be possible to coordinate across causally-disconnected regions of the universe. This possibility has been variously known as acausal trade, multiverse-wide cooperation via superrationality (MSR), or evidential cooperation across large worlds (ECL). I’ll use the MSR abbreviation (since I don’t want to assume that the cooperation needs to be motivated by evidential decision theory).
I broadly believe that the assumptions behind MSR make sense. But I think the standard presentations of it don’t give very good intuitions for how it might actually work in practice. In this post I present an alternative framing of MSR centered around the concept of “psychohistory”.
Background
Assume that humanity colonizes the universe at a large scale (e.g. controlling many galaxies), and also discovers that we live in some kind of much-larger multiverse (e.g. any of Tegmark's four types of multiverse).
Assume that humanity becomes a highly coordinated entity capable of making large-scale commitments, and also converges towards believing that some version of superrationality or functional decision theory is normatively correct.
Reasoning about psychohistory
In order to make accurate predictions about other causally-disconnected civilizations, we would need to develop a scientific understanding of the dynamics of civilization development, and in particular the values and governance structures that other civilizations are likely to end up with. Call this science “psychohistory”.
Why should we think that psychohistory is even possible? If we build intergalactic civilizations of astronomical complexity, then they might become more difficult to understand in proportion to our increased collective intelligence. But in many domains that we study, there are a few core principles which provide a great deal of insight (e.g. atoms, DNA, relativity). Surprisingly, this also occurs in domains which involve complex interactions between many different agents (e.g. evolution, economics). Economics is still a long way from being able to reliably predict novel macro-level phenomena, but we can at least imagine how a deep understanding of economics might allow us to do so. (For example, Malthus was very prescient in predicting “general gluts”, aka depressions.) The relationship between microeconomics and macroeconomics provides one intuition about the relationship between psychology and psychohistory.
Also, while it may be very difficult to study complex structure at the largest scale of civilization, we may be able to get a very long way by studying complex structure at lower levels, especially if future civilization is very modular. For example, it seems plausible that, once we colonize multiple galaxies, the events in one galaxy will be affected very little by what happens in another galaxy. It might even be the case that, once we’ve colonized multiple solar systems, the events in each of them are relatively disconnected from each other. Extreme modularity arises in classic AI risk scenarios, in which the values that determine the future arise in the context of a single training process, and are then copied across the entirety of the lightcone.
Another key problem is lack of data. Psychohistorical theories would initially only be based on our own trajectory. However, theories could be tested by running simulations. And if we ever encounter aliens, they could provide a wealth of data for testing our theories. Over time, we should expect psychohistorical theories to become much more sophisticated. Early theories might focus on civilizations that started from human-like species; later theories could include civilizations that started from non-human-like species on earth-like planets; later still, civilizations that emerged on very different planets, but in universes with very similar physics; later still, civilizations that emerged under any laws of physics.
Reasoning about trades
Assume that we have a good picture of the distribution of other civilizations throughout the multiverse. There are too many possible civilizations to reason about them on a case-by-case basis; instead, we’d map out high-level clusters. Civilizations might be clustered in terms of their size, their governmental structures, their cultures, etc. For the purposes of MSR, though, the most important factors to cluster them on are the values they care about; and their ability and willingness to carry out MSR.
The former seems like it could be inferred from a sophisticated understanding e.g. evolutionary, psychological, and cultural dynamics. The latter intuitively seems a bit more complicated, since it relies on the high-level decisions made by the civilization. But rather than trying to predict fine-grained details of how different civilizations would carry out MSR, we could try to cluster them at a higher level. For example, perhaps there are only a small number of multi-agent bargaining solutions that have good theoretical properties (e.g. the Nash bargaining solution, the Kalai–Smorodinsky bargaining solution, etc). If so, we can identify which actions each bargaining solution would recommend, and then carry out those actions in proportion to how many civilizations we predict would prefer each bargaining solution. (Of course, many civilizations won’t be interested in doing MSR at all, but they can be ignored for current purposes.)
In practice, calculating complete bargaining solutions may well be intractable. And so the best way of approximating the overall bargaining solution might involve identifying specific “trade deals” that different clusters would engage in if they could negotiate directly. For example, a cluster of highly individualistic civilizations might decide to become more collectivist in exchange for the nearest cluster of collectivist societies becoming more individualist.
Note that this all assumes that other civilizations have a roughly similar “map” of all possible civilizations as we do. The simpler and more powerful our theories of psychohistory, the more likely it is that other civilizations will converge to similar maps. So MSR would be most effective if a few key principles allowed the identification of robust attractors in the space of possible civilizations. If so, the clusters which will benefit most from trade are the ones which are the robust attractors under simple, powerful theories of psychohistory. For this reason, many civilizations will try to make themselves easy to predict.
In order to evaluate these clusters we’d also need some kind of distribution telling us how much to weigh each—aka a metric of “how real they are”. Within a quantum multiverse, we get that “for free”, but across different types of multiverse, this may require arbitrary choices. There doesn’t need to be a single canonical answer in order for MSR to be useful, though, as long as different civilizations converge to similar distributions. (The question of how much to converge seems closely related to the question of how “updateless” to be when applying updateless decision theory.)
Reasoning about values
Each civilization within a cluster will have some very specific values (e.g. “run many human-like minds”). But the values that they’re able to all communicate about, and trade for, are likely to be fairly broad ones, such as:
If a few key axes explain a large proportion of variance between civilizations, then converging to a cooperative strategy might be relatively simple. For example, if civilizations tend to be clustered into those which value individual freedom, and those which value the pursuit of knowledge, then every civilization adjusting to partially optimize for the other value would be strongly beneficial by everyone’s values.
Clusters will exist at multiple different scales. For example, we might be able to develop very detailed knowledge of other pathways that humanity could have taken, and therefore be able to identify fine structure in clusters of those civilizations; but we might only have hazy guesses about what would happen in universes with very different laws of physics, and therefore only be able to cluster them very broadly. Civilizations too similar to ours might already be doing most of the things we’d want them to; civilizations too different from ours might be too alien to trade with. So I expect that there’s some “sweet spot” of cluster scale which it’s optimal to focus our cooperative efforts on.
“Negotiations” between many different parties in general get far more complicated, and so there may well be a high premium on being able to identify Schelling points which many civilizations will identify as possibilities to converge to. Again, though, most of the work of finding these Schelling points can likely be done by developing similar theories of psychohistory; Schelling points could then be points which can be represented in unusually simple ways in those theories.
Fine-grained MSR
The previous discussion assumed that cooperation would happen in terms of fairly broad values, such as the ones listed above. Could more complicated types of coordination be plausible—ones which don’t just require civilizations to shift their high-level values, but rather to build complex structures? This will depend both on how fine-grained our understanding of other civilizations is, and also how easily negotiation can take place.
The easiest other civilizations to do fine-grained MSR with would be versions of humanity that diverged from our own, since they’re the “closest” to us, and we can reason about them most easily. As one toy example, we might imagine a world in which the Industrial Revolution happened in China rather than Europe, extrapolate out what the resulting civilization might look like, and then predict concrete details of what they likely care about (e.g. preserving and acting on Confucian teachings). However, gains from this type of trade seem relatively small, because traditional Chinese values already have significant power in our current world, and so they’ll likely already have pushed for the cheapest gains.
A more sophisticated version of fine-grained MSR might be to build complex structures which span multiple universes—e.g. an art piece consisting of many different components spread across many different universes. This sort of thing seems unlikely to really matter much, though, especially because aesthetic preferences (or other complex preferences, like humor) seem likely to vary greatly across different civilizations. (Though Carl Shulman commented that universes with access to very large or infinite amounts of computation might be able to trade a lot in exchange for “poorer” universes fulfilling idiosyncratic preferences like these.)
Explaining MSR
Given all of the above, the default way I’d explain MSR to someone unfamiliar with it is now something like: