Let's consider a scenario.

You've become aware that there exists a pair of brains, and that you are one of them, but you can't tell which. There are practical reasons you would really like to know: Each of the brains are destined for a different fate, they must do different things to prepare. And let's say they're selfish, they're each only optimizing for their own outcome.

Crucially: You know that the brains are physically implemented in very different ways. One has twice the mass as the other (it's an older model), while the other consists of three redundant instances (for robustness to error), so strictly speaking the triplicate brain is pretty much three brains, they just happen to be synchronized and wired out to one body.

The following question has clarified my thoughts about the hard problem of consciousness a lot. It can only be answered once we have a theory of how moments of experience concentrate around physical systems.

What is your probability that you're the heavier brain?

A naive answer is "1/2", but ignoring differences in the brains' physical implementations turns out to be kind of indefensible.

Consider how many different ways of thinking about indexical prior/observer-measure/anthropic binding there are, see that they predict different answers and see that they are all uncomfortably weird:

• A count theory might give the answer 1/4, because there is a sense in which the heavier brain counts as one object and the other counts as three objects (and there are other senses where they don't. Which sense would nature care about?)
• A mass theory gives the answer 2/3, because the heavier brain consists of twice as much of the apparent basic stuff of existence than the triplicate brains combined.
• A force-oriented theory (not recommended) says 1/2, because each system commands equal quantities of physical force, labor, or votes, and so the force theorist reasons that neither matters more than the other. The force theorist doesn't care which one plays host to more moments of life, being, feeling, witness, pleasure or suffering, the quantity of the moments of experience don't give that brain any additional realpolitik leverage over it, so it just asserts the equality of the two objects.
  But being equal in that sense simply doesn't imply equality of probability of observing from within.
  Force theory does not help you to answer the focus question, to predict which brain is you. It ignores the problem. It's less a theory of measure binding and more of a... common way of accidentally denying the existence of the problem.
  • It is possible to formulate a realpolitik-utilitarian type agent who should tend to follow the Force theory, for whom it's correct, but humans, with our selfish aspects, we run into this problem, and I'll argue a bit further down that compassion, concern for the quantity of experience in another, also runs into this problem.
• The pattern theory (also not recommended) would predict 1/2, because the two brains have roughly equal computational heft. A simulation operating with scarce compute would notice they're mostly the same computation, doing the same thing, so most of the computation would be shared, with a few inert tags on the outside saying things like "and it's heavy" or "and there are three of them". Those extra features would be unimportant on the physical computing substrate level, and so would not effect anthropic measure's binding to any physical systems.
  That  theory is wrong because nature is not lossily compressed in that way. Even if it the universe is a simulation (and to be fair it probably is), the next level above the simulation is probably not a simulation, and the next level above that is almost certainly not a simulation, or whatever, it's not going to be so deep that we can justify founding our anthropic principles on the assumption that we don't need a theory of anthropic measure for basic physical reality. We will need one at some point.
  Why would anyone believe the pattern theory? I get the impression that there's this kind of intellectual, to them the whole world is just patterns to be understood, so they apprehend the weight, or the nature, of an object, to be all about its computational pattern, they don't think anything else about it could possibly matter, because in their professional life it wouldn't, for their purposes of understanding and anticipating the system's outward behaviour, nothing else matters. So they don't think about its other aspects like, whether, or to what extent the computing elements exist.
  • A concerning implication that pattern theory seems to have: If you believe that the set of everything that exists, is very big (most of us do), then you'll get a tendency towards marginal outcomes where, since identical observer-moments only count as one moment, normal moments would not be more frequent than abnormal moments. If all patterns have the same measure regardless of their incidence rate, then rare ones count just as much as common ones, which means that since there are more distinct patterns of moment in the weird fringes, most of the moments are weird and you should expect to live statistics-defyingly weird life. And there's no denying that this is a weird implication, especially given that you seem to be living a life that follows statistics quite closely.

(personally, I'm mostly a mass-energy theorist, but I'm not sure how to explain why my anthropic moment has landed in biological intelligence, panpsychism seems not to be completely true, it seems like something more complicated than mass-energy theory is probably going on. But the more I dig at this the more I conclude that for reasons I can't quite articulate yet, reading into the ontological implications of our one observer-moment observation doesn't make sense and can't tell us anything. There seems to be a sense in which it is not a real observation, there's no information being transmitted from base reality to physical brain here, so what do you think you're observing?)

So, even though you can't run a decision theory with a humanlike utility function without running into this question, there is no normal answer. Whichever way you want to go, we're going to have to do some work to acclimate ourselves to it.

The question suggests a sort of binding weighting associated with every physical system in your reference class, that determines the probability of finding out that you're that one, rather than another one.

Let's also name that entity in our utility function that gives a thing moral personhood, a compassion measure, that determines the rate at which a thing outside of your reference class, that almost certainly isn't you, can experience suffering or joy or meaning or grace, as a component in the extent to which you will care what it's experiencing.

It would seem to me that the binding weighting and the compassion measure are basically the same concept. It's hard to justify separating them.

This is not really in the domain of observation or reason, though, it's ethos, It's in the domain of the inspection of our desires. While the binding weighting could belong to any selfish agent, the compassion measure belongs to the human utility function. Whether this strange thing I've introduced to you is recognizable to you as your utility function's compassion measure will depend on the way your utility function binds to objects in the external reality under shifts in frame.

All I can really say to convince you that these things are the same, is to mention that my utility function, does reliably bind its "compassion measure" tags to any simulacra I bear of others' anthropic binding weighting. It does that. And since you and I are the same species, I'll assume for now that yours does the same.

So, for us, the question of how anthropic binding works is highly consequential: If we don't have a theory, then we wont know the extent to which the artificial brains that we mean to create, will experience being. We need to know, as that tells us how good or bad it is when they experience beauty or suffering or whatever else. Making heaven for a bunch of brains with negligible anthropic measure would be a catastrophic waste. It is burdensome that we must attend such metaphysical issues, but our utility function seems to care about these things.

The question is extremely hard to empirically test despite having great material impact, because each of us constitute the relation of just one physical system to just one anthropic moment, and we cannot ever witness another without leaving our current datapoint behind. Somehow, we'll have to draw all of our inferences from this one association that we have, between,

a human brain,

and the experience of reaching the end of a short philosophy post.

This post presents the same points as the scifi short The Mirror Chamber, but with less adornment, worldbuilding, and with more structure and concision.

I feel like I should credit Eliezer's Ebborians Post as formative, but in retrospect I really don't have any trace of having read it. I think I might have actually missed it. I think we both just ended up asking similar questions because there's no way to think really deeply about epistemology without noticing that anthropic measure is seriously weird.