This is a special post for quick takes by Dach. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
(2020 - 10 - 03) EDIT: I have found the solution: the way I was thinking about identity turns out to be silly.
In general, if you update your probability estimates of non-local phenomenon based on anthropic arguments, you're (probably? I'm sure someone has come up with smart counterexamples) doing something that includes the sneaky implication that you're conducting FTL communication. I consider this to be a reductio ad absurdum on the whole idea of updating your probability estimates of non-local phenomena based on anthropic arguments, regardless of the validity of the specific scenario in this post.
If you conduct some experiment which tries to determine which world you're in, and you observe x thing, you haven't learned (at least, in general) anything about what percentage of exact copies of you before you did the experiment observed what you observed.
If you do update and If you claim the update you're making corresponds to reality, you're claiming that non-local facts are having a justified influence on you. Whenever you put it like that, it's very silly. By adjusting the non-local worlds, we can change this justified influence on you (otherwise your update does not correspond to reality), and we have FTL signaling.
The things you're experiencing aren't any evidence about the sorts of things that most exact copies of your brain are experiencing, and if you claim it is you're unknowingly claiming FTL communication is possible, and that you're doing it right now.
I'll need to write something more substantial about this problem.
(End Edit)
(This bit is edited to redact irrelevant/obviously-wrong-in-hindsight information)
So, let us imagine our universe is "big" in the sense of many worlds, and all realities compatible with the universal wavefunction are actualized- or at least something close to that. This seems increasingly likely to me.
Aligned superintelligences might permute through all possible human minds, for various reasons. They might be especially interested in permuting through minds which are thinking about the AI alignment problem- also for various reasons.
As a result, it's not evident to me for normal reasons that most of the "measure" of me-right-now flows into "normal" things- it seems plausible (on a surface level- I have some arguments against this) that most of the "measure" of me should be falling into Weird Stuff. Future superintelligences control the vast majority of all of the measure of everything (more branches in the future, more future in general, etc.), and they're probably more interested in building minds from scratch and then doing weird things with them.
If, among all of the representations of my algorithm in reality, (100%) * (1 - 10^-30) of my measure was "normal stuff", I'd still expect to be "diverted" basically instantly, if we assume there's one opportunity for a "diversion" every planck second.
However, this is, of course, not happening. We are constantly avoiding waking up from the simulation.
Possible explanations:
(Apply the comments in the edit above to the original post. If I think that the fact I'm not observing myself being shoved into a simulation is evidence that most copies of my algorithm throughout reality are not being shoved into simulations, I also need to think that the versions of me which don't get shoved into simulations are justifiably updating in correspondence with facts of arbitrary physical separation from themselves, thus FTL signaling. Or, even worse, inter-universe signaling.)
Right, that isn't an exhaustive list. I included the candidates which seemed most likely.
So, I think superintelligence is unlikely in general- but so is current civilization. I think superintelligences have a high occurrence rate given current civilization (for lots of reasons), which also means that current civilization isn't that much more likely than superintelligence. It's more justified to say "Superintelligences which make human minds" have a super low occurrence rate relative to natural examples of me and my environment, but that still seems to be an unlikely explanation.
Based on the "standard" discussion on this topic, I get the distinct impression that the probability our civilization will construct an aligned superintelligence is significantly greater than, for example, 10^-20%, and the large amounts of leverage that a superintelligence would have (There's lots of matter out there) would produce this same effect.
Future superintelligences could steal minds to cure "past sufferings" and to prevent s-risks, and to resurrect all the dead. These is actually a good thing, but for the resurrection of the dead they have to run the whole world simulation once again for last few thousands years. In that case it will look almost like normal world.