Perhaps the single most important thing to realize when adopting the (non-relativistic) MWI is that physical three-dimensional space is not the fundamental space of the theory. The arena in which the theory takes place is configuration space.
The sort of information you're looking for, about which physical space you return to, is encoded in configuration space and the Schrodinger equation. Even though the two different versions of point A you're talking about inhabit the same location in physical space, they inhabit distinct locations in configuration space, so the laws of quantum mechanics can distinguish between them.
Think of a simple case where you perform a spin measurement experiment at location A, and say that in your branch the measuring device at the location shows that the particle you measured was spin up. There will be another branch where the measuring device shows spin down, but this branch will have separated quite substantially from your branch in configuration space. Now your question amounts to this: if you leave location A and then return, why don't you return to find that you're in the other branch (the spin-down one). Looking at things from the configuration space perspective, the question doesn't seem all that troubling. This other branch is in a totally separate region of configuration space. Moving around in physical space won't jump you across configuration space to this other branch. The Schrodinger equation is local in configuration space. All that happens when you move around in physical space is that your branch (the one with the spin up measurement) moves around with you in configuration space. Since you always stay within that branch, returning to location A will reveal that the detector still shows spin up.
Perhaps what I say in this comment about "worlds" in MWI not being places might also help. You can't move out of your world (your branch) by travelling far enough in physical space.
Perhaps the single most important thing to realize when adopting the (non-relativistic) MWI is that physical three-dimensional space is not the fundamental space of the theory. The arena in which the theory takes place is configuration space.
Of course, this pretty much destroys the "locality" argument.
Today's post, Spooky Action at a Distance: The No-Communication Theorem was originally published on 05 May 2008. A summary (taken from the LW wiki):
Discuss the post here (rather than in the comments to the original post).
This post is part of the Rerunning the Sequences series, where we'll be going through Eliezer Yudkowsky's old posts in order so that people who are interested can (re-)read and discuss them. The previous post was Bell's Theorem: No EPR "Reality", and you can use the sequence_reruns tag or rss feed to follow the rest of the series.
Sequence reruns are a community-driven effort. You can participate by re-reading the sequence post, discussing it here, posting the next day's sequence reruns post, or summarizing forthcoming articles on the wiki. Go here for more details, or to have meta discussions about the Rerunning the Sequences series.