Perhaps the single most important thing to realize when adopting the (non-relativistic) MWI is that physical three-dimensional space is not the fundamental space of the theory. The arena in which the theory takes place is configuration space.

The sort of information you're looking for, about which physical space you return to, is encoded in configuration space and the Schrodinger equation. Even though the two different versions of point A you're talking about inhabit the same location in physical space, they inhabit distinct locations in configuration space, so the laws of quantum mechanics can distinguish between them.

Think of a simple case where you perform a spin measurement experiment at location A, and say that in your branch the measuring device at the location shows that the particle you measured was spin up. There will be another branch where the measuring device shows spin down, but this branch will have separated quite substantially from your branch in configuration space. Now your question amounts to this: if you leave location A and then return, why don't you return to find that you're in the other branch (the spin-down one). Looking at things from the configuration space perspective, the question doesn't seem all that troubling. This other branch is in a totally separate region of configuration space. Moving around in physical space won't jump you across configuration space to this other branch. The Schrodinger equation is local in configuration space. All that happens when you move around in physical space is that your branch (the one with the spin up measurement) moves around with you in configuration space. Since you always stay within that branch, returning to location A will reveal that the detector still shows spin up.

Perhaps what I say in this comment about "worlds" in MWI not being places might also help. You can't move out of your world (your branch) by travelling far enough in physical space.