The spatial and temporal viewpoint analogy doesn't quite work, because you can sensibly talk about a movement through space, since movement means change in space/change in time. But you can't really talk about movement through time because that would be change in time/change in time. So if we set time equal to a constant, and look at space, your viewpoint is only at one spatial point. But if we look at time, your viewpoint is at a continuum of places, sort of a "line" through time.

Your analysis of the neural construction of spatial viewpoints is good, and I think it holds for the neural construction of temporal viewpoints. If I knew these neural constructions, then I would know exactly why you feel a subjective experience of a viewpoint moving through space and time. I could understand these causal mechanisms an be satisfied with my knowledge of the process. But I might still be confused about my feeling of subjective experience, because it doesn't explain why I feel things the way that I do. I've been reluctant to use the word "qualia" but essentially that's what I'm getting at. Hence my analogy with red: Even if I knew the parts of the brain that responded to red, would I know why red looks the way it does?

So if we want to talk about other people, then I think we're all on the same page. These sensations of spatial/temporal movement could be explained with neuroscience, and have no profound philosophical implications.