Somewhat related: I think we do have a 3D map of the environment even for things that we aren't looking at at the moment.

I do not appear to have that - or, at least, I don't get much use out of it if it's in here. While I can keep track of who is where in the house, I do so more in the form of a list of Last Known Locations, not in any sort of map (2D or 3D).

Possibly related - I am notorious for getting lost easily while driving, and can get very badly turned around if I am merely a short distance away from where I should be. I tend to navigate by memorising a route from A to B, as a list of directions (turn left at the third corner, then it's the fourth street on the right...) and then I get into trouble if I can't follow that route. (Nowadays, I tend to lean heavily on GPS when going to new places).