However sometimes being connected in a certain way to another agent does not make everything that is true for them true for you.

The words "current time" refer to different facts when used by agent at different times. It's better to see different agents as asking different questions (even if using the same words), this way they won't disagree with each other and will be able to refine understanding of the facts which either of them contemplates. A future agent could point out that the past agent wasn't entirely correct, for example, and the past "current time" was actually 12:02 and not 12:03.

This is similar to the problem of making sense of subjective/objective aspects of value. The best method, as far as I know, is to consider different agents as pursuing different ideas, but with the ideas themselves (the laws of developing them) arbitrarily representable, that is objective.