You're multiplying hypothesises unnecessarily.

Every member of Hogwarts could actually be Dumbledore with polyjuice and a time turner. Remember we only know about the 6 hour limit from him (or people that could be him, or forged by him). There's no reason it couldn't be so, just like there's no reason that the person Harry was having a conversation with couldn't have changed out by a an invisible man with a 'changemyvoiceio' spell.

But it's way more reasonable to assume that people are who they think they are, and that the person that starts a conversation is the same one that finishes it.