I think this is a really good question. I might suppose that people intuitively trust memory more than prediction and so when they see the two don't match they assume its the prediction that's wrong.

That said, one thing we know from research is that memory is really unreliable. It removes things , adds details, changes when you tell or hear stories, shifts to fit narratives, etc. I wonder if our instinct here ought to be that if memory and prediction don't match it's because the memory changed and not that the prediction was wrong.