Good insight! This would also explain why "Harry's worst memory" was something he shouldn't actually remember. If it was actually Voldemort's memory passed through Harry's loyalties and emotional valuation, it might be the thing that popped out. Which also makes Harry having revealed this memory to Dumbledore in Ch 82 pretty significant, and suggests a radically different interpretation to this text:

"It's a funny thing," Harry said, his voice wavering like something seen through underwater. "Do you know, the day I went in front of the Dementor, what my worst memory was? It was my parents dying. I heard their voices and everything."

The old wizard's eyes widened behind the half-moon glasses.

"And here's the thing," Harry said, "here's the thing I've been thinking about over and over. The Dark Lord gave Lily Potter the chance to walk away. He said that she could flee. He told her that dying in front of the crib wouldn't save her baby. 'Step aside, foolish woman, if you have any sense in you at all -'" An awful chill came over Harry as he spoke those words from his own lips, but he shook it off and continued. "And afterward I kept thinking, I couldn't seem to stop myself from thinking, wasn't the Dark Lord right? If only Mother had stepped away. She tried to curse the Dark Lord but it was suicide, she had to have known that it was suicide. Her choice wasn't between her life and mine, her choice was for herself to live or for both of us to die! If she'd only done the logical thing and walked away, I mean, I love Mum too, but Lily Potter would be alive right now and she would be my mother!" Tears were blurring Harry's eyes. "Only now I understand, I know what Mother must have felt. She couldn't step aside from the crib. She couldn't! Love doesn't walk away!"

It was like the old wizard had been struck, struck by a chisel that shattered him straight down the middle.

The eyes widening and seeming to be shattered... what if Harry and the reader are meant to think this is just Dumbledore being all human and weak and sympathetic and stuff, but actually Dumbledore was surprised by learning that Harry is Harrymort in a deeper sense than he'd realized and then covering it with acting?