We saw it right up until the moment Voldemort cast the Killing Curse

You're misremembering the chapter, that scene ends like this:

And the boy in the crib saw it, the eyes, those two crimson eyes, seeming to glow bright red, to blaze like miniature suns, filling Harry's whole vision as they locked to his own -

Thank you for quoting the bit in chapter 46. I had forgotten it, and it is worth taking into account. But in context, I don't think it shows what you think it does:

Lord Voldemort had killed James Potter. He had preferred to spare Lily Potter's life. He had continued his attack, therefore, with the sole purpose of killing their infant child.

Dark Lords were not usually scared of infant children.

So prior to recovering that memory, Harry didn't have nearly as much reason to think Voldemort had been afraid of him. In Ch. 3, for all Harry knew Voldemort was just the sort of person who would murder his enemies' children given the opportunity, and he would have been largely correct to think that.

But it seems Harry's inference that Voldemort meant to kill him may not be as safe as Harry assumes. It is equally consistent with everything Harry notices to think that Voldemort meant to do something else to Harry. Consider this part:

"I give you this rare chance to flee," said the shrill voice. "But I will not trouble myself to subdue you, and your death here will not save your child. Step aside, foolish woman, if you have any sense in you at all!"

"Not Harry, please no, take me, kill me instead!"

The empty thing that was Harry wondered if Lily Potter seriously imagined that Lord Voldemort would say yes, kill her, and then depart leaving her son unharmed.

"Very well," said the voice of death, now sounding coldly amused, "I accept the bargain. Yourself to die, and the child to live. Now drop your wand so that I can murder you."

Suppose Voldemort meant to do something else to Harry other than kill him, and that he succeeded in doing whatever he meant to do. If so, "your death here will not save your child" would turn out to be true. In that case, perhaps what amused Voldemort was realizing that Lily had misunderstood what he was going to do to Harry, and that she had offered up her life to prevent something that was not going to happen anyway. In that case, "Yourself to die, and the child to live" also reflects Voldemort's true intentions.

At this point, it looks to me very much like Voldemort somehow decided killing baby Harry was not the right response to the prophecy. The prophecy looks like a key clue here; probably Voldemort wouldn't have bothered with such a complicated plot as he appears to be pursuing without the prophecy. But what thought process led to that plot?