All of which worked fine, except babies forget just about everything that happens before the fifth year of life so the immaculate transfer of Voldemort's entire mind got wiped by infant amnesia.

I assumed that babies just didn't store long-term memories in the first place, but I've looked it up on Wikipedia and it sounds like I was mistaken. Huh.