On the other hand, pretty much everyone has forgotten most of their infancy—presumably, Dementors don’t bring such memories back because they’re almost never as traumatic as Harry’s—and Remembralls don’t start glowing like mad in everybody’s hands, so something is still missing there.

Much more than infancy - apparently everything before 10 is suspect, and everything before 4 especially so according to Wikipedia on childhood amnesia.