They have just been through a war - many such people may have died. It's also possible that most wizards/witches had children much later, and the relatively 'young' families we see were a response to the war.