I would speculate that there's some physiological component involved in spellcasting ability that grows with age, in much the same way that older children are often more coordinated and stronger than younger children. I have no evidence to back this up other than the repeated mentions of 'age matters with spells', however.