Rowling-canon strongly implies that Dumbledore was the reason Voldemort didn't attack Hogwarts; that is, Dumbledore's presence at Hogwarts protected the castle, and not the other way around. Hogwarts fell very soon after Dumbledore's death. However, while Dumbledore may have prevented Voldemort from outright capturing Hogwarts, in the same way he stopped Voldemort from outright capturing the Ministry, he proved unable to prevent Voldemort and friends from repeatedly sneaking into the castle, and he doesn't have the time to personally guard the Potters 24/7.