In the original books, Harry's cohort was born ten years into an extremely bloody civil war. I always assumed birth rates were extremely low for Harry's age group, which would imply that the overall population is much larger than what you'd extrapolate from class sizes.

Of course, the numbers still don't work. There are 40 kids in canon!Harry's class. Even if you assume that's a tenth of the normal birthrate and the average person lives to 150, you get a wizarding population of 6,000.

In MoR, class sizes are around 120 (more than half the kids are in the armies, and armies are 24 each), which is still problematic - with the generous assumptions above, you get a population of 18,000. But MoR does seem to hint there are other magical schools: Daphne at one point wonders if it's worth going to the same school as Harry just to go to the same school as everybody important, which supports the theory that there are other magic schools, but that almost everyone influential went through Hogwarts.