Some eleven-year-olds might be that way, but if your sample consists mostly of relatives of geniuses, it's going to be pretty skewed, I would think.

There's no causal link between Harry and Draco and Hermione and Blaise and... I dunno who else people are claiming is unrealistic. Still, four unrelated genius-level children out of the, I don't know, one hundred first year Hogwarts students? It's not entirely unfair to see that as statistically unlikely, even if theoretically possible.