According to Eliezer, the "muggle-born" are in fact the children of people who have one copy of the magical gene (and thus are descendants of Squibs who were cast into muggle society.). That only one in ten thousand people in Britain (who are not wizards themselves) have a copy of the gene for magic is not that improbable.