I tend to agree with you that numbers are inessential in a scientific theory, and that Darwin's theory is a good example of this. But your critics also have a point that some nice math has been added to the theory since Darwin's time. (Not enough of a point to justify downvoting you, though, IMHO).

As a smaller scale example of a non-numerical scientific theory, consider the theory that the historical branching order of the Great Ape family tree is "First orangutan, then gorilla, then man, leaving the two species of chimp." That is a meaningful and testable scientific theory as it stands, even though there are no numbers involved. But what spoils my example a little is the observation that this theory is improved by adding numbers. "Orangutan branched ~12M years ago, gorilla 6M, man 5M, bonobo 0.5M."