Still, they could count with beads or rocks, à la the magic sheep-counting bucket.

Here's a decent argument that they probably don't have words for numbers because they don't count, rather than the other way round, contra pop-Whorfianism. (Otherwise I guess they'd just borrow the words for numbers from Portuguese or something, as they probably did with personal pronouns from Tupi.)