Right, that's the claim about the Piraha at least: their language has no numerals, only two terms for 'smaller amount' and 'larger amount' and then circumlocutions for things like 'many':

Frank et al. (2008) describes two experiments on four Pirahã speakers that were designed to test these two hypotheses. In one, ten batteries were placed on a table one at a time and the Pirahã were asked how many were there. All four speakers answered in accordance with the hypothesis that the language has words for 'one' and 'two' in this experiment, uniformly using hói for one battery, hoí for two batteries, and a mixture of the second word and 'many' for more than two batteries.

The second experiment, however, started with ten batteries on the table, and batteries were subtracted one at a time. In this experiment, one speaker used hói (the word previously supposed to mean 'one') when there were six batteries left, and all four speakers used that word consistently when there were as many as three batteries left. Though Frank and his colleagues do not attempt to explain their subjects' difference in behavior in these two experiments, they conclude that the two words under investigation "are much more likely to be relative or comparative terms like 'few' or 'fewer' than absolute terms like 'one'".

I haven't seen other studies, but I'd assume that people in cultures without full number systems would get confused by large numbers, just since they don't have practice with them.

Now, the claim about the Piraha is that they wanted to learn to count -- after Everett noticed they couldn't count, they got worried that they were getting ripped off in trade -- but couldn't. I don't know how much to trust that, though.