Being a Brahmin does not put rice on the table. Again, he was on the brink of starving, he says; this screens off any group considerations - we know he was very poor.

It screens off any wealth considerations, with the exception of his education (which is midlly relevant). It has a big impact on the question of average IQ and ancestry, though. Brahmin average IQ is probably north of 100,* and so a first-rank mathematician coming from a Brahmin family of any wealth level is not as surprising as a first-rank mathematician coming from a Dalit family.

So we still need to explain the absence (as far as I know) of first rate Dalit mathematicians. Gould argues that they're there, and we're missing them; the hereditarian argues that they're not there. One way to distinguish between the two is to evaluate the counterfactual statement "if they were there, they wouldn't be missed," and while Ramanujan is evidence for that statement it's weakened because of the potential impact of caste prejudice / barriers.

(It seems like the example of China might be better; it seems that young clever people have had the opportunity to escape sweatshops and cotton fields and enter the imperial service / university system for quite some time. Again, though, this is confounded by Han IQ being probably slightly north of 100, and so may not generalize beyond Northeast Asia and Europe.)

*Unfortunately, there is very little solid research on Indian IQ by caste.