He claimed that there were outright historic geniuses laboring in the fields.

And this part seems entirely plausible. American slaves had no opportunity to become famous mathematicians unless they escaped, or chanced to have an implausibly benevolent Dumbledore of an owner.

Gould makes a much stronger claim, and I attach little probability to the part about the present day. But even there, you're ignoring one or two good points about the actions of famous mathematicians. Demanding citations for 'trying to kill people can ruin your life' seems frankly bizarre.