Yes, but why? There's not a gene for verbal skills; there's not even a gene for language use, nor any single smoking-gun neuroanatomical correlate of it. The ones you may have heard about -- Broca's area, FOXP2 -- are pretty broad in function and do a bunch of things, a failure of any one of which would clearly impair the ability to perform spoken language.

Is it possible that the trait we think of as verbal skill is rooted in some ultimately-genetic factor? Sure, it's possible -- but that idea isn't particularly rigorously-supported by the available evidence, either. Meanwhile there are all these other possible contributing factors that could influence such a trait.

I'm not sure what position you think you're arguing against. The ev-psych position is that the presence of a Y chromosome ultimately causes the difference in verbal skills (along with a lot of other things) between men and women. (Most of this influence probably passes through the SRY gene and the presence of sex hormones, but that's less certain than the effect itself.)

Your counter-argument appears to be that there isn't a single node in the causal diagram that corresponds to just the the effect on verbal skills. I agree that there probably doesn't exist such a node but fail to see why we should expect it to exist if ev-psych explanation is correct.