Yes, but why? There's not a gene for verbal skills; there's not even a gene for language use, nor any single smoking-gun neuroanatomical correlate of it. The ones you may have heard about -- Broca's area, FOXP2 -- are pretty broad in function and do a bunch of things, a failure of any one of which would clearly impair the ability to perform spoken language.
Is it possible that the trait we think of as verbal skill is rooted in some ultimately-genetic factor? Sure, it's possible -- but that idea isn't particularly rigorously-supported by the available evidence, either. Meanwhile there are all these other possible contributing factors that could influence such a trait.
I'm not sure what position you think you're arguing against. The ev-psych position is that the presence of a Y chromosome ultimately causes the difference in verbal skills (along with a lot of other things) between men and women. (Most of this influence probably passes through the SRY gene and the presence of sex hormones, but that's less certain than the effect itself.)
Your counter-argument appears to be that there isn't a single node in the causal diagram that corresponds to just the the effect on verbal skills. I agree that there probably doesn't exist such a node but fail to see why we should expect it to exist if ev-psych explanation is correct.
I don't mean to claim that there should be a conflict.
Most likely the conflict arises because of many things, such as 1)Women having been ostracized for much of our society's existence 2)People failing at the is-ought problem, and committing the Naturalistic Fallacy 3)Lots of media articles saying unbelievably naïve evolutionary statements as scientific fact 4)Feminists as a group being defensive 5)Specially defensive when it comes to what is said to be natural. 6) General disregard by people, and politically engaged people (see The Blank Slate, by Steve Pinker) of the existence of a non Tabula Rasa nature. 7) Lack of patience of Evolutionary Psychologists to make peace and explain themselves for the things that journalists, not them, claimed. and others...
But the fact is, the conflict arose. It has only bad consequences as far as I could see, such as people fighting over each other, breaking friendships, and prejudice of great intensity on both sides.
How to avoid this conflict? Should someone write a treatise on Feminist Evolutionary Psychology? Should we get Leda Cosmides to talk about women liberation?
There are obviously no incompatibilities between reality and the moral claims of feminism. So whichever facts about evolutionary psychology are found to be true with the science's development, they should be made compatible. Compatibilism is possible.
But will the scientific community pull it off?
Related: Pinker Versus Spelke - The Science of Gender and Science
http://www.edge.org/3rd_culture/debate05/debate05_index.html
David Buss and Cindy Meston - Why do Women Have Sex?
http://www.youtube.com/watch?v=KA0sqg3EHm8