When I first read the Conley study a couple of years ago, it struck me as odd that people could think you could overturn an empirical study with a hypothetical one. Though experiments can't overturn empirical ones.
I would be a lot more satisfied if someone could do an actual empirical study where women were approached by a man who was known to be good in bed and whether they said yes or no to casual sex on the spot. Maybe a guy being seen in the company of a lot of women doing the proposal? I dunno.
Dr. Aaron Sell recently wrote an interesting piece about political incentives and shoddy statistical work in science. In particular, he was highly critical of the much-publicized Conley article in the Journal of Personality and Social Psychology which attempted to demonstrate that "large gender differences... [in willingness to engage in] casual sex [have] more to do with perceived personality characteristics of the female versus male proposers than with gender differences."
[[The full article can be found here. Please note - I have deliberately chosen to omit quoting any material which Less Wrongers may find mindkilling. It would be nice if we could keep it that way.]]
He points out that Conley's results were obtained by hypothetical pseudo-experiments, and spurious controls. For example, when Conley found that men were more willing to sleep with their friends than women, she made this difference "evaporate" by:
Yet for bottom line reasons, we now find Eagly and Woods reading Conley uncritically, and even claiming that she has overturned Clark and Hatfield's classic (and truly experimentally based) article demonstrating that women are radically less willing than men to have casual sex.
Sell concludes:
While empiricism is great, I have long believed that the social and organisational structures in which science is practised makes it especially vulnerable to political capture, so this plays right into my biases. Am I missing something important?