These posts at Slate Star Codex seem relevant:
But I worry that most smart people have not learned that a list of dozens of studies, several meta-analyses, hundreds of experts, and expert surveys showing almost all academics support your thesis – can still be bullshit.
The Control Group Is Out Of Control
How do I know it’s crap? Well, I use my personal judgment. How do I know my personal judgment is right? Well, a smart well-credentialed person like James Coyne agrees with me. How do I know James Coyne is smart? I can think of lots of cases where he’s been right before. How do I know those count? Well, John Ioannides has published a lot of studies analyzing the problems with science, and confirmed that cases like the ones Coyne talks about are pretty common. Why can I believe Ioannides’ studies? Well, there have been good meta-analyses of them. But how do I know if those meta-analyses are crap or not? Well…
For every controversial subject I've heard of, there are always numerous very smart experts on either side. So I'm curious how it is that rational non-experts come to believe one side or the other.
So, what are your meta-arguments for going with one side or the other for any given controversial subject on which you have an opinion?