It's a good point, and I think enough to call the paper's findings seriously into question, but I don't think fixing it would be enough to salvage the methodology. Ideological bias tends to be transparent from the inside: I'd expect any academic with a strong commitment to academic neutrality to punish perceived ideological bias in proportion to its magnitude, but I'd also expect the same academics to perceive viewpoints leaning toward their own ideology as less biased than the alternatives. Probably much less.
You'd need to do something a lot more clever to filter that out: maybe something like asking academics about the perceived rates of each type of ideological bias in papers and grants they evaluate, and normalizing based on that.
Summary: Current social psychology research is probably on average compromised by political bias leftward. Conservative researchers are likely discriminated against in at least this field. More importantly papers and research that does not fit a liberal perspective faces greater barriers and burdens.
An article in the online publication inside higher ed on a survey on anti-conservative bias among social psychologists.
The link above is worth following. The problems that arise remind me of the situation with academic and our own ethics in light of this paper.
I can't help but think that self-assessments are probably too generous. For predictive power of how an individual behaves when the behaviour in question is undesirable, I'm more likely to take their estimate of how "colleagues" behave than their estimate of how they personally do.
This shouldn't be surprising to hear since to quote CharlieSheen: "we even have LW posters who have in academia personally experienced discrimination and harassment because of their right wing politics."
While I can see Lammers' point that this as disturbing from a fairness perspective to people grinding their way through academia and should serve as warning for right wing LessWrong readers working through the system, I find the issue of how this our heavy reliance on academia for our map of reality might lead to us inheriting such distortions of the map of reality much more concerning. Overall in light of this if a widely accepted conclusion from social psychology favours a "right wing" perspective it is more likely to be correct than if no such biases against such perspectives existed. Conclusions that favour "left wing" perspective are also somewhat less likely to be true than if no such biases existed. We should update accordingly.
I also think there are reasons to think we may have similar problems on this site.