I didn't have in mind just psychology; I was responding to your comment about soft and wannabe-hard fields in general. In particular, this struck me as unwarranted optimism:
[A] paper that uses statistics in a flawed fashion is indicative of how much progress the soft sciences have made in terms of being real sciences in that one needs bad stats to get bad ideas through rather than just anecdotal evidence.
That is true if these sciences are nowadays overwhelmingly based on sound math and statistics, and these bad stats papers are just occasional exceptions. The pessimistic scenario I have in mind is the emergence of bogus fields in which bad formalism is the standard -- i.e., in which verbal bad reasoning of the sort seen in, say, old-school Freudianism is replaced by standardized templates of bad formalism. (These are most often, but not always, in the form of bad statistics.)
This, in my opinion, results in an even worse situation. Instead of bad verbal reasoning, which can be criticized convincingly in a straightforward way, as an outside critic you're now faced with an abstruse bad formalism. This not only makes it more difficult to spot the holes in the logic, but even if you identify them correctly, the "experts" can sneer at you and dismiss you as a crackpot, which will sound convincing to people who have't taken the effort to work through the bad formalism themselves.
Unless you believe that such bogus fields don't exist (and I think many examples are fairly obvious), they are clear counterexamples to your above remark. Their "mathematization" has resulted in bullshit being produced in even greater quantities, and shielded against criticism far more strongly that if they were still limited to verbal sophistry.
Another important point, which I think you're missing, concerns your comment about problematic fields having a relatively small, and arguably less important scope relative to the (mostly) healthy hard fields. The trouble is, the output of some of the most problematic fields is used to direct the decisions and actions of the government and other powerful institutions. From miscarriages of justice due to pseudoscience used in courts to catastrophic economic crises, all kinds of calamities can directly follow from this.
No substantial disagreement with most of your comment. I will just note that most of your points (which do show that I was being overly optimistic) don't as a whole substantially undermine the basic point being made about Eliezer's claim.
I think your point about small fields being able to do damage is an interesting one (and one I've never seen before) and raises all sorts of issues that I'll need to think about.
Related to: Parapsychology: the control group for science, Dealing with the high quantity of scientific error in medicine
Some of you may remember past Less Wrong discussion of the Daryl Bem study, which claimed to show precognition, and was published with much controversy in a top psychology journal, JPSP. The editors and reviewers explained their decision by saying that the paper was clearly written and used standard experimental and statistical methods so that their disbelief in it (driven by physics, the failure to show psi in the past, etc) was not appropriate grounds for rejection.
Because of all the attention received by the paper (unlike similar claims published in parapsychology journals) it elicited a fair amount of both critical review and attempted replication. Critics pointed out that the hypotheses were selected and switched around 'on the fly' during Bem's experiments, with the effect sizes declining with sample size (a strong signal of data mining). More importantly, Richard Wiseman established a registry for advance announcement of new Bem replication attempts.
A replication registry guards against publication bias, and at least 5 attempts were registered. As far as I can tell, at the time of this post the subsequent replications have, unsurprisingly, failed to replicate Bem's results.1 However, JPSP and the other high-end psychology journals refused to publish the results, citing standing policies of not publishing straight replications.
From the journals' point of view, this (common) policy makes sense: bold new claims will tend to be cited more and raise journal status (which depends on citations per article), even though this means most of the 'discoveries' they publish will be false despite their p-values. However, this means that overall the journals are giving career incentives for scientists to massage and mine their data for bogus results, but not to challenge bogus results by others. Alas.
1 A purported "successful replication" by a pro-psi researcher in Vienna turns out to be nothing of the kind. Rather, it is a study conducted in 2006 and retitled to take advantage of the attention on Bem's article, selectively pulled from the file drawer.
ETA: The wikipedia article on Daryl Bem makes an unsourced claim that one of the registered studies has replicated Bem.
ETA2: Samuel Moulton, who formerly worked with Bem, mentions an unpublished (no further details) failed replication of Bem's results conducted before Bem submitted his article (the failed replication was not mentioned in the article).
ETA3: There is mention of a variety of attempted replications at this blog post, with 6 failed replications, and 1 successful replication from a pro-psi researcher (not available online). It is based on this ($) New Scientist article.
ETA4: This large study performs an almost straight replication of Bem (same methods, same statistical tests, etc) and finds the effect vanishes.
ETA5: Apparently, the mentioned replication was again submitted to the British Journal of Psychology: