You're looking at Less Wrong's discussion board. This includes all posts, including those that haven't been promoted to the front page yet. For more information, see About Less Wrong.

JoshuaZ comments on Satire of Journal of Personality and Social Psychology's publication bias - Less Wrong Discussion

26 Post author: CarlShulman 05 June 2012 12:08AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (7)

You are viewing a single comment's thread.

Comment author: JoshuaZ 05 June 2012 02:44:46AM 8 points [-]

The essential argument here is clever- their extremely high rate of turning up hypotheses that are confirmed suggest that they must have psychic abilities.

There's a legitimate criticism here. But it may not be as legitimate as it seems at first glance. The most obvious problem here is publication bias (very bad), but there are other explanations as well, such as a tendency to write papers after the fact like the result was the initial hypothesis (bad but not nearly as bad as publication bias in terms of the long-term damage). Also, in areas like psychology, there's another possibility- humans have some developed intuitions for psychological behavior. In that regard, we may do a better job intuiting the correct results in psychology than in other fields, and thus hypotheses are more likely to be correct in psychology. The result here is too extreme for this last bit to account for everything at hand, and certainly there are areas where human intuition about psychology is woefully bad, but this may account for a substantial fraction of what is going on here.

In any event, there really are serious problems in psychology as a subject as a whole, and the JPSP is not an isolated case.

Comment author: Will_Newsome 06 June 2012 06:08:45PM *  1 point [-]

In any event, there really are serious problems in psychology as a subject as a whole, and the JPSP is not an isolated case.

This is what I've been thinking quite often during Carl's mini-series --- beating up on people who believe in magical powers, or on a specific journal, is of course quite easy, but I'm worried that people won't catch the more general and significantly more important point --- e.g., that all those heuristics and biases results that are cited around here, might not be so trustworthy.

(ETA: Also, strange you wrote almost exactly the same comment I would have; I don't think we normally have similar intuitions &c.)

Comment author: beoShaffer 05 June 2012 04:34:21PM 1 point [-]

Somewhat related Einstein's Arrogance.