Study shows existence of psychic powers.

bentarm

According to the New Scientist, Daryl Bern has a paper to appear in Journal of Personality and Social Psychology, which claims that the participants in psychological experiments are able to predict the future. A preprint of this paper is available online. Here's a quote from the New Scientist article:

In one experiment, students were shown a list of words and then asked to recall words from it, after which they were told to type words that were randomly selected from the same list. Spookily, the students were better at recalling words that they would later type.

In another study, Bem adapted research on "priming" – the effect of a subliminally presented word on a person's response to an image. For instance, if someone is momentarily flashed the word "ugly", it will take them longer to decide that a picture of a kitten is pleasant than if "beautiful" had been flashed. Running the experiment back-to-front, Bem found that the priming effect seemed to work backwards in time as well as forwards.

Question: even assuming the methodology is sound, given experimenter bias, publication bias and your priors on the existence of psi, what sort of p-values would you need to see in that paper in order to believe with, say, 50% probability that the effect measured is real?

In one experiment, students were shown a list of words and then asked to recall words from it, after which they were told to type words that were randomly selected from the same list. Spookily, the students were better at recalling words that they would later type.

In another study, Bem adapted research on "priming" – the effect of a subliminally presented word on a person's response to an image. For instance, if someone is momentarily flashed the word "ugly", it will take them longer to decide that a picture of a kitten is pleasant than if "beautiful" had been flashed. Running the experiment back-to-front, Bem found that the priming effect seemed to work backwards in time as well as forwards.

what sort of p-values would you need to see in that paper in order to believe with, say, 50% probability that the effect measured is real?

A lot more than one study. But the other issue is, if the measured effect is real, the correct response is, "I see that I am confused," not, "They must have psychic powers!" Unexpected stuff happening in an experiment is a rather more probable prior than people having psychic powers. Moreover, you'd need very precise experiments to figure out exactly what's going on. This experiment may in fact show more about the invalidity of priming - if priming works in either direction, and people don't have psychic powers, then something other than the priming is probably responsible for their reactions.

It is a very, very common error in psychology to take limited experimental evidence with "statistical significance" and then infer extremely complicated and precise notions of human psychology from those, despite the fact that such precise explanations would require vastly more evidence. (A hypothetical example would be: Men respond to red letters at a faster rate than women, as compared with blue letters. Therefore, the color red must have had some specific importance in hunting during our evolutionary past. The latter observation is an arbitrarily privileged hypothesis with no real evidenciary support over thousands of other hypotheses. I've seen more absurd jumps, as well as people simply ignoring evidence that did not conform to their prior theory.)

Voted up because I agree that there's way too much inference from very limited psychological experiments.