Douglas_Knight comments on Beware of WEIRD psychological samples - Less Wrong

38 Post author: ciphergoth 13 September 2009 11:28AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (28)

You are viewing a single comment's thread. Show more comments above.

Comment author: Douglas_Knight 14 September 2009 11:49:59AM 4 points [-]

This sort of thing only works if you don't get too much garbage in your replications.

You have way too much trust in the professors. Just a few students naive enough to do what they're supposed to would be an improvement on the status quo.

Comment author: gwern 08 September 2011 10:50:08PM 4 points [-]

The problem is, we already have replications being done by Indian and Chinese scientists and... they're not very good. Here's one: "Local Literature Bias in Genetic Epidemiology: An Empirical Evaluation of the Chinese Literature", 2005:

"We targeted 13 gene-disease associations, each already assessed by meta-analyses, including at least 15 non-Chinese studies. We searched the Chinese Journal Full-Text Database for additional Chinese studies on the same topics. We identified 161 Chinese studies on 12 of these gene-disease associations; only 20 were PubMed-indexed (seven English full-text). Many studies (14–35 per topic) were available for six topics, covering diseases common in China. With one exception, the first Chinese study appeared with a time lag (2–21 y) after the first non-Chinese study on the topic. Chinese studies showed significantly more prominent genetic effects than non-Chinese studies, and 48% were statistically significant per se, despite their smaller sample size (median sample size 146 versus 268, p < 0.001). The largest genetic effects were often seen in PubMed-indexed Chinese studies (65% statistically significant per se). Non-Chinese studies of Asian-descent populations (27% significant per se) also tended to show somewhat more prominent genetic effects than studies of non-Asian descent (17% significant per se)."

Comment author: Larks 14 September 2009 07:24:56PM 2 points [-]

The huge amount of data that could be gathered should allow for checking; data that is both different from what westerners would expect, and consistent over several independent students, is likely to be accurate. Or at least, not inaccurate because of lazy students.