Even worse than language difficulties, I would think, would be large differences in cultural framing of questions. Every culture brings a different set of background issues to the types of questions asked in many psychological studies. The problem has mostly been solved for IQ type tests, but, even without considering the amount of work involved in developing the cross-cultural IQ tests, framing would be a bigger problem for personality and other "softer" tests. (I have, but have only leafed through, Jensen's "Bias in Mental Testing"; I can already tell it's going to take a lot of work, and it's a bit dated, so I've been putting it off since it's only a peripheral interest.)
What evidence do you have that "the problem has mostly been solved for IQ type tests"?
Sorry, that sounded challenging, and it isn't meant to be. Would you please point me to any books, papers, and so on?
Most of the research on cognitive biases and other psychological phenomena that we draw on here is based on samples of students at US universities. To what extent are we uncovering human universals, and to what extent facts about these WEIRD (Western, Educated, Industrialized, Rich, and Democratic) sample sources? A paper in press in Behavioural and Brain Sciences the evidence from studies that reach outside this group and highlights the many instances in which US students are outliers for many crucial studies in behavioural economics.
Epiphenom: How normal is WEIRD?
Henrich, J., Heine, S. J., & Norenzayan, A. (in press). The Weirdest people in the world? (PDF) Behavioral and Brain Sciences.
Broad claims about human psychology and behavior based on narrow samples from Western societies are regularly published in leading journals. Are such species-generalizing claims justified? This review suggests not only that substantial variability in experimental results emerges across populations in basic domains, but that standard subjects are in fact rather unusual compared with the rest of the species - frequent outliers. The domains reviewed include visual perception, fairness, categorization, spatial cognition, memory, moral reasoning and self‐concepts. This review (1) indicates caution in addressing questions of human nature based on this thin slice of humanity, and (2) suggests that understanding human psychology will require tapping broader subject pools. We close by proposing ways to address these challenges.