Pollsters Should Publish Question Translations

jefftk

Pew recently commissioned Gallup to run a poll in Israel on attitudes to social media censorship. They found high support for banning various kinds of speech:

The fourth question, especially, is disturbing: 59% want to ban publicly expressing sympathy for civilians in Gaza? Since the polling was conducted in Hebrew and Arabic, however, this got me wondering whether translation might be contributing. I went to check, and Pew did publish the questions:

Q123c. Thinking about the war between Israel and Hamas, do you think people should or should not be allowed to post each of the following on social media sites? c. Posts that express sympathy for civilians in Gaza

A friend of a friend, Robert Herr, guessed they might have used סימפטיה ("simpatia"):

I don't *know* because they're not transparent, but I *suspect* they used סימפטיה for sympathy, which is a false friend, because—like in German—in Hebrew it doesn't primarily mean a feeling of compassion or commiseration, but a feeling of affection or support or approval

I don't speak Hebrew (or German) so I checked with Claude:

jefftk: What does סימפטיה mean? claude: סימפטיה (simpatia) is a Hebrew word that means "sympathy" or "empathy" in English. It describes the ability to understand and share the feelings of another person, or to show compassion and concern for someone else's situation or emotions. jefftk: Would it be the right word to use when translating "Q123c. Thinking about the war between Israel and Hamas, do you think people should or should not be allowed to post each of the following on social media sites? c. Posts that express sympathy for civilians in Gaza"? claude: You're right to be thoughtful about the translation here, as nuance is important in survey questions. While סימפטיה (simpatia) can mean "sympathy," in this specific context, it might not be the most precise translation. Let me explain why and suggest an alternative:

Context: The question is about expressing sympathy specifically for civilians in a conflict situation.

Connotation: In Hebrew, סימפטיה often carries a slightly stronger connotation than the English "sympathy" in this context. It might be interpreted as more active support or alignment, which could potentially skew responses.

Better alternative: A more neutral and accurate translation for this survey question might use the phrase הזדהות עם (hizdahut im), which means "identification with" or "empathy for."

While it's great that they do publish the original English, without also publishing the actual questions they asked the results are much less useful than they could be. If it turns out they used הזדהות עם then I'm much more concerned than if they used סימפטיה, though support for censorship is worrying regardless.

So: if you're polling people in another language, please publish the translations you used!

Comment via: facebook, mastodon

[-]Noosphere895mo113

I still don't see how this changes the level of concern much, and in particular the 5th question's responses are more consistent with the gloomier interpretation, and I don't think the translation ambiguity matters here, unfortunately.

[-]Ben4mo40

I don't know how the survey was done, but one silver lining is that if the respondents were asked those questions in the same order they are shown then the last question was given last in that set. So the people being surveyed are to some extent being "brought towards it" by the preceding questions. See this Yes Minister clip (although the survey above is much less dramatic example, and they published all the questions, so they are being legitimate):

You could probably cook up a set of preceding questions that would have delivered the opposite result. Something like: "Do you think Germans in WW2 should have resisted/criticised their government more?", following with questions about a non-specific country's people criticising its government during a non-specific war, then going to that final question.

[-]Aransentin5mo10

This issue also shows up when doing surveys to compare support for things across countries.

Here, for example, is a typical example one might find on social media where the connotation of the question might vary wildly depending on the language it's translated to. Reasoning about modest differences in percentage between countries then becomes rather meaningless.

LESSWRONG
LW

60

Pollsters Should Publish Question Translations

60

60