when framed in terms of social interactions, people's performance dramatically improves
From the Wikipedia article, after invoking evolutionary psychology and social interaction to explain the improvement:
Alternatively, it could just mean that there are some linguistic contexts in which people tend to interpret "if" as a material conditional, and other linguistic contexts in which its most common vernacular meaning is different.
It shouldn't be hard to present the test as a real world example that doesn't involve social interaction (e.g. "If lights are on, there is electricity in the house").
/me goes off to test this on a couple of linguistics students
Result: One correct and one incorrect answer.
Two results isn't enough to get a hold of probabilities like 40% and 70%; can we get ten linguistics students surveyed? I know three and could test them. Can you describe the test in more detail?
The Wason Selection Task is the somewhat famous experimental problem that requires attempting to falsify a hypothesis in order to get the correct answer. From the wikipedia article:
Aside from an illustration of the rampancy of confirmation bias (only 10-20% of people get it right), the task is interesting for another reason: when framed in terms of social interactions, people's performance dramatically improves:
However, apparently psychopaths perform nearly as badly on the "social contract" versions of this experiment as they do on the normal one. From the Economist:
The original (gated) research appears to be here.