When these people encounter this question, they're slogging through this huge survey. They're not doing an IQ test. This is more casual. They're being asked stuff like "How many partners do you have?" By the time they get down to that question, they're probably in a casual answering mode, and they're probably a little tired and looking for an efficient way to finish... If we really want to test them, we need to make it clear that we're testing them.

My first thought about this is that people's rationality 'in real life' totally is determined by how likely they are to notice a Bayes question in an informal setting, where they may be tired and feeling mentally lazy. In Keith Stanovich's terms, rationality is mostly about the reflective mind: it's someone's capacity and habits to re-compute a problem's answer, using the algorithmic mind, rather than accept the intuitive default answer that their autonomous mind spits out.

IQ tests tend to be formal; it's very obvious that you're being tested. They don't measure rationality in the sense that most LWers mean it; the ability to apply thinking techniques to real life in order to do better.

It might still be valuable to know how LWers do on a more formal test of probability-related knowledge; after all, most people in the general public don't know Bayes' theorem, so it'd be neat to see how good LW is at increasing "rationality literacy". But that's not the ultimate goal. There are reasons why you might want to measure a group's ability to pick out unexpected rationality-related problems and activate the correct mindware. If your Bayesian superpowers only activate when you're being formally tested, they're not all that useful as superpowers.