I'm indulging in the simple pleasure of drawing large conclusions from a single study.... Why exams are nothing out of context:
the story about the maths ability of Brazilian street kids living in the in the favelas of Recife. This story helped both of us realise the importance of carrying out usability tests in context. Three researchers (see: Carraher, Carraher, and Schliemann 1985) carried out research with children aged 9 to 15. These kids had dropped out of school, and were selling sun screen, and chewing gum on the streets. The researchers worked out that they could set the kids questions by purchasing goods off them. For example, 1,000 minus 300 is the same as giving the kid a 1,000 Cruzeiros note for a product that costs 300 Cruzeiros. Multiplication can be done by asking the kids how much 3 of a product would cost. In these tests the Brazilian street kids scored 98%. But when they were put into a formalised test setting, and asked instead of how much would 3 apples cost or what 3×9 is, the kids performance dropped to just 37%.
What is scary is that the researchers later tested middle class children in a private school. These kids did very well in the formal exam. But when they had to do transactions with real money in the street, using the same maths, they failed in being able to do the transactions.
Is it possible that the correlation between g and success isn't about raw intelligence, it's about being able to access one's intelligence in situations (like classrooms) which involve thresholds for easily improving one's status?
Although I have to wonder if the real explanation was some sort of mixup, rather than a grand scale flaw in the testing methodology, I have met one person who I would estimate to have been at least as intelligent as myself, who revealed his tested IQ, and it was approximately half of my own. I don't remember if he ever mentioned what type of IQ test it was he took, but if the tests are at all properly designed, individuals of similar intelligence should not fall large distances apart from each other on opposite sides of the median depending on which test they took.