fubarobfusco comments on Checking for the Programming Gear - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (53)
The idea of programming as a gear is still controversial, but the specific hypothesized gear is that people who can build a consistent model of a language will be successful at programming, whereas those who can't won't be. This was tested by giving students a test on Java before they had been taught java; their answers were checked, not for correctness, but for consistency. See "The Camel Has Two Humps." Even then the test is far from perfectly predictive- ~28% of the consistent group failed, ~19% of the inconsistent group passed, and membership in the groups as indicated by the tests assigned shifted over time. If you do want to test this, you can reuse the original test.
However, there have been numerous attempted replications, none of which succeeded- though none found a negative result either. They were generally either confounded by the presence of experienced programmers, setup poorly, or not statistically significant. To quote the original authors:
HT Gwern
It irritates me to no end that the original study is so much better known than the utter failure to replicate. I have to suspect that this has something to do with how conveniently it fits many programmers' notion that programmers are a special sort of person, possessed of some power beyond merely a lot of practice at programming and related skills.
I feel much the same way about dual n-back studies. There was an article this month or last about WM training with Jaeggi and Buschkuel as authors... and it mentioned not a single issue. Gah!
The more recent meta-analysis appears to support their initial conclusion.
I think that interesting results which fail to replicate are almost always better-known than the failure to replicate. I think it's a fundamental problem of science, rather than a special weakness of programmers.