cousin_it comments on How not to move the goalposts - Less Wrong

4 Post author: HopeFox 12 June 2011 03:45PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (71)

You are viewing a single comment's thread. Show more comments above.

Comment author: cousin_it 12 June 2011 10:01:47PM *  5 points [-]

If you successfully design your C++ hiring criteria to be colorblind — to not notice the color of weasels, but only to notice how good they are at C++ — then performance on the hiring criteria will shadow weasel color as an indicator of C++ ability.

My comment kinda assumed that hiring criteria meeting your strict standard of colorblindness are unexpectedly hard to design. Let's say all red weasels and most blue ones suck at C++, but some blue weasels completely rule. Also, once a month every weasel (blues and reds equally) unpredictably goes into a code frenzy for 182 minutes and temporarily becomes exactly as good as a blue one that rules. Your standardized test will mostly admit blue weasels that rule, but sometimes you'll get a random-colored weasel that sucks. If you're colorblind, you have no hope of weeding out the random suckers. But if you're color-aware, you can weed out half of them. Of course it also works if a tiny minority of red weasels can code instead of none.

Comment author: taryneast 13 June 2011 01:12:46PM 2 points [-]

The problems begin when the ability-distribution of red/blue weasels change, and the hiring-committee is still using restrictions based on the old distribution. eg red weasel ability has been steadily increasing, but the old hiring criteria still says "don't hire red weasels as they have no technical ability to speak of!"

but yes I agree - it's all difficult because it's hard to create a test that is as accurate as actual real-life working with a person. That's why the popularity of those awful "three month probation periods".