Wei_Dai comments on How not to move the goalposts - Less Wrong

4 Post author: HopeFox 12 June 2011 03:45PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (71)

You are viewing a single comment's thread. Show more comments above.

Comment author: Wei_Dai 12 June 2011 10:13:54PM 9 points [-]

The problem seems even worse than that. Suppose I can somehow magically determine the actual C++ ability of any weasel, and hire the first ten I come across that is above some threshold, then someone who doesn't have my magical ability would still (rationally) expect that the average skill among red weasels that I hire is lower than the average skill among blue weasels that I hire. (And I would expect this myself before I started the hiring process.) Similarly if decide to gather some fixed number of candidates and hire the top 10%.

One way Perplexed could be right is if I have the magical ability (or a near perfect test), and I decide to hire only weasels whose C++ ability is exactly X (no higher and no lower), but that seems rather unrealistic. What other situations could produce the result that Perplexed claimed?

Comment author: Perplexed 12 June 2011 11:56:39PM 5 points [-]

Ouch. I wish I had read this before dismissing cousin_it's citation of Robin's point.

Ok, my nothing was unjustified. Though I will point out that if I hire for a second-tier engineering organization which pays no higher than it has too, then the blue weasels that I hire will probably not be much better than the red weasels. All the blue super-weasels will get jobs elsewhere. In fact, it will be found that they are not weasels at all, but rather martins or minks.

Comment author: prase 14 June 2011 04:28:35PM 2 points [-]

True, under the assumption that the weasels are selected only by the threshold test. Actually, since time immemorial red weasels program in Fortran and C++ is thought to be a blue weasel domain. Therefore few red weasels actually plan to be hired as a C++ programmer, only those who are extraordinarily apt apply for such a job. As a coincidence, among the weasels who apply the average C++ ability is significantly higher withing the red subset.

Comment author: FAWS 12 June 2011 11:45:39PM *  2 points [-]

Depends on the distribution of C++ ability. Suppose C++ weasels are a mix of normal weasels and geniuses, that geniuses have far higher ability than normal weasels, and that across both groups blue weasels are on average better by a constant considerably smaller than that difference. Your test could leave you with mostly genius red weasels and a mix of normal and genius blue weasels such that the average ability of red weasels who pass is higher.

Alternatively if far fewer red weasels learn C++ and the red weasels who do are selected for aptitude the average aptitude of red weasels who learn C++ could be higher than that of blue weasels.

Comment author: cousin_it 12 June 2011 10:19:59PM *  0 points [-]

Thanks! You could produce Perplexed's claimed outcome by fiat: use your magic detector to hire weasels so that they fit the desired distribution :-) Or you could set the threshold higher for red weasels and get the same result. Both options seem unsatisfactory...