gwern comments on LW Women: LW Online - Less Wrong

29 [deleted] 15 February 2013 01:43AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (590)

You are viewing a single comment's thread. Show more comments above.

Comment author: gwern 16 February 2013 07:39:12PM 1 point [-]

In general, the reliability coefficient doesn't take into account extra distributional knowledge. If you knew that scores were power-law distributed in the population but the test error were normally distributed, for example, then you would want to calculate the posterior the long way: with the population data as your prior distribution and the the measurement distribution as your likelihood ratio distribution, and the posterior is the renormalized product of the two. I don't think that using a linear correction based on the reliability coefficient would get that right, but I haven't worked it out to show the difference.

That makes sense, but I think the SAT is constructed like IQ tests to be normally rather than power-law distributed, so in this case we get away with a linear correlation like reliability.