You're looking at Less Wrong's discussion board. This includes all posts, including those that haven't been promoted to the front page yet. For more information, see About Less Wrong.

gwern comments on LW Women: LW Online - Less Wrong Discussion

29 [deleted] 15 February 2013 01:43AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (590)

You are viewing a single comment's thread. Show more comments above.

Comment author: gwern 16 February 2013 07:39:12PM 1 point [-]

In general, the reliability coefficient doesn't take into account extra distributional knowledge. If you knew that scores were power-law distributed in the population but the test error were normally distributed, for example, then you would want to calculate the posterior the long way: with the population data as your prior distribution and the the measurement distribution as your likelihood ratio distribution, and the posterior is the renormalized product of the two. I don't think that using a linear correction based on the reliability coefficient would get that right, but I haven't worked it out to show the difference.

That makes sense, but I think the SAT is constructed like IQ tests to be normally rather than power-law distributed, so in this case we get away with a linear correlation like reliability.