Thanks Ben!

In this case, with only selectivity ratings, you can't do better than 50% accuracy (any person wants to date any other person with 50% probability). But with eye-color ratings, you can get it perfect.

Edit: I initially misread your remark. I tried to clarify the setup with:

In this blog post I’m restricting consideration to signals of the partners’ general selectivity and general desirability, without considering how their traits interact.

Is this ambiguous?

My impression is that there are significant structural correlations in your data that I don't really understand the impact of. (For instance, at least if everyone rates everyone, I think the correlation of attr with attrAvg is guaranteed to be positive, even if attr is completely random.)

I may not fully parse what you have in mind, but I excluded the rater and ratee from the averages. This turns out not to be enough to avoid contamination for subtle reasons, so I made a further modification. I'll be discussing this later, but if you're wondering about this particular point, I'd be happy to now.

The relevant code is here. Your remark prompted me to check my code by replacing the ratings with random numbers drawn from a normal distribution. Using 7 ratings and 7 averages, the mean correlation is 0.003, with 23 negative and 26 positive.

Nitpick: There are only 10 distinct such correlations that are not 1 by definition.

Thanks, that was an oversight on my part. I've edited the text.

Model accuracy actually isn't actually a great measure of predictive power, because it's sensitive to base rates. (You at least mentioned the base rates, but it's still hard to know how much to correct for the base rates when you're interpreting the goodness of a classifier.)

I suppressed technical detail in this first post to make it more easily accessible to a general audience. I'm not sure whether this answers your question, but I used log loss as a measure of accuracy. The differentials were (approximately, the actual final figures are lower):

For Men: ~0.690 to ~0.500. For Women: ~0.635 to ~0.567. For Matches: ~0.432 to ~0.349

I'll also be giving figures within the framework of recommendation systems in a later post.

As far as I know, if you don't have a utility function, scoring classifiers in an interpretable way is still kind of an open problem, but you could look at ROC AUC as a still-interpretable but somewhat nicer summary statistic of model performance.

Thanks, I've been meaning to look into this.