Benja comments on Don't Revere The Bearer Of Good Info - Less Wrong

82 Post author: CarlShulman 21 March 2009 11:22PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (64)

You are viewing a single comment's thread. Show more comments above.

Comment author: pnrjulius 28 May 2012 11:21:49PM 0 points [-]

How about this? "The study had 27 participants, 15 women and 12 men. The difference between men and women was on average 2 points on a questionnaire ranging from 0 to 100 points." This clearly explains the (small) sample size and (weak) effect size without requiring any complicated statistics.

Comment author: Benja 16 August 2012 07:10:47PM *  3 points [-]

Actually, it doesn't tell you the effect size, since it doesn't include information about how much individuals in each group differ from each other. If the difference between the group means is 2 points and the standard deviation in each group is 5 points, that's the same effect size (in the technical Cohen sense) as if the difference is 10 points and the standard deviation is 25 points.

I think a useful way to report data like this might be a variation on, "If you chose one of the women and random and one of the men at random, the probability that the woman would have a higher score would be 53%."

Aaaand in order not to completely miss the point of the original article, ETA: I'm not sure how much of that suggestion is my own thinking, but I was certainly influenced by reading about the binomial effect size display which solves a related problem in a similar way, and after I had the idea myself I came across something very similar in Rebecca Jordan-Young's Brainstorm (p.52, and endnote 4 on p.299): mental rotation ability is "considered to be the largest and most reliable gender difference in cognitive ability"; using data from a meta-analysis, she notes that if you tried to guess someone's gender based on their score in a mental rotation test, using your best strategy you'd get it right 60% of the time. (I checked that math a while ago and got the same result, assuming normal distributions with equal variances in each group, with Cohen's d=.56; the meta-analysis is Voyer, Voyer & Bryden, 1995.)

It's annoying that IIRC, "guess the gender" and "in a random pair, who has the higher score" don't give the same number, though. Average readers will probably just see a percentage in each case and derive some measure of affect from the number, whichever interpretation you give.