qsz comments on How subjective is attractiveness? - Less Wrong

23 Post author: JonahSinick 13 January 2015 12:04AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (38)

You are viewing a single comment's thread. Show more comments above.

Comment author: [deleted] 16 January 2015 12:17:52PM *  3 points [-]

There's plenty of research on reliability of rating scales - and the sweet spot seems to be a range from 7-10 choices at least according to quite a few studies designed to address this directly. An influential paper in this regard is Preston & Coleman's (2000) "Optimal number of response categories in rating scales: reliability, validity, discriminating power, and respondent preferences." link to PDF

Abstract:

Using a self-administered questionnaire, 149 respondents rated service elements associated with a recently visited store or restaurant on scales that differed only in the number of response categories (ranging from 2 to 11) and on a 101-point scale presented in a different format. On several indices of reliability, validity, and discriminating power, the two-point, three-point, and four-point scales performed relatively poorly, and indices were significantly higher for scales with more response categories, up to about 7. Internal consistency did not differ significantly between scales, but test-retest reliability tended to decrease for scales with more than 10 response categories. Respondent preferences were highest for the 10-point scale, closely followed by the seven-point and nine-point scales.

Or if one prefers a more analytic approach, here's a 2012 conference proceedings paper by Kluver et al "How many bits per rating?" link to PDF