thomblake comments on 2011 Less Wrong Census / Survey - Less Wrong

77 Post author: Yvain 01 November 2011 06:28PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (694)

You are viewing a single comment's thread. Show more comments above.

Comment author: Viliam_Bur 01 November 2011 12:59:10PM 16 points [-]

I think it is generally good to avoid "other" options as much as possible.

There are a few biases related to filling questionnaires. For example, many psychological tests ask you the same question twice, in opposite direction. (Question #13 "Do you think Singularity will happen?" Question #74: "Do you think Singularity will never happen?") This is because some people use heuristics "when unsure, say yes" and some other people use heuristics "when unsure, say no". So when you get two "yes" answers or two "no" answers to opposite forms of the question, you know that the person did not really answer the question.

Another bias is that when given three choices "yes", "no" and "maybe", some people will mostly choose "yes" or "no" answers, while others will prefer "maybe" answers. It does not necesarily mean that they have different opinions on the subject. It may possibly mean that they both think "yes, with 80% certainty", but for one of them this means "yes", and for the other one this means "maybe". So instead of measuring their opinions on the subject, you are measuring their opinions on how much certainty is necessary to answer "yes" or "no" in the questionnaire.

Perhaps in some situations the "other" option is necessary, because for some people none of the available options is good even as a very rough approximation. But I think it should be used very carefully, because it encourages the "I am a special snowflake" bias. For example, if someone has no sexual feelings at all, then of course the "monogamy or polygamy" question does not make sense for them. But if it is "I like the idea of being in love with one special person, but I also like the idea of having sexual access to many attractive people" then IMHO this attitude does not deserve a separate category and can be rounded towards one of the choices.

Comment author: thomblake 01 November 2011 11:02:45PM 5 points [-]

There's an established way of correcting for this in market research (and other fields): coding. Let's say you have the following list:

  1. utilitarian
  2. deontologist
  3. virtue ethics
  4. other (please specify)

Then you have someone go through all the typed-in responses, and when someone types in "special snowflake utilitarian" you code that as a 1 rather than a 4.

This is also done for completely open-end responses. Sometimes something like "additional comments" will on the back end look like:

  1. positive
  2. negative
  3. neutral