I have some half-baked ideas about getting interesting information on lesswronger's political opinions.

My goal is to give everybody an "alien's eye" view of their opinions, something like "You hold position Foo on issue Bar, and justify it by the X books you read on Bar; but among the sample people who read X or more books on Bar, 75% hold position ~Foo, suggesting that you are likely to be overconfident".

Something like collecting:

• your positions on various issues

• your confidence in that position

• how important various characteristics are at predicting correct opinions on that issue (intelligence, general education, reading on the issue, age ("general experience"), specific work or life experience with the issue, etc.)

• How well you fare on those characteristics

• Whether you expect to be above or below average (for LessWrong) on those characteristics

• How many lesswrongers you expect will disagree with you on that issue

• Whether you expect those who disagree with you to be above or below average on the various characteristics

• How much you would be willing to change your mind if you saw surprising information

What data we could get from that

• Are differences in opinion due to different "criteria for rightness" (book-knowledge vs. experience), to different "levels of knowledge" (Smart people believe A, stupid people believe B), or to something else ?

Problems with this approach:

• Politics is the mind-killer. We may not want too much (or any) politics on LessWrong. If the data is collected anonymously, this may not be a huge problem.

• It's easier to do data-mining etc. with multiple-choice questions rather than with open-ended questions (because two people never answer the same thing, so it leaves space to interpretation), but doing that correctly requires very good advance knowledge of what possible answers exist.

• Questions would be veeery carefully phrased.

• Ideally I would want confidence factors for all answers, but the end result may be too intimidating :P (And discourage people from answering, which makes a small sample size, which means questionable results).

I would certainly be interested in seeing the result of such a survey, but for now my idea is too rough to be actionable - any suggestions ? Comments ?