= 27d567bc2d48d9f40e9ccc20ea370b15)
AnnaSalamon comments on Ask LW: What questions to test in our rationality questionnaire? - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (45)
Good idea - I heartily support this effort. That being said...
I would recommend not making the survey too long, as that reduces the chance that people will actually bother completing it. The longer it is, the likelier it is that only the people who actually have an emotional investment in the subject will finish it, biasing the results.
Once we gather more data, this can be partially avoided by making it into one of those online tests. "How rational are you? Fill this survey and find out."
Some of the questions on "real-world success" are pretty odd - what does having a driver's license have to do with success? (Living in an urban environment with good public transport, I don't have a license simply because almost anything has felt like a better investment of time and money.) Number of best friends probably is more influenced by your personality and relationship style than rationality. If you ask for GPA, you need to provide some sort of a conversion chart between the systems used in different countries. Note also that not all countries place as high an emphasis on GPA as the US does - many Finnish high school and college students have probably never even had any reason to bother calculating their average, so the motivational factor may be lessened. To help correct for this, ask for one's country in the geographical questions.
For myself, I have a tendency to start reading non-fiction books and then put them down before I've finished them but after I feel I've already gotten their message. Might be good to ask for the amount of pages read instead of books finished, to adjust for this. (I have read hardly if any non-fiction books to the end in the last month, even though I do think I've read several hundred pages worth.) May also ask how many academic, peer-reviewed articles one has read in the last month.
I'm not sure how "how good-looking are you" is useful with fulfilling the stated goals of section E.
If good looks aren't strongly correlated with the answers to other questions, we can use the responses of groups of test-takers to "how good-looking are you?" to see whether that group, in aggregate, tends to be positively self-deceived. It isn't definitive, but if e.g. the group of respondants whose questionnaires are low in a particular factor also have the property that 50% of them regard themselves as being in the best-looking 10% of the population, while for the group whose questionnaires are high in that factor, only 10% regard themselves as being in the best-looking 10%... that would be evidence (though not definitive evidence) that the factor correlates with accurate self-assessment.
Not sure whether this is feasible, but could you use the results of this sort of overoptimism calibration to adjust other subjective performance measures for bias? Maybe if you had more of these sorts of question in different domains, and overoptimism were strongly correlated across them all?