gwern comments on 2012 Survey Results - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (640)
I previously mentioned that item non-response might be a good measure of Conscientiousness. Before doing anything fancy with non-response, I first checked that there was a correlation with the questionnaire reports. The correlation is zero:
I am completely surprised. The results in the economics paper looked great and the rationale is very plausible. Yet... The 2 sets of data here have the right ranges, there's plenty of variation in both dimension, I'm sure I'm catching most of the item non-responses or NAs given that there are non-responses as high as 34, there's a lot of datapoints, and it's not that the correlation is the opposite direction which might indicate a coding error but that there's none at all. Yvain questions the Big Five results, but otherwise they look exactly as I would've predicted before seeing the results: low C and E and A, high O, medium N.
There may be something very odd about LWers and Conscientiousness; when I try C vs Income, there's a almost-zero correlation again:
I guess the next step is a linear model on income vs age, Conscientiousness, and IQ:
So all of them combined don't explain much and most of the work is being done by the age variable... There's many high-income LWers, supposedly (in this subset of respondents reporting age, income, IQ, and Conscientiousness, the max is 700,000), so I'd expect a cumulative r^2 of more than 0.173 for all 3 variables; if those aren't governing income, what is? Maybe everyone working with computers is rich and the others poor? Let's look at everyone who submitted salary and profession and see whether the practical computer people are making bank:
Wow. Just wow. 76k vs 43k. I mean, maybe this would go away with enough fiddling (eg. cost-of-living) but it's still dramatic. This suggests a new theory to me: maybe Conscientiousness does correlate with income at its usual high rate for everyone but computer people who are simply in so high demand that lack of Conscientiousness doesn't matter:
So for the CS people the correlation is small and non-statistically-significant, for non-CS people the correlation is almost 3x larger and statistically-significant.
There is a correlation of 0.13 between non-responses and N.
Of course, there's also a correlation of -0.13 between C and the random number generator.
People who had seen the RNG give a large number were primed to feel unusually reckless when taking the Big 5 test. Duh. (Just kidding.)
I am also surprised by this. I wonder about the effect of "I'm taking this survey so I don't have to go to bed / do work / etc.," but I wouldn't have expected that to be as large as the diligence effect.
Also, perhaps look at nonresponse by section? I seem to recall the C part being after the personality test, which might be having some selection effects.
What do you mean? I can't compare non-response with anyone who didn't supply a C score, and there were plenty of questions to non-response on after the personality test section.
It seems to me that other survey non-response may be uncorrelated with C once you condition on taking a long personality survey, especially if the personality survey doesn't allow nonresponse. (I seem to recall taking all of the optional surveys and considering the personality one the most boring. I don't know how much that generalizes to other people.) The first way that comes to mind to gather information for this is to compare the nonresponse of people who supplied personality scores and people who didn't, but that isn't a full test unless you can come up with another way to link the nonresponse to C.
I was thinking it might help to break down the responses by section, and seeing if nonresponse to particular sections was correlated with C, but the result could only be that some sections are anticorrelated if a few are correlated. So that probably won't get you anything.
Why would the strong correlation go away after adding a floor? That would simply restrict the range... if that were true, we'd expect to see a cutoff for all C scores but in fact we see plenty of very low C scores being reported.
Yes. You'd expect, by definition, that people who answered the personality questions would have fewer non-responses than the people who didn't... That's pretty obvious and true:
Were you expecting that people with high C would or wouldn't skip questions? I can see arguments either way. Conscientious people might skip questions they don't have answers to or that they aren't willing to put the time into to give a good answer, or they might put in the work to have answers they consider good to as many questions as possible.
Is it feasible to compare wrong sort of answer with C?
Is it possible that the test for C wasn't very good?
Wouldn't; that was the claim of the linked paper.
Not really, if it wasn't caught by the no-answer check or the NA check.
As I said, it came out as expected for LW as a whole, and it did correlate with income once the CS salaries were removed... Hard to know what ground-truth there could be to check the scores against.