= 27d567bc2d48d9f40e9ccc20ea370b15)
DaFranker comments on 2012 Survey Results - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (640)
On IQ Accuracy:
As Yvain says, "people have been pretty quick to ridicule this survey's intelligence numbers as completely useless and impossible and so on" because if they're true, it means that the average LessWronger is gifted. Yvain added a few questions to the 2012 survey, including the ACT and SAT questions and the Myers-Briggs personality type question that I requested (I'll explain why this is interesting), and that give us a few other things to check against, which has made the figures more believable. The ridicule may be an example of the "virtuous doubt" that Luke warns about in Overconfident Pessimism, so it makes sense to "consider the opposite":
The distribution of Myers-Briggs personality types on LessWrong replicates the Mensa pattern. This is remarkable since the patterns of personality types here are, in many significant ways, the exact opposite of what you'd find in the regular population. For instance, the introverted rationalists and idealists are each about 1% of the population. Here, they are the majority and it's the artisans and guardians who are relegated to 1% or less of our population.
Mensa's personality test results were published in the December 1993 Mensa Bulletin. Their numbers.
So, if you believe that most of the people who took the survey lied about their IQ, you also need to believe all of the following:
That most of these people also realized they needed to do IQ correlation research and fudge their SAT and ACT scores in order for their IQ lie to be believable.
Some explanation as to why the average of lurker's IQ scores would come out so close to the average of poster's IQ scores. The lurkers don't have karma to show off, and there's no known incentive good enough to get so many lurkers to lie about their IQ score. Vaniver's figures.
Some explanation for why the personality type pattern at LessWrong is radically different from the norm and yet very similar to the personality type pattern Mensa published and also matched my predictions. Even if they had knowledge of the Mensa personality test results and decided to fudge their personality type responses, too, they somehow managed to fudge them in such a way that their personality types accidentally matched my predictions.
That they decided not to cheat when answering the Bayes birthday question even though they were dishonest enough to lie on the IQ question, motivated to look intelligent, and it takes a lot less effort to fudge the Bayes question than the intelligence and personality questions. (This was suggested by ArisKatsaris).
That both posters and lurkers had some motive strong enough to justify spending 20+ minutes doing the IQ correlation research and fudging personality test questions while probably bored of ticking options after filling out most of a very long survey.
It's easier just to put the real number in the IQ box than do all that work to make it believable, and it's not like the liars are likely to get anything out of boasting anonymously, so the cost-benefit ratio is just not working in favor of the liar explanation.
If you think about it in terms of Occam's razor, what is the better explanation? That most people lied about their IQ, and fudged their SAT, ACT and personality type data to match, or that they're telling the truth?
Summary of criticism:
Possible Motive to Lie: The desire to be associated with a "gifted" group:
In re to this post, it was argued by NonComposMentis that a potential motive to lie is that if the outside world perceives LessWrong as gifted, then anyone having an account on LessWrong will look high-status. In rebuttal:
I figure that lurkers would not be motivated to fudge their results because they don't have a bunch of karma on their account to show off and anybody can claim to read LessWrong, so fudging your IQ just to claim that the site you read is full of gifted people isn't likely to be motivating. I suggested that we compare the average IQs of lurkers and others. Vaniver did the math and they are very, very close..
I argued, among other things, that it would be falling for a Pascal's mugging to believe that investing the extra time (probably at least $5 worth of time for most of us) into fudging the various different survey questions is likely to contribute to a secret conspiracy to inflate LessWrong's average IQ.
Did the majority avoid filling out intelligence related questions, letting the gifted skew the results?
Short answer: 74% of people answered at least one intelligence related question and since most people filled out only one or two, the fact that the self-report, ACT and SAT score averages are so similar is remarkable.
I realized, while reading Vaniver's post that if only 1/3 of the survey participants filled out the IQ score, this may have been due to something which could have skewed the results toward the gifted range, for instance, if more gifted people had been given IQ tests for schooling placement (and the others didn't post their IQ score because they did not know it) or if the amount of pride one has in their IQ score has a significant influence on whether one reported it.
So I went through the data and realized that most of the people who filled out the IQ test question did not fill out all the others. That means that 804 people (74% not 33%) answered at least one intelligence related question. As we have seen, the IQ correlations for the IQ, SAT and ACT questions were very close to each other (unsurprisingly, it looks like something's up with the internet test... removing those, it's 63% of survey participants that answered an intelligence related question). It's remarkable in and of itself that each category of test scores generated an average IQ so similar to the others considering that different people filled them out. I mean if 1/3 of the population filled out all of the questions, and the other 2/3 filled out none, we could say "maybe the 1/3 did IQ correlation research and fudged these" but if most of the population fills out one or two, and the averages for each category come out close to the averages for the other categories, why is that? How would that happen if they were fudging?
It does look to me like people gave whatever test scores they had and that not all the people had test scores to give but it does not look to me like a greater proportion of the gifted people provided an intelligence related survey answer. Instead it looks like most people provided an intelligence related survey answer and the average LessWronger is gifted.
Exploration of personality test fudging:
Erratio and I explored how likely it is that people could successfully fudge their personality tests and why they might do that.
There are a lot of questions on the personality test that have an obvious intelligence component, so it's possible that people chose the answer they thought was most intelligent.
There are also intelligence related questions where it's not clear which answer is most intelligent. I listed those.
The intelligence questions would mostly influence the sensing/intuition dichotomy and the thinking/feeling dichotomy. This does not explain why the extraversion/introversion and perceiving/judging results were similar to Mensa's.
Thanks for the analysis. I agree with your conclusion.
On a less relevant note, it does feel good to see more evidence that the community we hang out with is smart and awesome.
This also explains a lot of things. People regard IQ as if it is meaningless, just a number, and they often get defensive when intellectual differences are acknowledged. I spent a lot of time doing research on adult giftedness (though I'm most interested in highly gifted+ adults) and, assuming the studies were done in a way that is useful (I've heard there are problems with this), and my personal experiences talking to gifted adults are halfway decent as representations of the gifted adult population, there are a plethora of differences that gifted adults have. For instance, in "You're Calling Who A Cult Leader?" Eliezer is annoyed with the fact that people assume that high praise is automatic evidence that a person has joined a cult. What he doesn't touch on is that there are very significant neurological differences between people in just about every way you could think of, including emotional excitability. People assume that others are like themselves, and this causes all manner of confusion. Eliezer is clearly gifted and intense and he probably experiences admiration with a higher level of emotional intensity than most. If the readers of LessWrong and Hacker News are gifted, same goes for many of them. To those who feel so strongly, excited praise may seem fairly normal. To all those who do not, it probably looks crazy. I explained more about excitability in the comments.
I also want to say (without getting into the insane amount of detail it would take to justify this to the LW crowd - maybe I will do that later, but one bit at a time) that in my opinion, as a person who has done lots of reading about giftedness and has a lot of experience interacting with gifted people and detecting giftedness, the idea that most survey respondents are giving real answers on the IQ portion of the survey seems very likely to me. I feel 99% sure that LessWrong's average IQ really is in the gifted range, and I'd even say I'm 90%+ sure that the ballpark hit on by the surveys is right. (In other words, they don't seem like a group of predominantly exceptionally or profoundly gifted Einsteins or Stephen Hawkings, or just talented people at the upper ends of the normal range with IQs near 115, but that an average IQ in the 130's / 140's range does seem appropriate.)
This says nothing about the future though... The average IQ has been decreasing on each survey for an average of about two points per year. If the trend continues, then in as many years as LessWrong has been around, LessWrong may trend so far toward the mean that LessWrong will not be gifted anymore (by all IQ standards that is, it would still be gifted by some definitions and IQ standards but not others). I will be writing a post about the future of LessWrong very soon.
Would you predict then that people who're not gifted are in general markedly less inclined to praise things with a high level of intensity?
This seems to me to be falsified by everyday experience. See fan reactions to Twilight, for a ready-to-hand example.
My hypothesis would simply be that different people experience emotional intensity as a reaction to different things. Thus, some think we are crazy and cultish, while also totally weird for getting excited about boring and dry things like math and rationality... while some of us think that certain people who are really interested in the lives of celebrities are crazy and shallow, while also totally weird for getting excited about boring and bad things like Twilight.
This also leads each group to think that the other doesn't get similar levels of emotional intensity, because only the group's own type of "emotional intensity" is classified as valid intensity and the other group's intensity is classified as madness, if it's recognized at all. I've certainly made the mistake of assuming that other people must live boring and uninteresting lives, simply because I didn't realize that they genuinely felt very strongly about the things that I considered boring. (Obligatory link.)
(Of course, I'm not denying there being variation in the "emotional intensity" trait in general, but I haven't seen anything to suggest that the median of this trait would be considerably different in gifted and non-gifted populations.)
Ok, where do I find them?
If you have to go looking, you're lucky.
If you want to find them in person, the latest Twilight movie is still in theaters, although you've missed the people who made a point of seeing it on the day of the premier.
Haha, I guess so. I am very, very nerdy. I had fun getting worldly in my teens and early 20's, but I've learned that most people alienate me, so I've isolated myself into as much of an "ivory tower" as possible. (Which consists of me doing things like getting on my computer Saturday evenings and nerding so hard that I forget to eat.)
Not really.
What did they do when you saw them?
How do we distinguish the difference between the kind of fanaticism that mentally unbalanced people display for, say, a show that is considered by many to have unhealthy themes and the kind of excitement that normal people display for the things they love? Maybe Twilight isn't the best example here.
I didn't. I don't particularly have to go out of my way to find Twilight fans, but if I did, I wouldn't.
I think you're dramatically overestimating the degree to which fans of Twilight are psychologically abnormal. Harlequin romance was already an incredibly popular genre known for having unhealthy themes. Twilight, like Eragon, is a mostly typical work of its genre with a few distinguishing factors which sufficed to garner it extra attention, which expanded to the point of explosive popularity as it started drawing in people who weren't already regular consumers of the genre.
I wouldn't be surprised if this is true.
This still does not answer the question "What sample can we use that filters out fanaticism from mentally unbalanced people to compare the type of excitement that gifted people feel to the type of excitement that everyone else feels?" Not to assume that no gifted people are mentally unbalanced... I suppose we'd really have to filter those out of both groups.
Taboo "mentally unbalanced".
What distinction are you trying to make here?
we will all be brain-dead in 70 years.
It's true that the downward trend can't go on forever, and to say that it's definitely going to continue would be (all by itself, without some other arguments) an appeal to history or slippery slope fallacy. However, when we see a trend as consistent and as potentially meaningful as the one below, it makes sense to start wondering why it is happening:
IQ Trend Analysis
I was mostly just trying to point out that you are extrapolating from a sample size of three points. Three points which have a tremendous amount of common causes that could explain the variation. Furthermore you aren't extrapolating 10% further from the span of your data, which might be ok, but actually 100% further. You're extrapolating for as long as we have data, which is... absurd.
One, I am used to seeing the term "sample size" applied to things like the people being studied, not a number of points done in a calculation. If there is some valid use of the term "sample size" that I am not aware of would you mind pointing me in the correct direction?
Two, I am not sure where you're getting "three points" from. If you mean the amount of IQ points that LessWrong has lost on the studies, then it was 7.18 points, not three.
Two points per year, which could be explained in other ways, sure. No matter what the trend, it could be explained in other ways. Even if it was ten points per year we could still say something like "The smartest people got bored taking the same survey over and over and stopped." There are always multiple ways to explain data. That possibility of other explanations does not rule out the potential that LessWrong is losing intelligent people.
Not sure what these 10% and 100% figures correspond to. If I am to understand why you said that, you will have to be specific about what you mean.
Including all of the data rather than just a piece of the data is bad why?
Three points referred to the number of surveys taken, which I didn't bother to look up, but I believe is three.
10% and 100% referred to the time span over which these data points referred to, ie. three years. Basically, I might be OK with you making a prediction for the next three months (still probably not) but extrapolating for three years based on three years of data seems a bit much to me.
Oh I see. The problem here is that "if the trend continues" is not a prediction. "I predict the trend will continue" would be a prediction. Please read more carefully the next time. You confused me quite a bit.
If you're not making a prediction, then it's about as helpful as saying "If the moon crashes into North America next year, LW communities will largely cease to exist."
Looks like Aumann at work. My own readings, though more specifically on teenage giftedness in the 145+ range, along with stuff on ASD and asperger, heavily corroborate with this.
When I was 17, my (direct) family and I had strong suspicions that I was in this range of giftedness - suspicions which were never reliably tested, and thus neither confirmed nor infirmed. It's still up in the air and I still don't know whether I fit into some category of gifted or special individuals, but at some point I realized that it wasn't all that important and that I just didn't care.
I might have to explore the question a bit more in depth if I decide to return into the official educational system at some point (I mean, having a paper certifying that you're a genius would presumably kind of help when making a pitch at university to let you in without the prerequisite college credit because you already know the material). Just mentioning all of the above to explain a bit where my data comes from. Both of my parents and myself were all reading tons of books, references, papers and other information along with several interviews with various psychology professionals for around three months.
Also, and this may be another relevant point, the only recognized, official IQ test I ever took was during that time, and I had a score of "above 130"² (verbal statement) and reportedly placed in the 98th and 99th percentiles on the two sections of a modified WAIS test. The actual normalized score was not included in the report (that psychologist(?¹) sucked, and also probably couldn't do the statistics involved correctly in the first place).
However, I was warned that the test lost statistical significance / representativeness / whatever above 125, and so that even if I had an IQ of 170+ that test wouldn't have been able to tell - it had been calibrated for mentally deficient teenagers and very low IQ scores (and was only a one-hour test, and only ten of the questions were written, the rest dynamic or verbal with the psychologist). Later looking-up-stats-online also revealed that the test result distributions were slightly skewed, and that a resulting converted "IQ" of "130" on this particular test was probably more rare in the general population than an IQ of 130 normally represents, because of some statistical effects I didn't understand at the time and thus don't remember at all.
Where I'm going with this is that this doesn't seem like an isolated effect at all. In fact, it seems like most of North America in general pays way more attention to mentally deficient people and low IQs than to high-IQs and gifted individuals. Based on this, I have a pretty high current prior that many on LW will have received scores suffering from similar effects if they didn't specifically seek the sorts of tests recommended by Mensa or the likes, and perhaps even then.
Based on this, I would expect such effects to compensate or even overcompensate for any upward nudging in the self-reporting.
=====
I don't know if it was actually a consulting psychologist. I don't remember the title she had (and it was all done in French). She was "officially" recognized to be in legal capacity to administrate IQ tests in Canada, though, so whatever title is normally in charge of that is probably the right one.
Based on this, the other hints I mention in the text, and internet-based IQ tests consistently giving me 150-ish numbers when at peak performance and 135-ish when tired (I took those a bit later on, perhaps six months after the official one), 135 is the IQ I generally report (including in the LW survey) when answering forms that ask for it and seems like a fairly accurate guess in terms of how I usually interact with people of various IQ levels.