This straightforward question has a couple of straightforward limitations. For one, we might expect people who are involved in almost any activity to say that they benefit from it; self-reported benefit does not necessarily indicate actual benefit.
Well, the serious issue that comes to mind is selection bias. If someone thinks that LW is a -3 for them, are they going to stick around and take the survey?
But I don't think that's as important for asking the question of whether people who are already here should try to increase or decrease their involvement in the community, so there's still value in doing this analysis.
Idea: for the survey, get users to put their names into a hash function (link from the survey) and then next year, use the same function, so that you can compare users year over year. If there's concern about this being dangerous for the main LW survey, that column could just be removed before that data is published, and only made available for use by CFAR. Or something. This isn't a fully-fleshed out idea, just a quick thought on how one might go about turning basic correlational data into more longitudinal data.
I bet we could do this (with less confidence, because of rare false positives) just by mass-comparing results across years. We collected a fair amount of stable information. Some other system would be nice, though, if it adds reliability and makes it easier for us to vary the survey format or contents over time.
Another option is to generate a token that the user can save and re-enter when they take the survey again the next year. Then the data could be tagged with a hash of the token.
You'd get some misplaced tokens, but it should be pretty safe, privacy-wise. I think the biggest concern would be that linked results could narrow down the possibilities for someone's real-life identity, especially if they had a significant change of demographic status such as moving from one place to another or getting married.
by Dan from CFAR
Abstract: Two measures of the practical benefits of rationality, one a self-report of the benefits of being part of the rationality community and the other a measure of how often a person adds useful techniques to their repertoire, were included on the 2013 Less Wrong survey. In-person involvement with LW/CFAR predicted both measures of benefits, with friendships with LWers and attending a CFAR workshop showing the strongest and most consistent effects. Online Less Wrong participation and background had weaker and less consistent effects. Growth mindset also independently predicted both measures of practical benefits, and on the measure of technique acquisition there was an interaction effect suggesting that in-person LW/CFAR involvement may be especially beneficial for people high in growth mindset. However, some caution is warranted in interpreting these correlational, self-report results.
Though I first found Less Wrong through my habit of reading interesting blogs, the main reason why I've gotten more and more involved in the rationality community is my suspicion that this rationality stuff might be pretty useful. Useful not only for thinking clearly about tricky intellectual topics, but also in ways that have more directly practical benefits.
CFAR obviously has similar interests, as it aims to create a community of people who are effective at acting in the world.
The 2013 LW census/survey provided an opportunity for us to probe how the rationality community is doing so far at finding these practical benefits, as it allowed us to survey a large cross section of the Less Wrong community. Unfortunately, there is not a standard, simple measure of practical benefits which we could just stick on the survey, and we were only able to use a correlational research design, but we sought to get some relevant information by coming up with two self-report questions to include on the survey.
One question was somewhat broader than the set of practical benefits that we were interested in and the other was somewhat narrower. First, there was a broad self-report question asking people how much they had benefited from being involved in the rationality community. Second, we asked people more narrowly how often they successfully added a useful technique or approach to their repertoire. We were primarily interested in seeing whether involvement in the LW community would predict practical benefits on these two measures, and (if so) which forms of involvement would have the strongest relationship to these benefits.
About 1400 people answered the relevant survey questions, including about 400 who have read the sequences, about 150 who regularly attend LW meetups, about 100 who have attended a full CFAR workshop, about 100 who interact with other LWers in person all the time, and about 50 who met a romantic partner through LW. The survey also included a brief scale measuring growth mindset, and a question about age.
Some methodological notes: In the body of this post I’ve tried to put the results in a format that’s relatively straightforward to interpret. More technical details and additional analyses are included in footnotes, and I can add more details in the comments. Note that the study design is entirely correlational, and the questions are all self-report (unlike last year’s questions, which included tests of standard biases). This gives some reason for caution in interpreting the results, and I’ll note some places where that is especially relevant.
The simple, obvious thing to do, in order to investigate how much people have benefited from their involvement in the rationality community, is to ask them that question. So we did: "How much have you benefited from your exposure to and participation in the rationality community (Less Wrong, CFAR, in-person contact with LW/HPMOR readers, etc.)?" There were 7 response options, which we can scale as -3 to +3, where +3 is “My life is MUCH BETTER than it would have been without exposure to the rationality community” (and -3 is “... MUCH WORSE…”).
This straightforward question has a couple of straightforward limitations. For one, we might expect people who are involved in almost any activity to say that they benefit from it; self-reported benefit does not necessarily indicate actual benefit. Second, it could include a broad range of benefits, some of which might not have much to do with the usefulness of rationality (such as meeting your current romantic partner at a Less Wrong meetup). So we also included a narrower question related to competence which is less susceptible to these issues.
A simple model of how people are able to become highly competent/productive/successful/impressive individuals is that they try lots of things and keep doing the ones that work. A person’s work habits, the questions they ask during conversations, the methods that they use to make certain kinds of decisions, and many other things can all be developed through a similar iterative process. Over time, someone who has a good process in place of trying things & sticking with the helpful ones will end up collecting a large set of habits/techniques/approaches/principles/etc. which work for them.
The second set of questions which we included on the survey were based on this process, with the aim of measuring about how often people add a new useful technique to their repertoire. There were 3 survey questions based on a streamlined version of this process: first you hear about many different techniques, then you try some fraction of the techniques that you hear about, and then some fraction of the techniques that you try end up working for you and sticking as part of your repertoire. We first asked “On average, about how often do you *read or hear about* another plausible-seeming technique or approach for being more rational / more productive / happier / having better social relationships / having more accurate beliefs / etc.?”, then “...how often do you *try out* another plausible-seeming technique..”, and finally “...how often do you find another technique or approach that *successfully helps you at*...” This final question, about how frequently people acquire a new helpful technique, is our other main outcome measure of practical benefits.
In reality, people often generate their own ideas of techniques to try, and try many variations rather than just a single thing (e.g., many people end up with their own personalized version of the pomodoro technique). Focusing on the streamlined process of hear → try → acquire is a simplification which had two survey-specific benefits. First, having the context of “hearing about a technique and then trying it” was intended to make it clearer what to count as “a technique,” which is important since the outcome measure is a count of the number of techniques acquired. Second, including the “hearing” and “trying” questions allows us to probe this process in a bit more detail by (for example) breaking down the number of new techniques that a person acquired into two components: the number of new techniques that they tried and the hit rate (techniques acquired divided by techniques tried).
One other predictor variable which we included on the survey was a 4-item measure of growth mindset, which was taken from Carol Dweck’s research (sample item: “No matter what kind of person you are, you can always change substantially”).[1] A fixed mindset involves thinking that personal characteristics are fixed and unchangeable - you either have them or you don’t - while a growth mindset involves thinking that personal characteristics can change as a person grows and develops. Dweck and her colleagues have found that growth mindset about a characteristic tends to be associated with more productive behaviors and more improvement over time. For example, children with a growth mindset about being good at thinking tend to seek out intellectual challenges which stretch their abilities, while children with a fixed mindset tend to avoid tasks that they might fail at and seek tasks which they know they can do well.
A blog based on the idea of becoming less wrong sounds like it would reflect growth mindset more than fixed mindset, and many aspects of the local idea cluster seem to match that. Ideas like: there are systematic methods that you can learn which will allow you to form more accurate models of the world. Complex skills can be broken down into simple trainable components. Don't get too attached to a particular image of who you are and what you stand for. Mastering the right cognitive toolkit can make you more effective at accomplishing the things that you care about. Tsuyoku Naritai! In addition to these connections to LW thinking, growth mindset also seems like it could facilitate the process of becoming more successful by trying out various changes to the way that you do things and sticking with the ones that work. Thus, we wanted to investigate whether people who were more involved in the rationality community (in various ways) had more of a growth mindset, and whether people with more of a growth mindset reported more practical benefits.
The other main predictor variables were several different indicators of people's involvement in the rationality community:
LW background
A composite scale, which standardized and then averaged together four questions which all indicate a person’s amount of background with the lesswrong.com website (and which, as I found on previous years' surveys, all correlate with each other and show similar patterns of relationships with other variables). The four questions measured: having read the sequences (ranging from 1 “Never even knew they existed until this moment” to 7 “[Read] All or nearly all of the Sequences”), karma (log-transformed), LW Use (ranging from 1 “I lurk, but never registered an account” to 5 “I've posted in Main”), and length of time in the community (capped at 8 years).
Time per day on Less Wrong
“How long, in approximate number of minutes, do you spend on Less Wrong in the average day?” (log-transformed).
Meetup attendance
“Do you attend Less Wrong meetup?”
“Yes, regularly,” “Yes, once or a few times,” or “No” (categorical variable).
CFAR workshop attendance
Have you ever attended a CFAR workshop?
“Yes, I have been to a full (3+ day) workshop,” “I have been to at least one CFAR class, but not a full (3+ day) workshop,” or “No” (categorical variable).
LW friendships
“Is physical interaction with the Less Wrong community otherwise a part of your everyday life, for example you live with other Less Wrongers, or you are close friends and frequently go out with them?”
“Yes, all the time,” “Yes, sometimes,” or “No” (categorical variable).
LW romantic partner
Have you ever been in a romantic relationship with someone you met through the Less Wrong community?
“Yes,” “I didn't meet them through the community, but they're part of the community now,” or “No” (categorical variable).
I considered combining these four measures of in-person involvement with the LW community (LW meetups, CFAR workshops, LW friendships, and LW romantic partners) into a single scale of in-person LW involvement, but there ended up being a large enough sample size within these groups and strong enough effects for me to analyze them separately.
Respondents also reported their age (which was transformed by taking the square root).
I. Self-reported Benefit
"How much have you benefited from your exposure to and participation in the rationality community (Less Wrong, CFAR, in-person contact with LW/HPMOR readers, etc.)?"
The average response to this question was a 1.4 on a -3 to +3 scale (SD = 1.08), and 15% of people selected the scale maximum “My life is MUCH BETTER than it would have been without exposure to the rationality community.”
Which variables were associated with a larger self-reported benefit from the rationality community?
In short, all of them.
Each of the following variables was significantly related to this self-reported measure of benefit, and in a regression which controlled for the other variables all of them remained significant except for meetup attendance (which became p = 0.07).[2] For ease of interpretation, I have reported the percent of people in each of the following groups who selected the scale maximum. I have sorted the variables in order of effect size, from largest to smallest, based on the results of the regression (see the footnote for more details).
Percent of people in each subgroup answering “My life is MUCH BETTER than it would have been without exposure to the rationality community”
61% LW romantic partner (n = 54)
44% attended a full CFAR workshop (n = 100)
19% age 25 or less (younger people reported more benefit) (n = 724)
50% LW friendships (n = 88)
28% above 3.0 on growth mindset scale (n = 277)
25% high LW background (n = 137)
35% regularly attend meetups (n = 156)
31% acquire a new technique every 3 weeks or more often (n = 213)
18% use LW for 30+ min per day (n = 218)
15% all respondents (n = 1451)
Three noteworthy results:
One interpretation of these results is that getting involved in the rationality community causes people to acquire useful rationality skills which improve their lives, with larger effects for people who get involved in more depth through close relationships, shared housing, CFAR workshops, etc. However, as noted above, these effects could also be due to non-rationality-related benefits (e.g., finding friends or a romantic partner), a tendency to say nice things about activities & communities that you're a part of, or causal effects in the other direction (e.g., people who benefited the most from the Less Wrong website might be especially likely to attend a CFAR workshop or move into shared housing with other LWers).
It is worth noting that growth mindset and acquiring new techniques were both predictive of larger benefit from the rationality community even though neither variable is directly related to involvement in the community. That makes these effects less open to some of the alternative explanations which could account for the community involvement effects and provides some validation of the self-report measure of benefits, although other causal paths are still a possibility (e.g., people who have changed more since they started reading LW may have come to have more of a growth mindset and also report more benefits).
II. Acquiring New Techniques
"On average, about how often do you find another technique or approach that successfully helps you at being more rational / more productive / happier / having better social relationships / having more accurate beliefs / etc.?"
The average response was a 2.23 (SD = 1.31) on a 1 to 8 scale where 2 is “About once every six months” and 3 is “About once every 2 months.” This can be interpreted more intuitively as acquiring one new technique every 146 days (as a geometric mean).[3]
Which variables were associated with acquiring useful techniques more often?
Only some of them.
LW friendships and CFAR workshop attendance again had significant effects. The other two forms of in-person LW involvement, LW meetups and LW romantic partner, were also predictive of acquiring more techniques, but those effects did not remain significant in a regression controlling for the other variables. Time per day on Less Wrong had a weaker but reliable positive relationship with acquiring new techniques, while LW background had a significant relationship in the opposite direction: people with more LW background acquired fewer techniques. Younger age and growth mindset were again predictive of more benefit.
Based on the results of a regression, here is the number of days per new technique acquired (sorted by effect size, smaller numbers indicate faster technique acquisition).[4] In this list, both the number of days given and the order of the list reflect the results of the regression which controls statistically for the other predictor variables. (* = p < .05, ** = p < .01).
85 days: LW friendships *
87 days: Age (younger) **
95 days: Attended a full CFAR workshop **
114 days: LW romantic partner (p = .21)
118 days: Growth mindset **
174 days: LW background (negative effect) **
131 days: Time per day on Less Wrong **
151 days: Regularly attend meetups (p = .63)
146 days: all respondents
The pattern that was apparent on the self-report measure of benefit from the rationality community – that in-person interactions were more predictive of benefits than online participation – was even stronger on this measure. Attending a CFAR workshop and LW friendships had the largest effects, and these effects seem to be cumulative. People who both attended a full CFAR workshop and interacted with LW friends “all the time” (n = 39) acquired a new technique every 45 days on average, while people who had no in-person interaction with LWers by any of the 4 variables (n = 824) acquired a new technique every 165 days.
Some of the alternative explanations for the effects on self-reported benefit seem less plausible here. For example, it seems less likely that people who have LW friendships would say that they try and acquire more new techniques out of a general tendency to say nice things about communities that you're a part of. Alternative causal paths are still a clear possibility, though. People who tend to try more things may be more likely to go to LW meetups, sign up for CFAR workshops, or move to a city where they can hang out in person with people from their favorite website.
III. The Process of Trying & Acquiring New Techniques
“On average, about how often do you *read or hear about* another plausible-seeming technique or approach for being more rational / more productive / happier / having better social relationships / having more accurate beliefs / etc.?”
“On average, about how often do you *try out* another plausible-seeming technique or approach for being more rational / more productive / happier / having better social relationships / having more accurate beliefs / etc.?”
On average, people heard about a new technique every 12 days and tried a new technique about every 55 days. That means that (at least according to the streamlined model: hear → try → acquire) people tried about 22% of the techniques that they heard about, and added about 36% of the techniques that they tried to their repertoire.[5]
Breaking down acquiring techniques into its two components, techniques tried and hit rate (techniques acquired divided by techniques tried) all of the effects discussed above involving acquiring techniques appear to be due to trying techniques, and not to the hit rate. None of the variables discussed here were predictive of hit rate, and the variables that predicted acquiring techniques were similarly predictive of trying techniques (though in most cases the effect was slightly weaker). In particular, trying techniques predicted self-reported benefit from the rationality community, and people with more LW background tried fewer techniques. People who both attended a full CFAR workshop and interacted with LW friends “all the time” (n = 39) tried a new technique every 13 days, while people who reported no in-person interaction with LWers (n = 849) tried a new technique every 65 days.
These data provide some evidence that, if CFAR workshops, LW friendships, growth mindset, and time on Less Wrong cause people to acquire more techniques, a substantial portion of the effect comes from getting people to try more things (and not just getting them to be more effective at trying the things that they already have been trying).
However, these data do not clearly pin down is different about people's process of trying things. One might expect that hit rate reflects how good a person is at choosing what to try and actually trying it (in a way that makes useful techniques likely to stick), so the lack of effect on hit rate indicates that the difference is just in trying more things. But if someone improved at the process of trying things, becoming more efficient at getting useful-for-them techniques to stick and setting aside the not-useful-for-them techniques, then that might show up primarily as an increase in number of techniques tried (as they cycle through the try things process more rapidly & more frequently). Or, a person who lowers their threshold for what techniques to try might start trying five times as many things and finding twice as many that work for them, which would show up as a drop in their hit rate (they'd also be adding useful techniques to their repertoire twice as fast).[6]
IV. Growth Mindset
Sample item: “You can do things differently, but the important parts of who you are can't really be changed” (reverse-scored).
Growth mindset – seeing important parts of yourself as malleable, and focusing on what you can do to improve – seems like it could be related to the process of benefiting from the rationality community in multiple ways. Here are three:
Item 1 is what we've been looking at in the analysis of acquiring new techniques and self-reported benefit, with growth mindset as one of the predictor variables. The hypothesis is that people who score higher in growth mindset will report more benefit on those measures, and the data support that hypothesis (though these correlational results are also consistent with alternative causal hypotheses).
Item 2 identifies a hypothesis which treats growth mindset as an outcome variable instead of a predictor variable: do people who regularly attend LW meetups have more of a growth mindset? Or those who have more LW background, or who have attended a CFAR workshop, or who have LW friends, etc.? This hypothesis is relatively straightforward to examine with this data set, although the correlational design leaves it an open question whether involvement in the LW community led to a growth mindset or whether having a growth mindset led to people getting more involved in the LW community.
When looking at one variable at a time, each of the measures of in-person involvement in the LW community is significantly predictive of growth mindset. In order of effect size (given in Cohen's d, which counts standard deviations), growth mindset was predicted separately by LW romantic partner (d = 0.42), attending a CFAR workshop (d = 0.21), LW friendships (d = 0.20), and regularly attending meetups (d = 0.15). However, when controlling for the other predictor variables, only having a LW romantic partner remained statistically significant (d = 0.46, p = .03) and attending a CFAR workshop remained marginally significant (d = 0.18, p = .07); LW friendships and meetup attendance became nonsignificant (d < 0.10, p > 0.3).
LW background showed the opposite pattern: it was not related to growth mindset on its own (r = -0.04, p = .13), but it became a highly significant predictor of lower growth mindset when controlling for the other variables related to LW involvement (r = -0.11, p < .01). One plausible causal story that could explain this pattern of correlations is that people who are high in growth mindset who get involved in the website are more likely to also get involved in other in-person ways, while those lower in growth mindset are more likely to just stick with the website. This would lead to the negative relationship LW background and growth mindset when controlling for in-person LW involvement. According to this causal story, growth mindset is a cause of in-person LW involvement rather than a consequence.
Younger age was the strongest predictor of growth mindset, whether controlling for other variables (r = -0.15, p < .01) or not (r = -0.19, p < 0.01), and time per day on Less Wrong was not a significant predictor.
Item 3 from the list predicts an interaction effect between growth mindset and involvement in LW: the benefit of greater involvement in the LW community will be stronger among people high in growth mindset (or, equivalently, the benefit of growth mindset will be stronger among people who are more involved in the LW community). This hypothesis is particularly interesting because this interaction effect seems more plausible under the causal model where LW involvement and growth mindset both cause greater practical benefits than it does under the alternative causal theory that competence or a tendency to try things causes in-person LW involvement.
When predicting self-reported benefit from the rationality there was no sign of these interaction effects, whether looking at the predictor variables one at a time or including them all in a multiple regression. Growth mindset was an equally strong predictor of self-reported benefit for people who are closely involved in the LW community (by each of the various measures) and for people who are less closely involved in the LW community.
When predicting acquiring new techniques, these interaction effects were significant in several cases.[7] A growth mindset was associated more strongly with acquiring among techniques among people who regularly attend LW meetups (p = .003), people who are younger (p = .005), people who have attended a CFAR workshop (p = .04), and (with marginal statistical significance) among people with LW friendships (p = .06). In a multiple regression that included each of these variables, none of these interaction effects was individually statistically significant except the age x growth mindset interaction (presumably because of the various forms of LW involvement were all associated with each other, making it difficult to tease apart their effects).[8]
These results are consistent with the model that the various forms of in-person involvement with the rationality community are especially helpful at producing practical benefits for people who are high in growth mindset.
With this correlational research design there is a limit to how well we can distinguish the hypothesis that LW involvement leads to benefits from other causal stories, but each of the three main variables that we examined were related to in-person LW involvement in ways that were consistent with this hypothesis.
People who have been involved with the in-person LW/CFAR community were especially likely to indicate that their life is better due to the LW community. They tended to report that they tried out and acquired new useful techniques more frequently, especially if they were also high in growth mindset. If spending time with LWers or attending a CFAR workshop leads people to try more rationality-related techniques, find more things that work well for them, and reap the benefits, then these are the results that we would expect to see.
[1] The 4 mindset questions on the survey were taken from Dweck's book Mindset (p. 13). These questions and others like them have been used to measure mindset in many published studies. Many of the questions that have been used focus more narrowly on mindset about intellectual ability, while these four questions deal more broadly with personal qualities.
[2] Unless otherwise noted, all reported effects are significant both in tests with only the single predictor variable and also in tests which controlled for the other predictor variables. A regression was run predicting benefit based on the LW involvement variables and age (growth mindset and acquiring new techniques were not controlled for, since they could be consequences of LW involvement which mediate the benefit). Though all three levels of the categorical variables were included in the regression, the effect size used to order the variables in the list was calculated as the standardized difference in least square means between the highest level of the group (e.g., regularly attend meetups) and the lowest level (e.g., never attend meetups), leaving out intermediate levels (e.g., occasionally attend meetups). To estimate the effect size of continuous variables, the correlation coefficient was translated into an equivalent standardized mean difference by the formula d = 2r/sqrt(1-r2).
[3] The 8 response options were coded as a 1-8 scale, which was used for all analyses. Each scale point indicates a 3-4x multiplier in how often a person acquires new techniques. This 8-point scale can be interpreted as a log scale for the variable "days per technique acquired" (they are associated approximately by the equation 7*3^(5-x)) so a mean on this scale is equivalent to the geometric mean of the number of days. For example, a 3.5 on the 8-point scale translates into 36 days, which is the geometric mean of 21 days (a 4 on the scale) and 63 days (approximately a 3 on the scale).
[4] For categorical variables, the number of days is based on the least squares mean for the highest level of the group (e.g., regularly attend meetups). For continuous variables, it is based on the regression equation predicting the values one standard deviation above the mean of the predictor variable.
[5] On the 8 point scale, “heard about” has mean = 4.48 (SD = 1.62) and “tried” has mean = 3.12 (SD = 1.56). Rate of trying is simply “trying” minus “heard about,” mean = -1.37 (SD = 1.42), and hit rate had scale mean = -0.94 (SD = 0.84). These numbers can also be interpreted as being on a log base 3 scale, so -1 on the hit rate scale corresponds to an actual hit rate of 1/3 (1 technique acquired for every 3 techniques tried).
[6] Trying techniques can be further broken down into two components, hearing about techniques and percentage tried (techniques tried divided by techniques heard about). The data suggest that both are relevant, but they are harder to tease apart with the limited statistical power of this data set.
[7] When looking at a single categorical variable, I only looked at the highest level of the group and the lowest level, leaving out the intermediate level. For example, I tested whether growth mindset was more strongly related to acquiring techniques among people who regularly attend meetups than among people who never attend meetups (leaving out the group that occasionally attends meetups). In the regression including all predictor variables, I included the intermediate level groups (since otherwise it would have been necessary to exclude the data of anyone who was in an intermediate level group on any of the variables).
[8] When I combined the four variables related to in-person involvement into a single composite scale (scoring the highest level of involvement on each variable as a 2 and the lowest level as a 0), the interaction between growth mindset and this in-person involvement scale was statistically significant in a multiple regression predicting techniques acquired (p < .01).