Nice work! Seriously, I downloaded your survey analysis code and took a quick look--you deserve a huge thank you from all of us for the amount of effort you put in to this project. Please don't anyone forget to upvote this.
I started writing some commentary, but it got too long so I made a discussion post here.
I'm trying to reconcile the 353 people on LW with the 668 people who never left. What do you think is up there?
I think those people interpreted the question as "Never left the community" or some such, rather than the site itself.
Also if you add up the LW and LW Meetup numbers they're almost that, I'd have to play with the data real quick to see how much overlap there is. Eh it's just a SQL query I'll do it right now:
sqlite> select count(*) from data where ActiveMemberships_1 = "Yes";
354
sqlite> select count(*) from data where ActiveMemberships_2 = "Yes";
216
sqlite> select count(*) from data where ActiveMemberships_1 = "Yes" AND ActiveMemberships_2 = "Yes";
77
77 / 354 = 0.2175141242937853
77 / 216 = 0.35648148148148145
P(Meetups | LW Site) = 21.75%
P(LW Site | Meetups) = 35.63%
I think that goes a long way towards explaining that particular inconsistency.
However it still leaves a quarter unaccounted for:
(668 - (354 + (216 - (216 * 0.3563)))) / 668 = 0.2619173652694611
I wonder if the "average complainant" exists here - or more generally, what complaints correlate positively? Will look at the data tomorrow (precommitment!). I feel ill-disposed towards listening to people who fit some stereotype of "people who never really got it but want to slag it," but maybe that group of people is negligible and there's just a bunch of people with different feelings and experiences.
Impressions: I didn't do any actual statistical tests, just massaged the data a lot.
For almost every subject it was possible to complain about, the people who complained about it were a diverse group. Particularly, I noticed that they were diverse in terms of how much of LW they'd read. On the other hand, people who marked an unusual number of complaints were more likely than usual to never have commented.
Comparing people who have both read and commended on LW to everyone else, the full users had more complaints on average (meaning that the prolific complainers were small minority among non-commenters). Full users were less likely to complain (proportional to amount of complaining) about focus on AI or criticism of science, and more likely to complain about jargon. On the community side, they were more likely to care about overly high standards and website design, and also somewhat more likely to care about in-person interaction.
Complaints are not correlated very well. For example, the correlation between full users who complained about software and full users who complained about high standards was very low (though I think significantly positive). There didn't seem to be any large population of average complainers, just a bunch of people with different opinions.
I'm at the moment trying to look at the data and find it hard given that I don't have a good list of the questions.
I think it would be great if we have an additional spreadsheet that contains one column with the headers of the questions and one column with the content of the question.
The information about likert scale conversion could be added in a third column.
I don't know if you already have this and you're asking for a better list, but the printable version of the survey is here:
http://www.jdpressman.com/public/lwsurvey2016/lw_2016_survey_(2016_05_01).pdf
In that list I see "31 []With which of these meta-ethical views do you MOST identify? (Bonus Question)". The list doesn't contain information about how that row happens to be named in the data file.
It's also not possible to easily see what question 'EndOfWorkConcerns[3]' happens to be. If there would be an .csv file I could very easily do:
text_end_of_work3 = df_questions.query("name == '%s'" %'EndOfWorkConcerns[3]')
Small note, "jargon" and "none of the above" seem to be missing codes/numbers in the philosophy table.
Edit: Also, the the tagged community write-ins is a 404 (there's a missing slash in the URL).
Further edit: I suspect you've misunderstood the one "NERB"; I think it's complaining about the discussion of Roko's Basilisk, not about there not being enough of it. The question asked about problems, right? (I wouldn't bother pointing this out, as as you've said this is necessarily messy and subjectve, but this one particular one seemed directly backwards.)
Small note, "jargon" and "none of the above" seem to be missing codes/numbers in the philosophy table.
That's because they didn't get used. I should probably just remove them.
Edit: Also, the the tagged community write-ins is a 404 (there's a missing slash in the URL).
Fixed.
(I wouldn't bother pointing this out, as as you've said this is necessarily messy and subjectve, but this one particular one seemed directly backwards.)
You seem to be correct, I was trying really hard to power through these and I think I mixed these two up:
14:41 < namespace> "To be fair I wasn't around for the peak, however, I'd cite the aforementioned Basilisk. It suddenly presents a sort of 'Rational Devil' to a super intelligent Al's 'Rational Messiah'. It's silly and mildly off-putting. Thankfully the information on the site, especially the Sequences, is far too useful to just toss aside. | NERB"
14:42 < namespace> "A tendency to hyperfocus on the ridiculousness of Roko's Basilisk stopped most people (inside and outside of LessWrong) from thinking about more-plausible acausal-trade-based ideas. | TMRB"
Won't fix though because the numbers come out the same anyway.
2016 LessWrong Diaspora Survey Analysis
Overview
Introduction
Before it was the LessWrong survey, the 2016 survey was a small project I was working on as market research for a website I'm creating called FortForecast. As I was discussing the idea with others, particularly Eliot he made the suggestion that since he's doing LW 2.0 and I'm doing a site that targets the LessWrong demographic, why don't I go ahead and do the LessWrong Survey? Because of that, this years survey had a lot of questions oriented around what you would want to see in a successor to LessWrong and what you think is wrong with the site.
LessWrong Usage and Experience
How Did You Find LessWrong?
Been here since it was started in the Overcoming Bias days: 171 8.3%
Referred by a link: 275 13.4%
HPMOR: 542 26.4%
Overcoming Bias: 80 3.9%
Referred by a friend: 265 12.9%
Referred by a search engine: 131 6.4%
Referred by other fiction: 14 0.7%
Slate Star Codex: 241 11.7%
Reddit: 55 2.7%
Common Sense Atheism: 19 0.9%
Hacker News: 47 2.3%
Gwern: 22 1.1%
Other: 191 9.308%
How do you use Less Wrong?
I lurk, but never registered an account: 1120 54.4%
I've registered an account, but never posted: 270 13.1%
I've posted a comment, but never a top-level post: 417 20.3%
I've posted in Discussion, but not Main: 179 8.7%
I've posted in Main: 72 3.5%
[54.4% lurkers.]
How often do you comment on LessWrong?
I have commented more than once a week for the past year.: 24 1.2%
I have commented more than once a month for the past year but less than once a week.: 63 3.1%
I have commented but less than once a month for the past year.: 225 11.1%
I have not commented this year.: 1718 84.6%
[You could probably snarkily title this one "LW usage in one statistic". It's a pretty damning portrait of the sites vitality. A whopping 84.6% of people have not commented this year a single time.]
How Long Since You Last Posted On LessWrong?
I wrote one today.: 12 0.637%
Within the last three days.: 13 0.69%
Within the last week.: 22 1.168%
Within the last month.: 58 3.079%
Within the last three months.: 75 3.981%
Within the last six months.: 68 3.609%
Within the last year.: 84 4.459%
Within the last five years.: 295 15.658%
Longer than five years.: 15 0.796%
I've never posted on LW.: 1242 65.924%
[Supermajority of people have never commented on LW, 5.574% have within the last month.]
About how much of the Sequences have you read?
Never knew they existed until this moment: 215 10.3%
Knew they existed, but never looked at them: 101 4.8%
Some, but less than 25% : 442 21.2%
About 25%: 260 12.5%
About 50%: 283 13.6%
About 75%: 298 14.3%
All or almost all: 487 23.3%
[10.3% of people taking the survey have never heard of the sequences. 36.3% have not read a quarter of them.]
Do you attend Less Wrong meetups?
Yes, regularly: 157 7.5%
Yes, once or a few times: 406 19.5%
No: 1518 72.9%
[However the in-person community seems to be non-dead.]
Is physical interaction with the Less Wrong community otherwise a part of your everyday life, for example do you live with other Less Wrongers, or you are close friends and frequently go out with them?
Yes, all the time: 158 7.6%
Yes, sometimes: 258 12.5%
No: 1652 79.9%
About the same number say they hang out with LWers 'all the time' as say they go to meetups. I wonder if people just double counted themselves here. Or they may go to meetups and have other interactions with LWers outside of that. Or it could be a coincidence and these are different demographics. Let's find out.
P(Community part of daily life | Meetups) = 40%
Significant overlap, but definitely not exclusive overlap. I'll go ahead and chalk this one up up to coincidence.
Have you ever been in a romantic relationship with someone you met through the Less Wrong community?
Yes: 129 6.2%
I didn't meet them through the community but they're part of the community now: 102 4.9%
No: 1851 88.9%
LessWrong Usage Differences Between 2016 and 2014 Surveys
How do you use Less Wrong?
I lurk, but never registered an account: +19.300% 1125 54.400%
I've registered an account, but never posted: -1.600% 271 13.100%
I've posted a comment, but never a top-level post: -7.600% 419 20.300%
I've posted in Discussion, but not Main: -5.100% 179 8.700%
I've posted in Main: -3.300% 73 3.500%
About how much of the sequences have you read?
Never knew they existed until this moment: +3.300% 217 10.400%
Knew they existed, but never looked at them: +2.100% 103 4.900%
Some, but less than 25%: +3.100% 442 21.100%
About 25%: +0.400% 260 12.400%
About 50%: -0.400% 284 13.500%
About 75%: -1.800% 299 14.300%
All or almost all: -5.000% 491 23.400%
Do you attend Less Wrong meetups?
Yes, regularly: -2.500% 160 7.700%
Yes, once or a few times: -2.100% 407 19.500%
No: +7.100% 1524 72.900%
Is physical interaction with the Less Wrong community otherwise a part of your everyday life, for example do you live with other Less Wrongers, or you are close friends and frequently go out with them?
Yes, all the time: +0.200% 161 7.700%
Yes, sometimes: -0.300% 258 12.400%
No: +2.400% 1659 79.800%
Have you ever been in a romantic relationship with someone you met through the Less Wrong community?
Yes: +0.800% 132 6.300%
I didn't meet them through the community but they're part of the community now: -0.400% 102 4.900%
No: +1.600% 1858 88.800%
Write Ins
In a bit of a silly oversight I forgot to ask survey participants what was good about the community, so the following is going to be a pretty one sided picture. Below are the complete write ins respondents submitted
Issues With LessWrong At It's Peak
Philosophical Issues With LessWrong At It's Peak[Part One]
Philosophical Issues With LessWrong At It's Peak[Part Two]
Community Issues With LessWrong At It's Peak[Part One]
Community Issues With LessWrong At It's Peak[Part Two]
Issues With LessWrong Now
Philosophical Issues With LessWrong Now[Part One]
Philosophical Issues With LessWrong Now[Part Two]
Community Issues With LessWrong Now[Part One]
Community Issues With LessWrong Now[Part Two]
Peak Philosophy Issue Tallies
Well, those are certainly some results. Top answers are:
Narrow Scholarship: 20
Arrogance: 16
Too Much Theory: 14
Lack of Rigor: 14
Misfocused: 13
Nothing: 13
Reinvention (reinvents the wheel too much): 10
Personality Cult: 10
So condensing a bit: Pay more attention to mainstream scholarship and ideas, try to do better about intellectual rigor, be more practical and focus on results, be more humble. (Labeled Dataset)
Peak Community Issue Tallies
Top Answers:
Unwelcoming: 12
Too Much Theory: 12
Groupthink: 11
Not Enough Community Infrastructure: 10
Too Contrarian: 10
Insufficiently Indexed: 9
Cultish: 9
Again condensing a bit: Work on being less intimidating/aggressive/etc to newcomers, spend less time on navel gazing and more time on actually doing things and collecting data, work on getting the structures in place that will onboard people into the community, stop being so nitpicky and argumentative, spend more time on getting content indexed in a form where people can actually find it, be more accepting of outside viewpoints and remember that you're probably more likely to be wrong than you think. (Labeled Dataset)
One last note before we finish up, these tallies are a very rough executive summary. The tagging process basically involves trying to fit points into clusters and is prone to inaccuracy through laziness, adding another category being undesirable, square-peg into round-hole fitting, and my personal political biases. So take these with a grain of salt, if you really want to know what people wrote in my advice would be to read through the write in sets I have above in HTML format. If you want to evaluate for yourself how well I tagged things you can see the labeled datasets above.
I won't bother tallying the "issues now" sections, all you really need to know is that it's basically the same as the first sections except with lots more "It's dead." comments and from eyeballing it a higher proportion of people arguing that LessWrong has been taken over by the left/social justice and complaints about effective altruism. (I infer that the complaints about being taken over by the left are mostly referring to effective altruism.)
Traits Respondents Would Like To See In A Successor Community
Philosophically
Attention Paid To Outside Sources
More: 1042 70.933%
Same: 414 28.182%
Less: 13 0.885%
Self Improvement Focus
More: 754 50.706%
Same: 598 40.215%
Less: 135 9.079%
AI Focus
More: 184 12.611%
Same: 821 56.271%
Less: 454 31.117%
Political
More: 330 22.837%
Same: 770 53.287%
Less: 345 23.875%
Academic/Formal
More: 455 31.885%
Same: 803 56.272%
Less: 169 11.843%
In summary, people want a site that will engage with outside ideas, acknowledge where it borrows from, focus on practical self improvement, less on AI and AI risk, and tighten its academic rigor. They could go either way on politics but the epistemic direction is clear.
Community
Intense Environment
More: 254 19.644%
Same: 830 64.192%
Less: 209 16.164%
Focused On 'Real World' Action
More: 739 53.824%
Same: 563 41.005%
Less: 71 5.171%
Experts
More: 749 55.605%
Same: 575 42.687%
Less: 23 1.707%
Data Driven/Testing Of Ideas
More: 1107 78.344%
Same: 291 20.594%
Less: 15 1.062%
Social
More: 583 43.507%
Same: 682 50.896%
Less: 75 5.597%
This largely backs up what I said about the previous results. People want a more practical, more active, more social and more empirical LessWrong with outside expertise and ideas brought into the fold. They could go either way on it being more intense but the epistemic trend is still clear.
Write Ins
Diaspora Communities
So where did the party go? We got twice as many respondents this year as last when we opened up the survey to the diaspora, which means that the LW community is alive and kicking it's just not on LessWrong.
LessWrong
Yes: 353 11.498%
No: 1597 52.02%
LessWrong Meetups
Yes: 215 7.003%
No: 1735 56.515%
LessWrong Facebook Group
Yes: 171 5.57%
No: 1779 57.948%
LessWrong Slack
Yes: 55 1.792%
No: 1895 61.726%
SlateStarCodex
Yes: 832 27.101%
No: 1118 36.417%
[SlateStarCodex by far has the highest proportion of active LessWrong users, over twice that of LessWrong itself, and more than LessWrong and Tumblr combined.]
Rationalist Tumblr
Yes: 350 11.401%
No: 1600 52.117%
[I'm actually surprised that Tumblr doesn't just beat LessWrong itself outright, They're only a tenth of a percentage point behind though, and if current trends continue I suspect that by 2017 Tumblr will have a large lead over the main LW site.]
Rationalist Facebook
Yes: 150 4.886%
No: 1800 58.632%
[Eliezer Yudkowsky currently resides here.]
Rationalist Twitter
Yes: 59 1.922%
No: 1891 61.596%
Effective Altruism Hub
Yes: 98 3.192%
No: 1852 60.326%
FortForecast
Yes: 4 0.13%
No: 1946 63.388%
[I included this as a 'troll' option to catch people who just check every box. Relatively few people seem to have done that, but having the option here lets me know one way or the other.]
Good Judgement(TM) Open
Yes: 29 0.945%
No: 1921 62.573%
PredictionBook
Yes: 59 1.922%
No: 1891 61.596%
Omnilibrium
Yes: 8 0.261%
No: 1942 63.257%
Hacker News
Yes: 252 8.208%
No: 1698 55.309%
#lesswrong on freenode
Yes: 76 2.476%
No: 1874 61.042%
#slatestarcodex on freenode
Yes: 36 1.173%
No: 1914 62.345%
#hplusroadmap on freenode
Yes: 4 0.13%
No: 1946 63.388%
#chapelperilous on freenode
Yes: 10 0.326%
No: 1940 63.192%
[Since people keep asking me, this is a postrational channel.]
/r/rational
Yes: 274 8.925%
No: 1676 54.593%
/r/HPMOR
Yes: 230 7.492%
No: 1720 56.026%
[Given that the story is long over, this is pretty impressive. I'd have expected it to be dead by now.]
/r/SlateStarCodex
Yes: 244 7.948%
No: 1706 55.57%
One or more private 'rationalist' groups
Yes: 192 6.254%
No: 1758 57.264%
[I almost wish I hadn't included this option, it'd have been fascinating to learn more about these through write ins.]
Of all the parties who seem like plausible candidates at the moment, Scott Alexander seems most capable to undiaspora the community. In practice he's very busy, so he would need a dedicated team of relatively autonomous people to help him. Scott could court guest posts and start to scale up under the SSC brand, and I think he would fairly easily end up with the lions share of the free floating LWers that way.
Before I call a hearse for LessWrong, there is a glimmer of hope left:
Would you consider rejoining LessWrong?
I never left: 668 40.6%
Yes: 557 33.8%
Yes, but only under certain conditions: 205 12.5%
No: 216 13.1%
A significant fraction of people say they'd be interested in an improved version of the site. And of course there were write ins for conditions to rejoin, what did people say they'd need to rejoin the site?
Rejoin Condition Write Ins [Part One]
Rejoin Condition Write Ins [Part Two]
Rejoin Condition Write Ins [Part Three]
Rejoin Condition Write Ins [Part Four]
Rejoin Condition Write Ins [Part Five]
Feel free to read these yourselves (they're not long), but I'll go ahead and summarize: It's all about the content. Content, content, content. No amount of usability improvements, A/B testing or clever trickery will let you get around content. People are overwhelmingly clear about this; they need a reason to come to the site and right now they don't feel like they have one. That means priority number one for somebody trying to revitalize LessWrong is how you deal with this.
Let's recap.
Future Improvement Wishlist Based On Survey Results
Philosophical
Community
While that list seems reasonable, it's quite hard to put into practice. Rigor, as the name implies requires high-effort from participants. Frankly, it's not fun. And getting people to do un-fun things without paying them is difficult. If LessWrong is serious about it's goal of 'advancing the art of human rationality' then it needs to figure out a way to do real investigation into the subject. Not just have people 'discuss', as though the potential for Rationality is within all of us just waiting to be brought out by the right conversation.
I personally haven't been a LW regular in a long time. Assuming the points about pedanticism, snipping, "well actually"-ism and the like are true then they need to stop for the site to move forward. Personally, I'm a huge fan of Scott Alexander's comment policy: All comments must be at least two of true, kind, or necessary.
True and kind - Probably won't drown out the discussion signal, will help significantly decrease the hostility of the atmosphere.
True and necessary - Sometimes what you have to say isn't nice, but it needs to be said. This is the common core of free speech arguments for saying mean things and they're not wrong. However, something being true isn't necessarily enough to make it something you should say. In fact, in some situations saying mean things to people entirely unrelated to their arguments is known as the ad hominem fallacy.
Kind and necessary - The infamous 'hugbox' is essentially a place where people go to hear things which are kind but not necessarily true. I don't think anybody wants a hugbox, but occasionally it can be important to say things that might not be true but are needed for the sake of tact, reconciliation, or to prevent greater harm.
If people took that seriously and really gave it some thought before they used their keyboard, I think the on-site LessWrong community would be a significant part of the way to not driving people off as soon as they arrive.
More importantly, in places like the LessWrong Slack I see this sort of happy go lucky attitude about site improvement. "Oh that sounds nice, we should do that." without the accompanying mountain of work to actually make 'that' happen. I'm not sure people really understand the dynamics of what it means to 'revive' a website in severe decay. When you decide to 'revive' a dying site, what you're really doing once you're past a certain point is refounding the site. So the question you should be asking yourself isn't "Can I fix the site up a bit so it isn't quite so stale?". It's "Could I have founded this site?" and if the answer is no you should seriously question whether to make the time investment.
Whether or not LessWrong lives to see another day basically depends on the level of ground game its last users and administrators can muster up. And if it's not enough, it won't.
Virtus junxit mors non separabit!