Vaniver comments on 2012 Survey Results - Less Wrong

80 Post author: Yvain 07 December 2012 09:04PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Article Navigation

Comments (640)
Tags: survey meta community

Comments (640)

Sort By: Leading
Best Popular New Controversial Top Old Leading

You are viewing a single comment's thread. Show more comments above.

Comment author: Vaniver 02 December 2012 09:37:36PM *  7 points [-]

Related analysis on the public dataset:

1045 responders supplied a political orientation; they're 30% Libertarian, 3.1% Conservative, 37% Liberal, 29% Socialist, and 0.5% Communist.

226 responders supplied a political orientation and have been around since OB; they're 42% Libertarian, 3.5% Conservative, 31% Liberal, 23.5% Socialist, and 0% Communist.

242 responders supplied a political orientation and were referred from HPMoR; they're 30% Libertarian, 2.5% Conservative, 37% Liberal, 30% Socialist, and 0.4% Communist.

Note that analysis of current LW users who have been here since OB is not the same as OB users several years ago, but they are still significantly more libertarian than the current mix.

Comment author: Eugine_Nier 02 December 2012 10:51:52PM 4 points [-]

Also interesting that the HPMoR distribution almost exactly equals the current mix.

Comment author: gwern 03 December 2012 01:00:19AM *  7 points [-]

Oh yes, that reminds me - I've always wondered if MoR was a waste of time or not in terms of community-building. So let's divide the dataset into people who were referred to LW by MoR and people who weren't...

Summary: they are younger, lower karma, lower karma per month participating (karma log-transformed or not), more likely to be students; but they have the same IQ (self-report & test) as the rest.

So, Eliezer is successfully corrupting the youth, but it's not clear they are contributing very much yet.

R> lw <- read.csv("lw-survey/2012.csv")
R> hpmor <- lw[as.character(lw$Referrals) == "Referred by Harry Potter and the Methods of Rationality",]
R> hpmor <- lw[as.character(lw$Referrals) != "Referred by Harry Potter and the Methods of Rationality",]
R> t.test(hpmor$IQ, hpmor$IQ)
Welch Two Sample t-test
data: hpmor$IQ and hpmor$IQ
t = 0.5444, df = 99.28, p-value = 0.5874
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
-2.614 4.591
sample estimates:
mean of x mean of y
139.1 138.1
R> t.test(as.integer(as.character(hpmor$IQTest)), as.integer(as.character(hpmor$IQTest)))
Welch Two Sample t-test
data: as.integer(as.character(hpmor$IQTest)) and as.integer(as.character(hpmor$IQTest))
t = -0.0925, df = 264.8, p-value = 0.9264
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
-2.802 2.551
sample estimates:
mean of x mean of y
125.6 125.8
R> t.test(as.numeric(as.character(hpmor$Income)), as.numeric(as.character(hpmor$Income)))
Welch Two Sample t-test
data: as.numeric(as.character(hpmor$Income)) and as.numeric(as.character(hpmor$Income))
t = -4.341, df = 314.3, p-value = 1.917e-05
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
-29762 -11197
sample estimates:
mean of x mean of y
33948 54427
R> t.test(hpmor$Age, hpmor$Age)
Welch Two Sample t-test
data: hpmor$Age and hpmor$Age
t = -7.033, df = 484.4, p-value = 6.93e-12
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
-5.318 -2.995
sample estimates:
mean of x mean of y
24.51 28.67
R> t.test(as.character(hpmor$WorkStatus) == "Student", as.character(hpmor$WorkStatus) == "Student")
Welch Two Sample t-test
data: as.character(hpmor$WorkStatus) == "Student" and as.character(hpmor$WorkStatus) == "Student"
t = 4.154, df = 389.8, p-value = 4.018e-05
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
0.0791 0.2213
sample estimates:
mean of x mean of y
0.5224 0.3723
R> hpmortime <- hpmor$KarmaScore / as.numeric(as.character(hpmor$TimeinCommunity))
R> hpmortime <- hpmortime[!is.na(hpmortime) & !is.nan(hpmortime) & !is.infinite(hpmortime) ]
R> hpmortime <- hpmor$KarmaScore / as.numeric(as.character(hpmor$TimeinCommunity))
R> hpmortime <- hpmortime[!is.na(hpmortime) & !is.nan(hpmortime) & !is.infinite(hpmortime) ]
R> t.test(hpmortime, hpmortime)
Welch Two Sample t-test
data: hpmortime and hpmortime
t = 1.05, df = 642.7, p-value = 0.2942
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
-4.257 14.036
sample estimates:
mean of x mean of y
17.69 12.80
R> hpmortime <- log1p(hpmor$KarmaScore / as.numeric(as.character(hpmor$TimeinCommunity)))
R> hpmortime <- hpmortime[!is.na(hpmortime) & !is.nan(hpmortime) & !is.infinite(hpmortime) ]
R> hpmortime <- log1p(hpmor$KarmaScore / as.numeric(as.character(hpmor$TimeinCommunity)))
R> hpmortime <- hpmortime[!is.na(hpmortime) & !is.nan(hpmortime) & !is.infinite(hpmortime) ]
R> t.test(hpmortime, hpmortime)
Welch Two Sample t-test
data: hpmortime and hpmortime
t = 2.263, df = 396.9, p-value = 0.02416
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
0.03366 0.47878
sample estimates:
mean of x mean of y
1.1978 0.9415
Comment author: NancyLebovitz 03 December 2012 01:59:08AM 4 points [-]

The interesting question might be whether people whose primary interest is HPMOR are understanding and using ideas about rationality from it.

Comment author: gwern 03 December 2012 02:24:28AM 2 points [-]

Not sure how one would test that, aside from the CFAR questions which I don't know how to use.

Comment author: Unnamed 09 December 2012 11:49:56AM 3 points [-]

Looking at the four CFAR questions (described here), accuracy rates were:

74% OB folks ("Been here since it was started in the Overcoming Bias days", n=253)
64% MoR folks ("Referred by Harry Potter and the Methods of Rationality", n=253)
66% everyone else

So the original OB folks did better, but Methods influx is as good as the other sources of new readers. Breaking it down by question:

Question 1: disjunctive reasoning
OB: 52%
MoR: 42%
Other: 44%

Question 2: temporal discounting
OB: 94%
MoR: 89%
Other: 91%

Question 3: law of large numbers
OB: 92%
MoR: 85%
Other: 81%

Question 4: decoy effect
OB: 57%
MoR: 41%
Other: 49%

Comment author: NancyLebovitz 03 December 2012 03:26:35AM 2 points [-]

One possibility would be for Eliezer to ask people about it in his author's notes when he updates HPMOR.

On the second reading, I realize that I'm asking about HPMOR and spreading rationality rather than HPMOR and community building.

Comment author: Qiaochu_Yuan 03 December 2012 01:37:27AM 4 points [-]

Mean karma doesn't seem like the relevant metric; that reflects something like the contributions of the typical MoR user, which seems less important to me than the contributions of the top MoR users. The top users in a community generally contribute disproportionately, so a more relevant metric might be the proportion of top users who were referred here from MoR.

Comment author: gwern 03 December 2012 01:48:56AM 4 points [-]

The average user matters a lot, I think... But since you insist, here's the top 10% of each category:

R> sort(hpmor$KarmaScore, decreasing=T)[1:25]
[1] 9122 6815 4887 4500 2782 2600 2545 2117 2000 1800 1300 1017 1000 1000 858 771 694 575 560
[20] 443 425 422 350 285 274
R> sort(other$KarmaScore, decreasing=T)[1:83]
[1] 47384 32394 27418 15000 12200 11094 11000 10000 9000 8799 8000 8000 8000 6164 5000 5000
[17] 5000 5000 4658 4000 4000 4000 3960 3800 3693 3600 3500 3500 3500 3353 3300 3000
[33] 3000 3000 3000 3000 3000 3000 3000 2700 2500 2486 2400 2300 2204 2200 2100 2000
[49] 2000 2000 2000 2000 1977 1975 1900 1800 1800 1800 1750 1700 1653 1650 1648 1600
[65] 1590 1540 1520 1500 1500 1500 1500 1500 1500 1400 1253 1250 1200 1200 1115 1095
[81] 1044 1000 1000

The top MoR referral user is somewhere around 10th place in the other group (which is 3.3x larger).

Comment author: Vaniver 03 December 2012 03:49:33PM 2 points [-]

I imagine that when you divide karma by months in the community (while still restricting yourself to the top ten percent of absolute karma) the MoR contributors will look better. I'll do it tonight if you don't.

Comment author: gwern 03 December 2012 06:25:30PM *  2 points [-]

They do a bit better at the top; the sample size at "top 10%" is getting small enough that tests are losing power, though:

R> lw <- read.csv("lw-survey/2012.csv")
R>
R> hpmor <- lw[as.character(lw$Referrals) == "Referred by Harry Potter and the Methods of Rationality",]
R> other <- lw[as.character(lw$Referrals) != "Referred by Harry Potter and the Methods of Rationality",]
R>
R> hpmor <- hpmor[order(hpmor$KarmaScore, decreasing=TRUE),][1:25,]
R> other <- other[order(other$KarmaScore, decreasing=TRUE),][1:83,]
R>
R> hpmortime <- hpmor$KarmaScore / as.numeric(as.character(hpmor$TimeinCommunity))
R> hpmortime <- hpmortime[!is.na(hpmortime) & !is.nan(hpmortime) & !is.infinite(hpmortime) ]
R> othertime <- other$KarmaScore / as.numeric(as.character(other$TimeinCommunity))
R> othertime <- othertime[!is.na(othertime) & !is.nan(othertime) & !is.infinite(othertime) ]
R>
R> sort(hpmortime, decreasing=TRUE)
[1] 506.78 300.00 283.96 203.62 138.46 133.95 117.61 115.92 72.22 66.67 59.09 50.00 36.92
[14] 35.05 35.00 33.90 28.60 26.67 24.82 23.96 20.36 19.28 17.71 11.91
R> sort(othertime, decreasing=TRUE)
[1] 1895.36 647.88 456.97 338.89 263.16 250.00 250.00 235.71 184.90 183.33 173.91 166.67
[13] 165.00 146.93 146.65 145.83 142.86 133.33 133.33 133.33 125.00 125.00 125.00 116.45
[25] 102.73 100.00 97.22 84.38 83.33 83.33 83.33 83.33 75.00 75.00 74.51 72.00
[37] 69.60 68.88 66.67 66.67 63.33 61.11 60.71 60.34 58.33 57.14 55.95 53.43
[49] 52.17 50.00 50.00 50.00 50.00 48.48 46.46 44.12 43.75 41.67 41.43 40.00
[61] 39.66 36.36 35.91 33.33 33.33 31.67 31.32 30.00 30.00 30.00 27.50 27.47
[73] 26.95 25.33 25.00 25.00 24.06 23.33 22.73 18.25 16.67 16.67
R>
R> t.test(hpmortime,othertime)
Welch Two Sample t-test
data: hpmortime and othertime
t = -0.544, df = 72.4, p-value = 0.5881
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
-87.52 49.99
sample estimates:
mean of x mean of y
98.44 117.20
Comment author: Qiaochu_Yuan 03 December 2012 01:54:37AM *  2 points [-]

The average user that sticks around might matter a lot, but people with low karma are probably less likely to stick around so they'll have less of an impact (positive or negative) on the community. So maybe look at the distribution of karma, but among veteran users resp. veteran MoR users?

Comment author: gwern 03 December 2012 02:53:19AM 2 points [-]

What's 'veteran'? (And how many ways do you want to slice the data anyway...)

Comment author: dbaupp 04 December 2012 11:29:33AM 1 point [-] 
R> hpmor <- lw[as.character(lw$Referrals) == "Referred by Harry Potter and the Methods of Rationality",]
R> hpmor <- lw[as.character(lw$Referrals) != "Referred by Harry Potter and the Methods of Rationality",]

Is this a typo? Or some text that was lost in the copy-paste?

Comment author: gwern 04 December 2012 03:48:07PM 0 points [-]

Typo. I was operating on two variables, hpmor and others, but I guess a search-replace went awry...

Powered by Reddit Powered by Reddit