Qiaochu_Yuan comments on 2012 Survey Results - Less Wrong

80 Post author: Yvain 07 December 2012 09:04PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Article Navigation

Comments (640)
Tags: survey meta community

Comments (640)

You are viewing a single comment's thread. Show more comments above.

Comment author: gwern 03 December 2012 01:00:19AM *  7 points [-]

Oh yes, that reminds me - I've always wondered if MoR was a waste of time or not in terms of community-building. So let's divide the dataset into people who were referred to LW by MoR and people who weren't...

Summary: they are younger, lower karma, lower karma per month participating (karma log-transformed or not), more likely to be students; but they have the same IQ (self-report & test) as the rest.

So, Eliezer is successfully corrupting the youth, but it's not clear they are contributing very much yet.

R> lw <- read.csv("lw-survey/2012.csv")
R> hpmor <- lw[as.character(lw$Referrals) == "Referred by Harry Potter and the Methods of Rationality",]
R> hpmor <- lw[as.character(lw$Referrals) != "Referred by Harry Potter and the Methods of Rationality",]
R> t.test(hpmor$IQ, hpmor$IQ)
Welch Two Sample t-test
data: hpmor$IQ and hpmor$IQ
t = 0.5444, df = 99.28, p-value = 0.5874
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
-2.614 4.591
sample estimates:
mean of x mean of y
139.1 138.1
R> t.test(as.integer(as.character(hpmor$IQTest)), as.integer(as.character(hpmor$IQTest)))
Welch Two Sample t-test
data: as.integer(as.character(hpmor$IQTest)) and as.integer(as.character(hpmor$IQTest))
t = -0.0925, df = 264.8, p-value = 0.9264
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
-2.802 2.551
sample estimates:
mean of x mean of y
125.6 125.8
R> t.test(as.numeric(as.character(hpmor$Income)), as.numeric(as.character(hpmor$Income)))
Welch Two Sample t-test
data: as.numeric(as.character(hpmor$Income)) and as.numeric(as.character(hpmor$Income))
t = -4.341, df = 314.3, p-value = 1.917e-05
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
-29762 -11197
sample estimates:
mean of x mean of y
33948 54427
R> t.test(hpmor$Age, hpmor$Age)
Welch Two Sample t-test
data: hpmor$Age and hpmor$Age
t = -7.033, df = 484.4, p-value = 6.93e-12
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
-5.318 -2.995
sample estimates:
mean of x mean of y
24.51 28.67
R> t.test(as.character(hpmor$WorkStatus) == "Student", as.character(hpmor$WorkStatus) == "Student")
Welch Two Sample t-test
data: as.character(hpmor$WorkStatus) == "Student" and as.character(hpmor$WorkStatus) == "Student"
t = 4.154, df = 389.8, p-value = 4.018e-05
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
0.0791 0.2213
sample estimates:
mean of x mean of y
0.5224 0.3723
R> hpmortime <- hpmor$KarmaScore / as.numeric(as.character(hpmor$TimeinCommunity))
R> hpmortime <- hpmortime[!is.na(hpmortime) & !is.nan(hpmortime) & !is.infinite(hpmortime) ]
R> hpmortime <- hpmor$KarmaScore / as.numeric(as.character(hpmor$TimeinCommunity))
R> hpmortime <- hpmortime[!is.na(hpmortime) & !is.nan(hpmortime) & !is.infinite(hpmortime) ]
R> t.test(hpmortime, hpmortime)
Welch Two Sample t-test
data: hpmortime and hpmortime
t = 1.05, df = 642.7, p-value = 0.2942
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
-4.257 14.036
sample estimates:
mean of x mean of y
17.69 12.80
R> hpmortime <- log1p(hpmor$KarmaScore / as.numeric(as.character(hpmor$TimeinCommunity)))
R> hpmortime <- hpmortime[!is.na(hpmortime) & !is.nan(hpmortime) & !is.infinite(hpmortime) ]
R> hpmortime <- log1p(hpmor$KarmaScore / as.numeric(as.character(hpmor$TimeinCommunity)))
R> hpmortime <- hpmortime[!is.na(hpmortime) & !is.nan(hpmortime) & !is.infinite(hpmortime) ]
R> t.test(hpmortime, hpmortime)
Welch Two Sample t-test
data: hpmortime and hpmortime
t = 2.263, df = 396.9, p-value = 0.02416
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
0.03366 0.47878
sample estimates:
mean of x mean of y
1.1978 0.9415
Comment author: Qiaochu_Yuan 03 December 2012 01:37:27AM 4 points [-]

Mean karma doesn't seem like the relevant metric; that reflects something like the contributions of the typical MoR user, which seems less important to me than the contributions of the top MoR users. The top users in a community generally contribute disproportionately, so a more relevant metric might be the proportion of top users who were referred here from MoR.

Comment author: gwern 03 December 2012 01:48:56AM 4 points [-]

The average user matters a lot, I think... But since you insist, here's the top 10% of each category:

R> sort(hpmor$KarmaScore, decreasing=T)[1:25]
[1] 9122 6815 4887 4500 2782 2600 2545 2117 2000 1800 1300 1017 1000 1000 858 771 694 575 560
[20] 443 425 422 350 285 274
R> sort(other$KarmaScore, decreasing=T)[1:83]
[1] 47384 32394 27418 15000 12200 11094 11000 10000 9000 8799 8000 8000 8000 6164 5000 5000
[17] 5000 5000 4658 4000 4000 4000 3960 3800 3693 3600 3500 3500 3500 3353 3300 3000
[33] 3000 3000 3000 3000 3000 3000 3000 2700 2500 2486 2400 2300 2204 2200 2100 2000
[49] 2000 2000 2000 2000 1977 1975 1900 1800 1800 1800 1750 1700 1653 1650 1648 1600
[65] 1590 1540 1520 1500 1500 1500 1500 1500 1500 1400 1253 1250 1200 1200 1115 1095
[81] 1044 1000 1000

The top MoR referral user is somewhere around 10th place in the other group (which is 3.3x larger).

Comment author: Qiaochu_Yuan 03 December 2012 01:54:37AM *  2 points [-]

The average user that sticks around might matter a lot, but people with low karma are probably less likely to stick around so they'll have less of an impact (positive or negative) on the community. So maybe look at the distribution of karma, but among veteran users resp. veteran MoR users?

Comment author: gwern 03 December 2012 02:53:19AM 2 points [-]

What's 'veteran'? (And how many ways do you want to slice the data anyway...)

Comment author: Vaniver 03 December 2012 03:49:33PM 2 points [-]

I imagine that when you divide karma by months in the community (while still restricting yourself to the top ten percent of absolute karma) the MoR contributors will look better. I'll do it tonight if you don't.

Comment author: gwern 03 December 2012 06:25:30PM *  2 points [-]

They do a bit better at the top; the sample size at "top 10%" is getting small enough that tests are losing power, though:

R> lw <- read.csv("lw-survey/2012.csv")
R>
R> hpmor <- lw[as.character(lw$Referrals) == "Referred by Harry Potter and the Methods of Rationality",]
R> other <- lw[as.character(lw$Referrals) != "Referred by Harry Potter and the Methods of Rationality",]
R>
R> hpmor <- hpmor[order(hpmor$KarmaScore, decreasing=TRUE),][1:25,]
R> other <- other[order(other$KarmaScore, decreasing=TRUE),][1:83,]
R>
R> hpmortime <- hpmor$KarmaScore / as.numeric(as.character(hpmor$TimeinCommunity))
R> hpmortime <- hpmortime[!is.na(hpmortime) & !is.nan(hpmortime) & !is.infinite(hpmortime) ]
R> othertime <- other$KarmaScore / as.numeric(as.character(other$TimeinCommunity))
R> othertime <- othertime[!is.na(othertime) & !is.nan(othertime) & !is.infinite(othertime) ]
R>
R> sort(hpmortime, decreasing=TRUE)
[1] 506.78 300.00 283.96 203.62 138.46 133.95 117.61 115.92 72.22 66.67 59.09 50.00 36.92
[14] 35.05 35.00 33.90 28.60 26.67 24.82 23.96 20.36 19.28 17.71 11.91
R> sort(othertime, decreasing=TRUE)
[1] 1895.36 647.88 456.97 338.89 263.16 250.00 250.00 235.71 184.90 183.33 173.91 166.67
[13] 165.00 146.93 146.65 145.83 142.86 133.33 133.33 133.33 125.00 125.00 125.00 116.45
[25] 102.73 100.00 97.22 84.38 83.33 83.33 83.33 83.33 75.00 75.00 74.51 72.00
[37] 69.60 68.88 66.67 66.67 63.33 61.11 60.71 60.34 58.33 57.14 55.95 53.43
[49] 52.17 50.00 50.00 50.00 50.00 48.48 46.46 44.12 43.75 41.67 41.43 40.00
[61] 39.66 36.36 35.91 33.33 33.33 31.67 31.32 30.00 30.00 30.00 27.50 27.47
[73] 26.95 25.33 25.00 25.00 24.06 23.33 22.73 18.25 16.67 16.67
R>
R> t.test(hpmortime,othertime)
Welch Two Sample t-test
data: hpmortime and othertime
t = -0.544, df = 72.4, p-value = 0.5881
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
-87.52 49.99
sample estimates:
mean of x mean of y
98.44 117.20
Powered by Reddit Powered by Reddit