Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Comment author: Yvain 12 October 2014 03:34:04AM 4 points [-]

Any particular implementation details on OCEAN? Exact same as last time?

Comment author: VincentYu 13 October 2014 01:05:33PM *  6 points [-]

I suggest including the Big Five Inventory (BFI) in the survey itself. I've created an example of this on Google Forms. (I've reordered the inventory such that the first 11 items constitute the BFI-10, so that respondents can choose between the 44-item and 11-item versions).

The BFI is the inventory that was used in the online test to which the 2012 LW census linked. See also my comment about this in the 2012 LW census thread.

Comment author: Morendil 28 August 2014 11:02:03PM 0 points [-]

Haga, William J. "Perils of professionalism." Management Quarterly (1974): 3-10.

Comment author: VincentYu 19 September 2014 01:46:21PM 4 points [-]

Unfortunately, my university library reports that they have exhausted all possible sources and no library was able to supply this paper.

Comment author: VincentYu 18 September 2014 06:53:16AM *  7 points [-]

First, let me point out that the "behavioral changes" that the authors described were investigated over only three posts subsequent to each positive/negative evaluation, so it is unclear whether these effects remain over the long term.

Second, I find questionable the authors' conclusion that negative evaluations cause the subsequent decline in post quality and increase in post frequency, since they did not control the positive/negative evaluations. They model the positive/negative evaluations as random acts of chance (which is what we want for an RCT) and justify this by reporting that their bigram classifier assigns no difference in quality between the positively- and negatively-evaluated posts (across two posts by a pair of matched subjects). However, I find it likely that their classifier makes sufficiently many misclassifications to call into question their conclusion.

For instance, if bad posts have a tendency to occur in streaks of frequent posts (as is the case in flame wars), then we can explain their observations without assigning causal potency to negative evaluations: once in a while the classifier will erroneously assign a high quality to a bad post near the start of a flame war, but on average it will correctly assign low qualities to the subsequent three posts by the same poster in the flame war, and thus we see the effects that the authors described (without assigning any causal effect to the negative evaluation given by other users to the post near the start of the flame war). To test this explanation, the authors can ask the Crowdflower workers (p. 4) to label each b_0 (described on p. 5) to check whether their classifier is indeed misclassifying b_0 by assigning it too high a quality.

Since the authors did not conduct an RCT, we can come up with many alternative explanations, and I find them plausible. (Is it feasible to conduct an RCT on a site featuring upvotes and downvotes? Yes, it's been done before.)

Despite my criticisms, I think the paper is not bad. I just don't think the authors' methods provide sufficient evidence to warrant their seemingly strong confidence in their conclusions.

Comment author: gwern 12 September 2014 11:33:35PM *  2 points [-]

A scan/photograph/transcription of page 415 of Hays 1973, Statistics for the social sciences. (2nd ed.) New York: Holt, Rinehart & Winston; or heck the whole book if anyone can find it.

(Meehl in his 1990 "Why summaries of research on psychological theories are often uninterpretable" claims Hays agrees with him about the null hypothesis always being false, but I'm interested in exactly what Hays said and how he said it - albeit not enough to buy the book just to look at one page, and Google Books won't show me the relevant part regardless of how I try to chain my search queries.)

Comment author: VincentYu 16 September 2014 01:54:22AM 1 point [-]
Comment author: gwern 12 September 2014 11:33:35PM *  2 points [-]

A scan/photograph/transcription of page 415 of Hays 1973, Statistics for the social sciences. (2nd ed.) New York: Holt, Rinehart & Winston; or heck the whole book if anyone can find it.

(Meehl in his 1990 "Why summaries of research on psychological theories are often uninterpretable" claims Hays agrees with him about the null hypothesis always being false, but I'm interested in exactly what Hays said and how he said it - albeit not enough to buy the book just to look at one page, and Google Books won't show me the relevant part regardless of how I try to chain my search queries.)

Comment author: VincentYu 13 September 2014 01:36:34AM 3 points [-]

Requested.

Comment author: hg00 04 September 2014 09:28:08AM 0 points [-]
Comment author: VincentYu 13 September 2014 01:36:23AM 3 points [-]
Comment author: Morendil 28 August 2014 11:02:03PM 0 points [-]

Haga, William J. "Perils of professionalism." Management Quarterly (1974): 3-10.

Comment author: VincentYu 13 September 2014 01:35:50AM 3 points [-]

Requested.

Comment author: Morendil 17 August 2014 09:33:54AM 0 points [-]

(2011) Costa, AC. and Anderson, NR., Measuring trust in teams: development and validation of a multi-faceted measure of formative and reflexive indicators of team trust, European Journal of Work and Organizational Psychology 20 (1) : 119- 154

or in a pinch

(2003) Costa, AC., Work team trust and effectiveness, Personnel Review 32 (5) : 605- 622

Comment author: VincentYu 19 August 2014 11:01:58AM *  3 points [-]
Comment author: gwern 17 July 2014 09:16:18PM *  1 point [-]

Zuehlke, T. (2003). "Estimation of a Tobit model with unknown censoring threshold". Applied Economics 35,1163–9 (this is for a little analysis: https://plus.google.com/103530621949492999968/posts/TG98DXkHrrs )

Comment author: VincentYu 19 July 2014 04:28:30AM 1 point [-]
Comment author: gwern 05 July 2014 01:21:35AM *  3 points [-]

I would love a name for this too since the observation is important for why 'small' differences in means for normally distributed populations can have large consequences, and this occurs in many contexts (not just IQ or athletics).

Also good would be a quick name for log-normal distribution-like phenomenon.

The normal distribution can be seen as the sum of lots of independent random variables; so for example, IQ is normally distributed because the genetics is a lot of small additive variables. The log-normal is when it's the multiple of lots of independent variables; so any process where each step is necessary, as has been proposed for scientific productivity in having multiple steps like ideas->research->publication.

The normal distribution has the unintuitive behavior that small changes in the mean or variance have large consequences out on the thin tails. But the log-normal distribution has the unintuitive behavior that small improvements in each of the independent variables will yield large changes in their product, and that the extreme datapoints will be far beyond the median or average datapoints. ('Compound interest' comes close but doesn't seem to catch it because it refers to increase over time.)

Comment author: VincentYu 07 July 2014 01:47:51AM 2 points [-]

IQ is normally distributed because the genetics is a lot of small additive variables.

IQ is normally distributed because the distribution of raw test scores is standardized to a normal distribution.

View more: Next