Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

DanielVarga comments on Best of Rationality Quotes 2009/2010 - Less Wrong

24 Post author: DanielVarga 18 December 2010 09:36PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (48)

You are viewing a single comment's thread. Show more comments above.

Comment author: DanielVarga 22 December 2010 05:47:09AM *  2 points [-]

I think a good metric is this: Assuming we independently draw from the observed distribution of achieved karma scores, what is the probability that someone gets at least as much karma as Yvain when she posts as many quotes as Yvain? You can calculate this by iterated convolution. The assumption of total independence heavily favors Yvain, but I am fine with that.

I loaded the actual observed distribution, and calculated this score:

  • 0.00008 (12.48 in 54): Rain
  • 0.00066 (15.53 in 17): Yvain
  • 0.00128 (13.15 in 27): MichaelGR
  • 0.00174 (54.00 in 1): michaelkeenan
  • 0.00312 (13.29 in 21): RobinZ
  • 0.00766 (22.67 in 3): Tesseract
  • 0.00836 (18.80 in 5): Unnamed
  • 0.01499 (18.25 in 4): sketerpot
  • 0.02368 (10.15 in 47): Eliezer_Yudkowsky
  • 0.02473 (18.33 in 3): Kyre
  • 0.03460 (19.50 in 2): knb
  • 0.03831 (15.50 in 4): Lightwave
  • 0.04265 (23.00 in 1): Vlad
  • 0.04817 (16.00 in 3): Hariant
  • 0.05266 (12.86 in 7): Kutta
  • 0.05396 (22.00 in 1): DaveInNYC
  • 0.06051 (12.57 in 7): wuwei
  • 0.06789 (20.00 in 1): CSmith
  • 0.07663 (13.50 in 4): Apprentice
  • 0.07663 (13.50 in 4): komponisto
  • 0.08094 (19.00 in 1): Marcello
  • 0.08622 (14.00 in 3): jaimeastorga2000
  • 0.08622 (14.00 in 3): MichaelHoward
  • 0.09554 (11.38 in 8): billswift
  • 0.10009 (18.00 in 1): cata
  • 0.11401 (17.00 in 1): MarcTheEngineer
  • 0.12449 (8.77 in 81): RichardKennaway
  • 0.12763 (12.00 in 4): SilasBarta
  • 0.13055 (16.00 in 1): CaptainOblivious2
  • 0.13055 (16.00 in 1): Tyrrell_McAllister
  • 0.13092 (13.50 in 2): JamesAndrix
  • 0.13828 (12.33 in 3): Randaly
  • 0.14534 (15.00 in 1): Automaton
  • 0.14534 (15.00 in 1): loqi
  • 0.14534 (15.00 in 1): Nisan
  • 0.14534 (15.00 in 1): Patrick
  • 0.14534 (15.00 in 1): teageegeepea
  • 0.14695 (13.00 in 2): BenAlbahari
  • 0.15183 (10.83 in 6): DSimon
Comment author: RobinZ 22 December 2010 02:22:24PM 0 points [-]

I don't quite understand the methodology - how do you determine the karma distribution for each poster? And how is the list sorted?

Comment author: DanielVarga 22 December 2010 09:00:20PM 1 point [-]

I am afraid I don't understand either of your questions. I work with the karma distribution only in the quotes domain. It doesn't have to be determined, I collected all the data myself. The list is sorted by p-value.

We have the total list of quotes, with scores and posters. We know that Kutta scored 90 points from 7 quotes. Our null hypothesis is that he randomly selected 7 quotes from the total set of 1138 quotes. The p-value is the probability that he could achieve at least 90 points by this process. If his actual method yields better scores then random drawing, then the p-value will be low.

I have very low opinion of classical frequentist statistics, but it seemed to be very suitable for this task. I am sure that there is already a name for this method I reinvented. Of course, the null hypothesis is ridiculous, so we shouldn't assign much meaning to these numbers. It is just one of the many ways we can solve this ranking task.

Comment author: RobinZ 22 December 2010 09:16:16PM 1 point [-]

Okay, that makes sense - the number is the probability that they could have picked up as many points as they did by picking randomly from the set of all quotes. I understand now.

Comment author: wedrifid 22 December 2010 06:00:41AM 0 points [-]

That's brilliant. I like the theory and the ranking matches about what my intuitive manual ranking would have been too.