Comment Permalink

not sure how long it's supposed to take; any chance of a % bar? 5-10 minutes total.

The way it works is by going to the relevant section of someone's user page and hitting the 'next page' link until it runs out of results. You only figure out how many comments someone's made at the end, and so a % bar isn't that possible. (One could display the number of comments scraped so far.)

philh9y30

Going that route, I'd suggest 'date of oldest comment scraped so far' instead of/as well as the count.

0Elo9y

I think that would be better than nothing. Also the example of EY; and his karma score made it seem like it could handle a 1000 karma score in a few seconds.

See in context

36 Less Wrong Karma Chart Website

by ScottL

15th Feb 2016

2 min read

36

As a learning exercise, I wrote a web app which shows some charts on your karma score.

I recommend just going to the website and trying it out, but here is a description of it as well if you're interested. To use it just enter your user id in the text box at the top and then press the go button. It will show that it is loading and after a while five charts will be shown.

The first chart is a time series chart which shows you when you have posted a comment or discussion post. This chart allows you to zoom in on any desired area.
The second chart is a time series chart which shows you when you have posted a main post. This chart allows you to zoom in on any desired area.
The third chart shows your cumulative score. This chart allows you to zoom in on any desired area.
The fourth chart shows proportions, i.e. how many comments/posts you have made and how many were positive, neutral or negative.
The fifth and final chart shows information on your total positive and negative scores. This chart allows you to drill down and see where your points have originated from, i.e. from comments or discussion posts or main posts.

Please note that it may take a while to load. I am scraping all the information from your users page. It shouldn't take too long though. On my computer it takes less than a minute to load all the information on my karma score, but it did take around half an hour to load Eliezer_Yudkowsky's karma information. YMMV depending on what computer you are using. It is not your score that determines how long it will take, but the amount of comments and posts that you have made. I recommend using chrome as I haven't tested it in any other browsers.

The karma calculated by LessWrong also might be slightly different to what my web app shows. For example, my webapp shows Eliezers karma total score as 290096 and in LessWrong it is 290174. I am pretty sure that my code is right since I counted one example out and I do know of one bug in the LessWrong code that would effect the total score. There are also other things that LessWrong takes into account that I don't, e.g. karma awards and the troll tax. The difference shouldn't be too major, though, so it shouldn't be a big problem.

The website is hosted on github and the code can be found here.

TLDR: Try out the Less Wrong Karma Chart Website and let me know what you think or if you run into any issues.

Software ToolsSite Meta

Personal Blog

36

New Comment

26 comments, sorted by

top scoring

Click to highlight new comments since: Today at 12:27 PM

[-]philh9y50

Neat! And it seems to work on firefox. Some comments:

I think some users have simply been awarded karma, which your app wouldn't be able to catch. It also won't catch karma lost from paying the troll tax (replying to downvoted comments).

Pressing enter with the box focused didn't start the scraping. I had to click 'go'.

I'd put main and discussion upvotes on the same scale. (So the 'main' y-axis is just 10 times the 'discussion/comment' y-axis.) Right now they don't even have zero in the same place, which is really weird. Maybe also make the scale nonlinear.

It's hard to click-and-drag over the whole width or height of the chart.

I'm not sure how easy this would be, but I'd appreciate a distinction between meetups and non-meetups. (But I think some meetups are in main and some are in discussion.)

I'd like more context on posts/comments without having to visit them. For posts, the title; for comments, maybe the title of the attached post plus a few words (like in the sidebar). It might be too noisy to put that in the hover box; if so, perhaps if I click, the hover box expands and stays there until the next click, and includes an actual link?

You don't actually display total karma anywhere. I had to get it from positive-negative on the pie chart.

The pie chart doesn't have a slice for 'neutral'.

I'd also be interested in seeing cumulative karma as a time series, and 30-day karma as a time series. (This can't be calculated like I think LW calculates it - you'd need to do "karma on last 30 days of comments/posts" rather than "karma given in last 30 days")

This has been mostly negative, so I'll say again that this is really cool. Good job!

[-]ScottL9y50

Pressing enter with the box focused didn't start the scraping. I had to click 'go'.

Fixed

I'd put main and discussion upvotes on the same scale. (So the 'main' y-axis is just 10 times the 'discussion/comment' y-axis.) Right now they don't even have zero in the same place, which is really weird. Maybe also make the scale nonlinear.

I moved main into a separate graph. This should fix the issues.

It's hard to click-and-drag over the whole width or height of the chart.

I could change it so that you can only zoom in on the yaxis like it is here.

I'm not sure how easy this would be, but I'd appreciate a distinction between meetups and non-meetups. (But I think some meetups are in main and some are in discussion.)

Maybe I will look at that later.

I'd like more context on posts/comments without having to visit them. For posts, the title; for comments, maybe the title of the attached post plus a few words (like in the sidebar). It might be too noisy to put that in the hover box; if so, perhaps if I click, the hover box expands and stays there until the next click, and includes an actual link?

I have updated this. Try it out and let me know what you think.

You don't actually display total karma anywhere. I had to get it from positive-negative on the pie chart.

I fixed the total chart to have title that shows the total score.

The pie chart doesn't have a slice for 'neutral'.

The pie chart is meant to show the total score. Since neutral has 0 score I don't think it should be in the graph.

I'd also be interested in seeing cumulative karma as a time series

Would that be something like this with the total score moving up and down over time. I would do this by ordering the comment/post scores by their dates.

and 30-day karma as a time series.

Would this be similar to the cumulative chart above, but just for 30 days.

[-]philh9y30

I have updated this. Try it out and let me know what you think.

Yep, this is an improvement, thanks. I'd probably change the styling slightly, but I'm not sure exactly how. Right now the three

Oh, and I'd recommend adding the year to the timestamp, instead of making me read it off the x-axis.

Re timeseries: for cumulative, I mean the y-axis is "total karma accumulated on all things posted by date X", so essentially the integral of the existing graph. I guess it might look similar to the thing you posted, but (at least for me, and for others who mostly post neutral/positive things) it would only rarely go down.

The 30-day one would be "total karma accumulated on all things posted in the thirty days preceding X".

[-]ScottL9y20

I'd recommend adding the year to the timestamp, instead of making me read it off the x-axis.

Done

I also did the cumulative chart. I will think about allowing the user to set the dates that are shown on the cumulative chart. It would start with the first comment/post and end with the last, but it would allow you to change the start or end dates if you want.

[-]plex9y40

Aaron Swartz's highest ranked LW posts can be found with this. I bet a lot of people would love to be able to find his highest rated posts, and share some more widely.

[-]Vaniver9y40

You might want to force the 0 line for both axes to be the same; when mine first loaded my first thought was "why do I have so many negative red boxes?". That the points on the time series are clickable links is a very nice touch.

[-]ScottL9y50

I moved the main posts into a separate chart.. It should be less confusing now.

[-]Viliam9y20

Is this purely client-side? If there is a database available, it could be useful to cache the scraped data, just in case there are too many people wanting to see the results for Eliezer. The cached data could expire after a week.

[-]ScottL9y20

Is this purely client-side?

Yes. I hosted it in Github Pages because it's free. The down side is that it only serves static content.. I might be able to use something like firebase, but I don't really want to. I will see how easy it is to create a link that allows you to download the data to a csv file.

[-]Mati_Roy5y10

It doesn't seem to be working anymore. Maybe add this info at the top of the post.

[-][anonymous]9y00

This app pleases me.

Iamnotnarcissisticatall.

[This comment is no longer endorsed by its author]Reply

[-]Elo9y00

Any chance at getting all comments and posts on one of those graphs? For all users over time? I'd like to see which posts score >=5 and <5 karma..

[-]ScottL9y00

I don't think so. See Vaniver's comment which describes what the site is doing. I don't know the process, but maybe you can submit a DB request to the trike apps team for this data.

[-]Elo9y10

script should be able to be run on http://lesswrong.com/comments/ without much modification...

[-]Vaniver9y00

Yeah, I recommend against scraping LW that way instead of making a DB request / building a page that makes that DB request.

[-]Lumifer9y00

instead of making a DB request

Can one? In general, is there an available (and working) API to LW?

[-]Vaniver9y00

The following is what I had in mind for that section:

maybe you can submit a DB request to the trike apps team for this data.

There is an API but I think it doesn't have the ability to do everything that we'd like it to. See this issue as an example of a missing feature.

But if you have the ability, building that feature and submitting it as a pull request is better than doing it the long way (since it also helps everyone else who wants to do that sort of thing in the future).

[-]Lumifer9y40

building that feature

A feature that allows anyone on the 'net to bang on your backend database is not something to be taken lightly :-)

I think LW first needs a coherent API spec, then a framework (including thing like auth tokens, rate limiters, etc.), and only then specific abilities to e.g. ask for all comments of a particular user.

[-]ScottL9y00

You are probably right. I would assume that I can also get the post information from this: http://lesswrong.com/r/all/recentposts/ . A graph with this much data probably wouldn't be useful as it would be too busy. I will look into writing something else to get this data into a usable format.

Edit: Your link only has the main comments, not the discussion ones. I'm not sure what to get all the comment information from.

[-]Elo9y00

maybe 3 months worth; for the past 3 years? 12 graphs in total; showing the trend...?

I figured that with the zoom it would be able to be use-able. but separate graphs work too.

[-]Elo9y00

not sure how long it's supposed to take; any chance of a % bar? 5-10 minutes total.

excellent visualisation, very helpful. Used it to see which discussion posts of mine were good. Generally the ones I spent more time on and felt like they were higher quality posts. my one concern for myself is the difference between a top notch post and a crappy one is about 5 karma. Bad discussion posts by me are around 4 karma or less, good ones are 9+ with some in-betweeners. Doesn't seem like they are far apart in score even though they are far apart in quality (to me). Going to think about it and see if I can take further strides upwards from here...

[-]Vaniver9y20

not sure how long it's supposed to take; any chance of a % bar? 5-10 minutes total.

[-]philh9y30

Going that route, I'd suggest 'date of oldest comment scraped so far' instead of/as well as the count.

[-]ScottL9y10

I combined the above suggestions. While it is scraping the data, it now also has a table showing the last scraped elements posted date, the number of items scraped and the scraped score. This is split into two rows one for comments and one for posts.

[-]Elo9y00

I think that would be better than nothing. Also the example of EY; and his karma score made it seem like it could handle a 1000 karma score in a few seconds.

[-]Brillyant9y-10

This is so good. Great work.

Moderation Log