Epiphany comments on Number of Members on LessWrong - Less Wrong

3 Post author: Epiphany 17 August 2012 05:47AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (32)

You are viewing a single comment's thread. Show more comments above.

Comment author: Epiphany 18 August 2012 03:09:36AM *  -1 points [-]

Multiple pages aren't being counted. From what I understand, Google doesn't just follow dynamically generated next links like that. It spiders, going around in a web-like pattern. How many times would it end up visiting the same pages if it followed every comment to it's original discussion? A lot. That would be a waste of resources.

To test this, I looked at the url that appears when you press the next button. The site adds some pagination variables into the URL. The word "count" appears. So, you can do the following query and observe the following things:

site:lesswrong.com/user -"submitted by" -"comments by" -count

site:lesswrong.com/user -"submitted by" -"comments by" -com (for comparison)

And observe:

A. It does not divide the number of results into a small fraction of the original number like you'd expect it to. We're comparing 9,820 total users with the original method (at this moment) with 9,460.

B. Removing "com" from the query shows zero results which verifies that adding -count would be removing pages generated in those next links, had they been included.

C. If you click on random pages of Google results, you won't see those count and after variables in the URLs (Or at least I didn't and I feel fairly confident that they won't be there.)

D. If Vladmir is correct in this post then just looking at one of those lines where the user's comments are totaled (the line where 900 have 25 comments) reveals that, by removing "count" from the query, we should have lost at least 1800 from the total. Nowhere near that many were lost, and a lot more should have been lost than that because I only subtracted a tiny fraction of the comments pages on this site in the example.

Comment author: dbaupp 20 August 2012 06:04:23AM 2 points [-]

Google doesn't just follow dynamically generated next links like that

I'm pretty sure Google normally does follow dynamic links. In this case, though, it doesn't, since they are marked nofollow.