gwern comments on Rationalist sites worth archiving? - Less Wrong

22 Post author: gwern 11 September 2011 03:24PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (44)

Sort By: Best

You are viewing a single comment's thread. Show more comments above.

Comment author: gwern 13 September 2011 03:23:21PM 0 points [-]

I've done all those except delicious.com, because I don't know how to confine my spidering to just that tag.

Comment author: John_Maxwell_IV 16 September 2011 12:06:51AM 0 points [-]

I wasn't suggesting you spider everything associated with that tag, just look through it for more blogs. I guess maybe that's too much work?

Comment author: gwern 16 September 2011 12:12:53AM * 0 points [-]

At this point yeah. I now have 56k URLs in the queue, and at 20 seconds a URL... Pareto is the idea here, what are the main sites worth preserving?

Comment author: John_Maxwell_IV 18 September 2011 03:05:58AM 1 point [-]

I guess ribbon farm and Paul Graham would be the 2 big ones from my list.

Comment author: wedrifid 13 September 2011 04:33:06PM 0 points [-]

Comment author: gwern 13 September 2011 05:14:41PM 0 points [-]

How does that help? Going to the Pipe and putting in 'rationality' maxes out at 80 items.