wedrifid comments on Rationalist sites worth archiving? - Less Wrong

22 Post author: gwern 11 September 2011 03:24PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (44)

You are viewing a single comment's thread. Show more comments above.

Comment author: wedrifid 21 August 2013 05:42:12PM 2 points [-]

That would already be covered by my own reading of it, my browser history being the main source of URLs for archiver-bot.

You're making a permanent backup of everything you ever read on the internet? That's... that's... well I suppose data storage is cheap these days. It makes perfect sense. Reading your scripting instructions now.

Comment author: gwern 21 August 2013 06:52:29PM 1 point [-]

Not everything; I filter out things I am sure I won't want in the future and things I strongly expect to be available & which would take up a lot of space (Wikipedia in particular), and the bot is rate-limited by the IA/WebCite submissions. Increasingly more stuff is difficult to archive as sites load stuff via JS. But much of what I read, yes.