Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

arundelo comments on Intellectual Hipsters and Meta-Contrarianism - Less Wrong

149 Post author: Yvain 13 September 2010 09:36PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (335)

You are viewing a single comment's thread. Show more comments above.

Comment author: arundelo 09 July 2017 06:24:27PM *  0 points [-]

Why does archive.is not obey robots.txt?

Because it is not a free-walking crawler, it saves only one page acting as a direct agent of the human user.

--archive.is faq

A few months ago we stopped referring to robots.txt files on U.S. government and military web sites [...] As we have moved towards broader access it has not caused problems, which we take as a good sign. We are now looking to do this more broadly.

--archive.org blog, 2017-04-17