Some sites are free due to the coronavirus and we should scrape them. Please list any temporarily-free websites you know of, such as https://www.uptodate.com/

To download sites I recommend https://github.com/ArchiveTeam/grab-site

Consider whether you want to use --no-offsite-links. Otherwise it will pull every linked page and its resources (and sometimes even embedded videos).

New Comment
3 comments, sorted by Click to highlight new comments since:

Why is this a good idea?

I understand that this is great opportunity to scrape otherwise-inaccessible sites, but whom is this intended to benefit? Is it just the people scraping?

Is this not malicious abuse of charitable giving?

I agree with the question. I don't know if we should avoid this. But I think we should consider whether, for example, these sites are opening up with the implicit expectation that people will take advantage of it for covid-related purposes; and whether, if we just start scraping everything, that'll make them or similar sites less likely to open up in future.

The ACM is offering free download of their articles: https://dl.acm.org/