DataPacRat comments on Open thread, Sep. 12 - Sep. 18, 2016 - Less Wrong Discussion
You are viewing a comment permalink. View the original post to see all comments and the full post content.
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (110)
Depends how much storage space you are willing to buy.
One of my fantasies is a Raspberry Pi that automatically downloads all Wikipedia updates each month or so, to keep a local copy. The ultimate version of this would do the same for every new academic article available on Sci-Hub.
Sci-Hub is the largest collection of scientific papers on the planet, and has over 58 million academic papers. If they average 100 kB a piece, that's only 5.8 TB. If they average 1MB each, then you would need to shell out some decent cash, but you could in theory download all available academic papers.
Someone may even have already done something like this, and put the script on GitHub or somewhere. (I haven't looked.)
(Also, nice username. :) )
EDIT: It turns out there's a custom built app for downloading and viewing Wikipedia in various languages. It's available on PCs, Android phones, and there's already a version made specially for the Pi: http://xowa.org/home/wiki/Help/Download_XOWA.html
I wonder how difficult it would be to translate all of Sci-Hub into a wiki format that the app could add and read. You'd probably have to modify the app slightly, in order to divide up all the Sci-Hub articles among multiple hard drives. It might make the in-app search feature take forever, for instance. And obviously it wouldn't work for the Android app, since there's not enough space on a MicroSD card. (Although, maybe a smaller version could be made, containing only the top 32GB of journal articles with the most citations, plus all review articles.)
Even just converting science into a Wikipedia-like format would be useful for the sake of open access. Imagine if all citations in a paper were a hyperlink away, and the abstract would display if you hovered your mouse over the link. (The XOWA app does this for Wikipedia links.)
For Wikipedia, I've been reasonably satisfied with Kiwix for software, and their updated-every-month-or-three copies of Wikipedia, and the related Wikimedia foundation sites, at http://wiki.kiwix.org/wiki/Content_in_all_languages .
Unfortunately, I don't have "decent cash" to shell out. I've seem some setups at /r/DataHoarder that I would be extremely happy to ever own, but don't expect to until typical HDs are an order of magnitude or two bigger than today's. By which time I expect people will have come up with brand-new forms of data to fill the things with. :)
It's not just a nom-de-net, it's a way of life. :)