Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Print ready version of The Sequences

14 Post author: Jordan 06 November 2010 01:21AM

I've been wanting a printable copy of the sequences to read through in meatspace. I wrote a quick scraper and uploaded the results here http://pwnee.com/Sequences/list.html

Inter-linking doesn't work, but I just wanted a printable version anyway.

Comments (31)

Comment author: Vladimir_Golovin 06 November 2010 09:34:49AM *  8 points [-]

I'll throw together a python script to get the job done, and will post the result here.

LessWrong is written in Python, so instead of writing a scraper script (which I suppose you were going to do), you can become a volunteer, grab the code at GitHub, write a proper script that can access the database directly, and commit it into the LW codebase.

Comment author: ciphergoth 20 November 2010 04:45:54PM 6 points [-]

I've now written a fairly sophisticated scraper for Eliezer's blog posts based on lxml, which

  • follows the Author links in "Article Navigation" to fetch all articles
  • fetches and parses all articles
  • identifies the title, body, and date
  • fixes hrefs to internal references where possible, including where the reference is to Overcoming Bias and redirects back to Less Wrong.
  • fixes all the weird Unicode characters as best I can where I can make a plausible guess
  • finds and adds the forward references in all blog posts
  • caches all network operations in a very simple dumb way
  • writes them all out as very simple HTML with a very simple HTML contents page, in a form that Calibre works well on.

I'll share the script when I have time to sort out publishing via Mercurial, or email me if you'd like a snapshot copy - paul at ciphergoth dot org.

Comment author: multifoliaterose 20 November 2010 05:10:29PM 0 points [-]

Great to hear!

Comment author: Jordan 07 November 2010 12:56:04AM 3 points [-]

Hi all, thanks for the comments. I went ahead and wrote the scraper. I might do a proper PDF version, integrated into the LW codebase once I get some time.

Comment author: Vladimir_Golovin 07 November 2010 06:18:40AM *  3 points [-]

I haven't checked all the scraped sequences, but I saw at least one missing post: "Invisible Frameworks", which should be at the end of the Metaethics sequence according to the wiki index.

Comment author: Document 06 January 2011 08:00:46AM *  0 points [-]

Thanks; I've downloaded them for possible offline reading. I checked the post titles in How to Actually Change Your Mind against the wiki and found Politics is the Mind-Killer and the text of You Can Face Reality missing.

Comment author: imonroe 15 April 2012 08:05:38PM 1 point [-]

Just FYI, the link above (http://pwnee.com/Sequences/list.html) currently 404's.

Comment author: jb55 23 November 2010 12:17:59AM *  1 point [-]

I put together a repository for print-friendly versions of the Sequences. It consists of a pretty naive scraper which I feed into pandoc. It spits out markdown documents which I then convert to epub, pdf, etc.

I have the first two sequences completed and I plan on doing all of them eventually. Check it out:

http://jb55.com/lesswrong

https://github.com/jb55/lesswrong-print

Comment author: NihilCredo 07 November 2010 07:41:41PM 1 point [-]

There seems to be a relatively easy way to fix interlinking, at least within the same sequence:

1) Open the file with Notepad++ or any text editor of your choice.

2) Do a regex replacement to add an HTML anchor after every header, with the same anchor name as in the link in the header

3) Do a regex replacement to turn every link from /lw/tworandomcharacters/articlename into #articlename

I may get around to do it later.

Comment author: Document 06 January 2011 08:27:57AM 0 points [-]

Personally I'd rather have all the links point to lesswrong.com, so there'd be no broken ones and so the title links could be used to go to each post's comment threads.

Comment author: Document 06 November 2010 07:26:42PM *  1 point [-]

This page has posts from 2006 November 22 to 2008 March 13.

Comment author: Document 06 January 2011 08:45:55AM 0 points [-]

Also, the wiki links to lw2ebook by OneWhoFrogs, which has them in epub and mobi format.

Comment author: rhollerith_dot_com 06 November 2010 06:49:01PM *  1 point [-]

Forgive my ignorance: has the author of the Sequences given permission for the distribution (e.g., over the public internet) of EPUBs and such of his work -- as opposed to people making EPUBs or such for their own consumption, which is not considered by most people to be "distribution" of a work?

Comment author: jsalvatier 06 November 2010 07:12:32PM 1 point [-]

I haven't seen him explicitly do so, but I imagine he would.

Comment author: MichaelAnissimov 06 July 2012 02:51:23AM 0 points [-]

This is currently broken... any chance of getting it fixed? Thanks for doing this!

Comment author: benbenson 01 August 2011 03:16:36PM 0 points [-]

Where is Eliezer's Metaethics Sequence?

Comment author: MixedNuts 01 August 2011 03:33:40PM 0 points [-]

Wiki index

Welcome to Less Wrong!

Comment author: ciphergoth 10 November 2010 05:49:46PM 0 points [-]

http://pwnee.com/Sequences/GetSequences.py gives me "Internal Server Error"!

Comment author: Larks 06 November 2010 02:23:19PM 0 points [-]

Once Eliezer will finish the Rationality Books we'll be able to use them. I think much of Mysterious Answers to Mysterious Questions, Reductionism, How to Actually Change Your Mind and Map and Territory will be contained; the core stuff.

Comment author: Relsqui 06 November 2010 06:37:27AM 0 points [-]

Now you've got me wondering what it would take to make an ereader-ready file out of them.

Comment author: Vladimir_Golovin 06 November 2010 07:59:38AM *  7 points [-]

The most important decision is the format. PDF support is pretty ubiquitous, but it's not reflowable (i.e. can't adapt to different screen sizes and user-adjusted fonts automatically), which looks bad on devices that have screens of different size than the size encoded in the PDF. Many devices implement some form of PDF reflow, which may work well for simple-layout PDFs (with no pullquotes or columns).

There's EPUB, a good reflowable open-source XML-based format, but it has its downsides too: not all ereaders support it, and, as I gather from the wikipedia page, the situation with linking is unclear -- which is important because the sequences are heavily interlinked.

There's also CHM (Windows help file format) which is both reflowable and linkable since it's based on HTML/CSS, but, as far as I know, few ereaders support it.

Based on my own experience, I strongly prefer EPUB. However I have no experience with its linking support.

I'd first try a linked EPUB and see how well its linking works on popular devices, or, as a second option, try a simple, sinlge-column, no-pullquotes PDF encoded for a 6-inch screen and see how well it reflows on various readers.

Comment author: humpolec 06 November 2010 10:14:44AM 1 point [-]

PDFs are pretty much write-only, and in my experience (with Adobe Acrobat-based devices) reflow never works very well. As long as you use a sane text-based ebook format, Calibre can handle conversion to other formats.

So I recommend converting into - if not EPUB, then maybe just a clean HTML (with all the links retained - readers that support HTML should have no problems with links between file sections).

Comment author: Vladimir_Golovin 06 November 2010 04:01:11PM *  2 points [-]

then maybe just a clean HTML

Yes, some readers (e.g. Pocketbooks) can handle HTML, but even the latest Sony readers cannot. Kindle does have HTML support "via conversion" but I don't know if it can correctly convert 600 or so interlinked documents.

Comment author: humpolec 06 November 2010 05:33:39PM *  2 points [-]

600 or so interlinked documents

I was thinking more of a single, 600-chapter document.

(Actually this is why I think Sequences are best read on a computer, with multiple tabs open, like TVTropes or Wikipedia - not on an e-reader. I wonder how Eliezer's book will turn out...)

Comment author: Jordan 06 November 2010 08:20:26AM 0 points [-]

Thanks for the info!

I suppose for PDFs we could just have a list of various widths available.

Comment author: Vladimir_Golovin 06 November 2010 09:25:36AM *  2 points [-]

Yes, but you'll still have to change the page size and recompile each of the 600 or so PDFs -- this is a huge amount of work if you're planning to do it manually. Generating PDFs for different screen sizes automatically (e.g. using a Python library) would be a better solution. I'm not familiar with PDF generation under Python, but I did a quick Google search and here's what I found:

http://www.devshed.com/c/a/Python/Python-for-PDF-Generation/
http://www.reportlab.com/software/opensource/

Comment author: Risto_Saarelma 08 November 2010 09:59:05AM *  1 point [-]

I've had success turning texts in various formats into EPUB files using Calibre.

ETA: Tried Calibre for making an ebook of the scrapings. Downloaded each sequence file, plus the index html, into the same directory. Modified index to point to local files instead of fully qualified URLs. Pointed Calibre at the index file, and it slurped the actual sequence files nicely. Resulting epub looks fine, corresponds to a 1300 page physical book and includes the inline images.

Comment author: Risto_Saarelma 10 November 2010 09:12:44PM *  1 point [-]

Made a script to do this.

Install the prerequisites: apt-get install wget calibre tidy, or whatever works on your favorite OS.

Then do

#!/bin/bash

TMPDIR=$(mktemp -d)
pushd $TMPDIR
wget -H -np -nd -k -p -r -l 1 http://pwnee.com/Sequences/list.html
tidy -m *.html
ebook-convert list.html lesswrong.epub
popd
cp $TMPDIR/lesswrong.epub .

Comment author: Jordan 06 November 2010 07:30:21AM 0 points [-]

Once you've harvested the text it would be straightforward to make a PDF, but I'm not sure how many ereaders support that format.

Comment author: CarlShulman 06 November 2010 08:42:42AM 0 points [-]

You can convert a pdf for kindle reading automatically, but the formatting winds up a bit spotty.

Comment author: Relsqui 06 November 2010 07:55:37AM 0 points [-]

Me neither. Pretty sure the Kindle does but I'm not really familiar with the field, as I don't own one. (But I'd like to, which is why I was curious.)