If anyone is curious about regular archivers, Joyce became less compressible throughout his life. The compression ratios (bytes per 100 characters) for Dubliners, Portrait, Ulysses, and Wake are: by gzip -9: 38, 38, 42, 47; for paq8l -7: 24, 24, 26, 33. LZMA and PPMd interpolate these numbers in unsurprising ways. Dubliners and Portrait seem about as compressible as other fiction in English.

Of course, I performed these calculations using a server in Australia, where Finnegans Wake is in the public domain.

This comment was prompted by Finnegans Wake seeming like an odd choice of a novel. War and Peace is a more prototypical novel, so you probably didn't mean anything by the choice.

Can anyone suggest other hard to compress novels?

One interesting compression of Finnegans Wake that goes beyond a .zip file is a plot summary. Obviously it is lossy compression, but how lossy?

One way to think about it is to imagine you are given a plot summary of Finnegans Wake, and asked to reconstruct it. What additional information would you want? A reasonably extensive knowledge of the English language and its grammar, certainly. Most likely a description of Joyce's writing style. Knowledge of human psychology and of the setting.

Obviously, we only need to include as much information as is actually used. If Finnegans Wake never contains the word "indubitably" then we don't need its definition. Also, ideally, all of this is written in some sort of natural representation with no redundancy, rather than in English, but we can think about writing the above in English as an approximation.

Knowing the algorithm, we can then add corrections. Suppose, when we take the above information, and try to reproduce the text, we end up putting a semicolon instead of a period 600 characters in (perhaps semicolons are usually more consistent with Joyce's style, but here he was feeling capricious). We could add a note to the effect of "600 characters: period, not semicolon". A bunch of these notes (which don't really take up much space) together with the information above make up our perfectly compressed string.

Even if a computer simulated the universe, and even if it located earth, and even if it found a text titled "Finnegans Wake," there a very large number of things that, throughout the universe, might be titled Finnegans Wake! So the computer would still need some algorithm to recognize the right Finnegans Wake.

So, universe-independent compression is where it's at.

It contains lots of patterns making it easy to compress using a regular archiver, but can we do much better than that?

We can do a little bit better but not much better. We can use some tricks like dictionaries, PCFG grammars, and so on, but there's just a hard limit to how much we will be able to achieve, because we need to pay a model-specification cost to encode the dictionary or grammar, and a single novel just isn't all that long.

What I think is interesting is what happens when we consider larger datasets. What is the K-complexity of all the books in the library of Congress? Here we should be able to do much better by using a specialized compressor than by using a regular archiver. Because now we can afford to use all kinds of high-concept tricks, because the cost of encoding those tricks will be amortized over a far larger dataset.