[…] they set about looking for ways to make an operating system do most of what the current version of Windows does while being more [compressed].

We have an actual example of this here (also, the last progress report). The punchline is "Personnal computing in one book" (400 pages × 50 lines per page means 20K lines of code). It is meant to do basically the work of Windows + Office + IE + Outlook. And the compilers are included in those 20 thousand lines.

They end up doing a lot of things that are only applicable to their situation, and couldn’t be used to make a much more powerful operating system. For example, they might look for ways to recycle pieces of code, and make particular pieces of code do as many different things in the program as possible.

Well, no.

They do look for ways to maximize code recycling. However, the result is not less power. On the contrary, they achieve unmatched flexibility. Two examples:

• Their graphic stack draws everything, from characters on a page to the very windowing system. As a result, if you suddenly want to rotate a window (and its content) by any angle, you just need to write 2 lines of code to add the feature.
• Their language stack is tiny, yet quite capable. It goes from assembly to Javascript in less than 2000 lines. As a result, adding a new language (say Prolog) takes typically one or two hundred lines of additional code. That makes domain specific languages cheaper than they used to be.

Now to go from flexibility to power, one does need human input. But at least it's easier.

(Note that I have swept the runtime performance problems under the carpet. My bet is, if we generalize FPGA-like processors (with memristors?), it won't matter, because one could optimize the hardware for the software, instead of optimizing the software for the hardware.)