Here is some clarification from Zinsser himself (ibid.):

"Who am I writing for? It's a fundamental question, and it has a fundamental answer: You're writing for yourself. Don't try to visualize the great mass audience. There is no such audience - every reader is a different person.

This may seem to be a paradox. Earlier I warned that the reader is... impatient... . Now I'm saying you must write for yourself and not be gnawed by worry over whether the reader is tagging along. I'm talking about two different issues. One is craft, the other is attitude. The first is a question of mastering a precise skill. The second is a question of how you use the skill to express your personality.

In terms of craft, there's no excuse for losing readers through sloppy workmanship. ... But on the larger issue of whether the reader likes you, or likes what you are saying or how you are saying it, or agrees with it, or feels an affinity for your sense of humor or your vision of life, don't give him a moment's worry. You are who you are, he is who he is, and either you'll get along or you won't.

N.B: These paragraphs are not contiguous in the original text.