To answer my own question, here are a few write-ups I found about why the quality of an online community tends to decline over time, and what can be done about this problem:

• A Group Is Its Own Worst Enemy by Clay Shirky
• What I've Learned from Hacker News by Paul Graham
• Trolls by Paul Graham
• Well-Kept Gardens Die By Pacifism by Eliezer Yudkowsky

Also, the book "The Virtual Community" that Richard Hollerith mentioned is available online, although I wasn't able to find much information in it about the specific topic at hand.