I feel I should clarify that I don't think it's "good", so much as "less bad than the alternatives".

Your proposal requires a lot of work: both coding, and the social effort of getting everyone to use new custom software on their backends. So we should compare it not to existing alternatives, but to potential solutions we could implement at similar cost.

Let's talk about a concrete alternative: a new protocol, using JSON over HTTP, with an API representing CRUD operations over a simple schema of users, posts, comments, et cetera; with some non-core features provided over existing protocols like RSS. An optional extension could provide e.g. server push notifications, but that would be for performance or convenience, not strictly for functionality.

It would be simpler to specify (compared to contorting NNTP), and everyone's used to JSON/HTTP CRUD. It would be simpler to implement - almost trivial, in fact - in any client or server language, easier than writing an HTTP to NNTP gateway even though NNTP servers already exist. It would better match the existing model of forums and users. And it would (more easily) allow integration with existing forum software, so we don't have to tell everyone they have to find a Linux host and install custom software, rather than finding a Wordpress+MySql host and installing this one plugin.

Part of what got me on this track in the first place is the distributed nature of the diaspora. We have a network of more-and-more-loosely connected subcommunities that we'd like to keep together, but the diaspora authors like owning their own gardens. Any unified system probably needs to at least be capable of supporting that, or it's unlikely to get people to buy back in. It's not sufficient, but it is necessary, to allow network members to run their own server if they want.

I think the current model is fine. Posts and comments are associated with forums (sites), and links to them are links to those sites. (As opposed to a distributed design like NNTP that forwards messages to different hosts.) User accounts are also associated with sites, but sites can delegate authentication to other sites via Google/Facebook login, OpenID, etc. Clients can aggregate data from different sites and crosslink posts by the same users on different sites. A site owner has moderator powers over content on their site, including comments by users whose account is registered at a different site.

The UXs for posters, commenters, readers, and site owners all need to be improved. But I don't see a problem with the basic model.

That being said, it's of interest that NNTP doesn't have to be run distributed. You can have a standalone server, which makes things like auth a lot easier.

Then you suffer all the problems of NNTP's distributed design (which I outlined in my first comment) without getting any of the benefits.

The auth problem as I see it boils down to "how can user X with an account on Less Wrong post to e.g. SSC without needing to create a separate account, while still giving SSC's owner the capability to reliably moderate or ban them." There are a few ways to attack the problem; I'm unsure of the best method but it's on my list of things to cover.

It seems easy to me. The user account lives on LW, but the actual comment lives on SSC, so an SSC mod can moderate it or ban the user from SSC. There are plenty of competing cross-site authentication systems and we don't even have to limit ourselves to supporting or endorsing one of them.

Also, we can just as easily support non-site-associated accounts, which are authenticated by a pubkey. System designers usually don't like this choice because it's too easy to create lots of new accounts, but frankly it's also very easy to create lots of Google accounts. SSC even allows completely auth-less commenting, so anyone can claim another's username, and it hasn't seemed to hurt them too badly yet.

This is a huge value, though, because most extant web forum, blogging, etc software is terrible for discussions of any nontrivial size.

I'll just repeat my core argument here. Extant NNTP software is far more terrible, if you penalize it for things like not supporting incoming hyperlink, not allowing editing posts, not having karma, no existing Web clients, etc. Adding those things to NNTP (both the protocol and the software) requires more work than building a new Web-friendly forum standard and implementations, and would also be much more difficult for site admins to adopt and install.

That's a serious question, because I'd love to hear about alternative standards. My must-have list looks something like "has an RFC, has at least three currently-maintained, interoperable implementations from different authors, and treats discussion content as its payload, unmixed with UI chrome."

I don't know of any concrete ones, but I haven't really searched for them either. It just feels as though it's likely there were some - which were ultimately unsuccessful, clearly.

Having an RFC isn't really that important. There are lots of well-documented, historically stable protocols with many opensource implementations that aren't any worse just because they haven't been published via the IETF or OASIS or ECMA or what have you.

But consider the following problem: Find and display all comments by me that are children of this post, and only those comments, using only browser UI elements, i.e. not the LW-specific page widgets. You cannot -- and I'd be pretty surprised if you could make a browser extension that could do it without resorting to the API, skipping the previous elements in the chain above. For that matter, if you can do it with the existing page widgets, I'd love to know how.

If you mean parse the document object model for your comments without using an external API, it would probably take me about a day, because I'm rusty with WatiN (the tool I used to used for web scraping when that was my job a couple years ago). About four hours of that would be setting up an environment. If I was up to speed, maybe a couple hours to work out the script. Not even close to hard compared to the crap I used to have to scrape. And I'm definitely not the best web scraper; I'm a non-amateur novice, basically. The basic process is this: anchor to a certain node type that is the child of another node with certain attributes and properties, and then search all the matching nodes for your user name, then extract the content of some child nodes of all the matched nodes that contain your post.

WatiN:: http://watin.org/

Selenium: http://www.seleniumhq.org/

These are the most popular tools in the Microsoft ecosystem.

As someone who has the ability to control how content is displayed to me (tip - hit f12 in google chrome), I disagree with the statement that a web browser is not a client. It is absolutely a client and if I were sufficiently motivated I could view this page in any number of ways. So can you. Easy examples you can do with no knowledge are to disable the CSS, disable JS, etc.

But another is that web sites, able to provide a look and feel appropriate to their community, plainly outcompeted networks of plaintext content.

Ah, yes, porn as the engine of technology.

The web had pictures, "networks of plaintext content" did not. Case closed.

I think you are making the argument for what was known as "the semantic web" -- the term seem to have fallen into disuse, though.

I also think that my browser is a client. It's not a client for structured raw information, though, because there is no server which feeds it that (a client is just one half of a client-server pair, after all. A server-less client is not of much use). My browser is a client for web pages which used to mean mostly HTML and nowadays mean whatever JS can conjure.

By the way, where does RSS fit into your picture of the world?

It would make messages mutable (or perhaps expose an immutable history, but make the 'current' reference mutable, like git does)

As an aside, git is about as good a fit as NNTP (which is to say, neither is really all that good in my opinion).

Git has immutable messages, but it also has mutable references (branches) for edits, and the references can be deleted for retractions. It has a tree structure for comments. It has pseudonymous authentication (sign your commits). It has plenty of room for data and metadata (e.g. specify a standard mapping filenames to headers). It can run over HTTP and has existing servers and clients including Javascript ones. It can be deployed in a centralized model (everyone pushes to the same server) but others can mirror your server using the same protocol, and there are RSS and email gateways available. Messages (commits) have globally unique IDs, allowing for incoming links. It makes your server state trivial to backup and to mirror. I could go on.

In fact, someone has already thought of this and wrote a similar system, called GitRap! (I didn't know about it before I checked just now.) It doesn't do exactly what I described, and it's tied to github right now, but you can view it as a POC.

To be clear: I am 95% not serious about this proposal. Solving the vastly simpler centralized problem is probably better.

I completely agree with everything you've said in the first half of your post. But I strongly disagree that NNTP is a good choice for a backend standard. (At this point you can say that you'll argue your case in a future post instead of replying to this comment.)

The NNTP model differs from modern forums in a crucial respect: it is a distributed system. (I use this in the sense of 'decentralized'.) More precisely, it is a AP system: it doesn't provide consistency in synchronizing messages between servers (and it makes messages immutable, so it gets the worst of both worlds really). This directly leads to all the problems we'd have in using NNTP for a forum, such as no true editing or deleting of messages. Because a message is not tied to a domain (or a server), and is not referenced but copied to other servers, authentication (proving you own an identity and wrote a post) and authorization (e.g. mod powers) become nontrivial. Messages don't have globally unique IDs, or even just global IDs. Implementing something like karma becomes an interesting computer science exercise involving decentralized consensus algorithms, rather than a trivial feature of a centralized database. And so on.

But we don't need to deal with the problems of distributed systems, because web forums aren't distributed! What we want is a standard that will model the way forums already work, plus or minus some optional or disputed extensions. Making NNTP resemble a forum would require adding so many things on top that there's no point in using NNTP in the first place: it just doesn't fit the model we want.

A good forum model would tie users and messages to a particular server. It would make messages mutable (or perhaps expose an immutable history, but make the 'current' reference mutable, like git does). It would at least provide a substrate for mutable metadata that karma-like systems could use, even if these systems were specified as optional extensions to the standard. It would allow for some standardized message metadata (e.g. Content-Encoding and Content-Type equivalents). It would pretty much look like what you'd get if you designed the API of a generalized forum, talking json over http, while trying not imagine the clientside UI.

There's probably an existing standard or three like this somewhere in the dustbin of history.

NNTP also has a lot of more minor ugliness that I'd be happy to argue against. It's one of the http/mime/email family of headers-body encodings, which is well known for producing fragile implementations (quick, recite the header folding rules!) and are all subtly different from one another to make sure everyone's sufficiently confused. It relies on sometimes complex session state instead of simple separate requests. There's a bunch of optional features (many of historical interest), but at the same time the protocol is extremely underspecified (count how many times it says you SHOULD but not MUST do something, and MAY do quite the opposite instead). Any client-server pair written from scratch inevitably ends up speaking a highly restricted dialect, which doesn't match that of any other client or server.

Given all of this, the only possible value of using NNTP is the existing software that already implements it. But there's no implementation of an NNTP client in Javascript (unless you want to use emscripten), if only because Javascript in a browser can't open a raw TCP socket, so until the recent advent of websocket-to-tcp proxies, nobody could write one. And implementing a new HTTP-based server to a new (very simple) standard, basically just CRUD on a simple schema, is much easier than writing an NNTP JS client - IF you're willing to not make a distributed system.

A final note: one may well argue that we do want a distributed, decentralized system with immutable messages (or immutable old-message-versions), because such systems are inherently better. And in an ideal world I'd agree. But they're also far, far harder to get right, and the almost inevitable tradeoffs are hard to sell to users. I'm not convinced we need to solve the much harder distributed version of the problem here. (Also, many decentralization features can be added in a secondary layer on top of a centralized system if the core is well designed.)

Making karma an even less useful indicator strikes me as a poor trade-off for spiting Eugine...

I've already banned them and their comments.

And now that I actually write it down and compare it to previous online communities (including a few mixed online/offline) I've been part of and loved, and which have universally followed the same pattern of growth, overgrowth, loss of some driving valuable members without obvious replacement, slow decay into irrelevance (to me; at least 2 of them are going strong, just with a different feel than when I was involved)), I'm pretty pessimistic.

I'm going to put some effort into being OK with LW as it is, enjoying the parts I enjoy and being willing to follow those parts I'm missing to their new homes.

My analysis saw the fundamental problem as the yearning for consensus. What was signal? What was noise? Who was trolling? Designers of forum software go wrong when they believe that these are good, one place questions with actual one place answers. The software is designed in the hope that its operation will yield these answers.

My suggestion, Outer Circle got discussed on Hacker News under the title Saving forums from themselves with shared hierarchical white lists and I managed to flesh out the ideas a little.

Then my frail health got even worse and I never did anything more :-(