In reddit and the like, how is consensus built around whether someone is a troll and/or is trolling in a particular case?

On reddit there is just upvotes and downvotes. Reddit doesn't have developed social mechanisms for dealing with trolls, because the downvotes work most of the time. Developing troll technology like the concern troll and the downvote preventer to hack the hivemind/vote dynamic is the only way to succeed.

4chan doesn't have any social mechanisms either, just the culture. Communication is unnecessary for social/cultural pressure to work, interestingly. Once the countertroll/troll/troll-detector/trolled/troll-crier hierarchy is formed by the memes and mythology, the rest just works in your own mind. "fuck I got trolled, better watch out better next time", "all these people are getting trolled, but I know the OP is a troll; I'm better than them" "successful troll is successful" "I trolled the troll". Even if you don't post them and no-one reacts to them, those thoughts activate the social shame/status/etc machinery.

I think I understand concern trolling, which I understand to be giving advice which actually weakens the receiver's position, though I think the coinage "hlep" from is more widely useful

Not quite. A concern troll is someone who comes in saying "I'm a member of your group, but I'm unsure about this particular point in a highly controversial way" with the intention of starting a big useless flame-war.

Havn't heard of hlep. seems interesting.

but what's downvote preventer

The downvote preventer is when you say "I know the hivemind will downvote me for this, but..." It creates association in the readers mind between downvoting and being a hivemind drone, which people are afraid of, so they don't downvote. It's one of the techniques trolls use to protect the payload, like the way the concern troll used community membership.

I've been mulling the question of how you identify and maintain good will-- to my mind, a lot of community breakdown is caused by tendencies to amplify disagreements between people who didn't start out being all that angry at each other.

Yes. A big part of trolling is actually creating and fueling those disagreements. COINTELPRO trolling is disrupting peoples ability to identify trolls and goodwill. There is a lot of depth and difficulty to that.