Thanks for writing this. I think the bullets under "things I learned" is a reasonable description of the points I was trying to make.
I would describe my current overall position somewhat differently - it's not that I think people can't be poorly calibrated in this direction, but I do think "intuition designed for Dunbar numbers therefore useless in the modern world" is way too big of a pendulum swing. My model here - Benquo was saying this in the original demon thread post and I agree - is that the social intuitions relevant to demon threads are mostly about what is or is not entering common knowledge, and my sense is that these intuitions actually scale reasonably well with the size of a social group.
Because while it's true that modern social groups can be much larger than Dunbar it's also correspondingly true that we have much better social technologies designed to produce common knowledge among these larger groups, such as highly visible blog posts, community events and speeches given at them, etc. People have a lot of experience with these social technologies and basically understand their effects on common knowledge, e.g. most people can basically keep track of what their local social norms are, what they are and aren't allowed to say and wear, etc. and are sensitive to changes in these things.
I agree that the weirder your goals are the more you need to think about other things in addition to social reality, but that by no means allows you to just forget about social reality.
A report of my own recent experience taking a conversation to private chat:
As proposed in this subthread, zulupineapple and I took to a private chat (IRC via my private webchat server) to continue our recent argument about utility functions.
We chatted continuously for just under 3 hours. We attempted to cover all or most of the points brought up in the LW thread where the disagreement occurred, and did indeed touch on most of them. Our conversation was fairly civil, and I did not experience it as heated or emotional at any point, nor did it seem like zulup...
This is sort of a reply to Said (who asked 'what if they don't want to take it private?'), as well as zulupineapple (who asked 'why would taking it private help?'). But my impression from the overall thread here is that I skipped some important prequesites for this post to make sense. It seemed useful to spell them out in a comment separate from the ongoing conversation.
The "take it private" suggestion is predicated an approach of "make sure the conversation is collaborative, rather than working at cross purposes...
If the goal is for conversations to be making epistemic progress, with the caveat that individual people have additional goals as well (such as obtaining or maintaining high status within their peer group), and Demon Threads “aren’t important” in the sense that they help neither of these goals, then it seems the solution would simply be better tricks participants in a discussion can use in order to notice when these are happening or likely to happen. But I think it’s pretty hard to actually measure how much status is up for grabs in a given conversation. ...
I notice the term "demon thread" feels a bit too loaded for me to actually use. It is also somewhat misleading given there is such a thing as a benign demon thread.
It's probably too late in the game to alter the term though.
Step 1. Make it easy and common to take a conversation private if someone is feeling annoyed/threatened/angry/etc
Do people actually agree to offers to do this, if they are not already personally acquainted with their interlocutor? My experience is that they do not.
How do you propose to deal with people refusing or ignoring offers to take a conversation private?
I would appreciate an elaboration or restatement of "ii. Avoid bundling normative claims with descriptive claims." - I felt like I was understanding what you were saying but then "My point was more like: Arguing on the internet about the relative status of things is not effective altruism" felt like a nonsequitur, so I suspect I was misunderstanding the entire section.
"Is it a demon thread?" is more about "there is some weird force compelling more and more people to argue, raising tensions" than it's about "the argument is counterproductive or going in circles"
I had assumed it was both. Frankly, I don't care at all about the first kind of demon threads. If you entered a demon thread where nothing is actually being discussed, and ended up wasting your time and energy, that's your own fault. On the other hand, when people with good intentions start discussing a meaningful topic...
This post is intended as a working example of how I think Demon Threads should be resolved. The gist of my suggestion is:
Qiaochu had a criticism of the Demon Thread article. I had said:
He responded:
I initially responded publicly. (I think the details are important in their own right, <linked here>, but aren't the main point of this post)
We still disagreed, and the nature of the disagreement hinged on past threads full of social drama. This was exactly the sort of thing I didn't want to discuss publicly on the internet. Yes, the details mattered, but public discussion would have lots of bystanders showing up with opinions about the object-level-details about the social drama itself.
In this case, Qiaochu and I were able to discuss it privately, which:
So I've written this up as a post instead of a comment. I haven't run this by Qiaochu yet (I think getting formal permission/endorsement adds an "significant trivial inconvenience" that might disrupt the process too much), but I expect him to endorse the following, and I'll update/clarify if I got anything wrong.
Things I learned
i. Mattering-ness is orthogonal to Demon-Thread-ness
The most important update on my part. Qiaochu provided a few examples where it felt right to call a thing a demon thread, which the thing-in-question mattered in some sense - either because the tribal affiliation and status mattered, or because the actual ideas getting discussed mattered.
"Is it a demon thread?" is more about "there is some weird force compelling more and more people to argue, raising tensions" than it's about "the argument is counterproductive or going in circles" (although I think the latter is common).
Since matter-ness isn't part of the central definition, I've removed it from the description at the beginning of the post.
ii. Avoid bundling normative claims with descriptive claims.
One reason I think Demon Threads (often) don't matter is a normative claim about what people should value, and it is unfair to bundle this with claims about what people should do given what they currently value, even if I think they're being silly.
Conflating descriptive and normative claims can be a useful (but deceptive) rhetorical trick, and part of the point of LessWrong is to avoid doing that so we can think clearly about things.
Empirically, people care about what groups and ideas have relative status among their peers.
My point was more like: Arguing on the internet about the relative status of things is not effective altruism. People care about things other than accomplishing the greatest good for the least effort. It is perhaps most of what most people care about. And that's fine.
I think this claim is still relevant, because people often seem to think the thing they're doing is helping a much larger amount than it is, as well as accomplishing different things than they think it is. (i.e. you think talking about the president is having an impact on national policy, but it's mostly having an impact on what opinions are acceptable to express in your local peer group).
I do think, in most social/political-drama-laden threads, if people took a step back and thought about it, they would either realize an internet debate wasn't the best way to accomplish their goals, or they'd realize their goals were different than they thought they were.
iii. Maybe something didn't matter before the demon thread, but after a giant explosion of arguments happens, it may matter a lot (at least to the people involved).
I cited an example where, in a local community, people started arguing about [internet drama from several years ago]. Prior to the argument, it hadn't mattered what your opinions about that particular political drama was. But suddenly, everyone knew what many prominent community-member's opinions were, and people disagreed strongly, and there was a risk that if the thread went the wrong way, having one set of opinions might no longer be okay.
Qiaochu and I agreed it would have been better if the argument never happened, and that the political drama wasn't objectively important. But he argued, once it had exploded, it became relevant to the people involved. So the pressure to add your 2 cents was real and important.
This seems true, but also feeds back into my central claim, which is that it's best to stop malignant demon threads before they begin.
Outstanding Disagreements
Often, when people are coming from very different intuitions, they can argue a lot about factual claims, and agree that each other made good points... and still go back to basically holding their original position.
This can be frustrating, but understandable: people have a lot of background experience that feeds into whether something makes sense. Explicit arguments often can't fully address that background experience.
While we agreed with many of each other's claims in principle, each claim of Qiaochu and I included lots of words like "usually" and "sometimes", that were doing a lot of work, and our respective takeaways rounded those words in directions closer to our original positions.
Qiaochu's current overall position as I understand it is:
(I originally interpret this to basically be arguing against the "We're adapted for Dunbar Number tribes, therefore our intuitions for the modern world are useless" hypothesis, which seemed confusing. In the comments below Qiaochu clarified among other things that although our society is bigger, so are our tools for broadcasting signals. See his comment for more clarity)
My current position is: