The LW team is encouraging authors to review their own posts, so:
In retrospect, I think this post set out to do a small thing, and did it well. This isn't a grand concept or a vast inferential distance, it's just a reframe that I think is valuable for many people to try for themselves.
I still bring up this concept quite a lot when I'm trying to help people through their psychological troubles, and when I'm reflecting on my own.
I don't know whether the post belongs in the Best of 2018, but I'm proud of it.
For a data point, after having read the first paragraph I looked for examples and counterexamples. I found 1 example where a person strongly worries they are X while actually they’re anti-X (just like orthonormal describes), 2 examples where a person strongly worries they’re X and they’re indeed X, and 3 examples where they worry they’re X but actually they’re normal, i.e. between X and anti-X.
Interesting! Different experiences.
I do want to make it clear that people who are X often acknowledge that they are X, but don't intensely worry about it. E.g. a friend who knows he's abrasive, knows his life would be better if he were less abrasive on the margin, but doesn't have the emotional reaction "oh god, am I being abrasive?" in the middle of social interactions.
This was simple and probably helpful to think about, thanks!
(There's now some metaworry about the "What if I'm responding inappropriately to alarms!" alarm, but well, what can you do)
I have the seed of a post somewhere about trying to treat the symptoms of anxiety with counter anxiety. I think it's ultimately very costly.
What is "counter anxiety"? If your seed never makes it to a post, I'd like to at least have this clue.
Example: You get anxious about being unwanted, so you ask for friends for reassurance frequently. Eventually it occurs to you being asked to reassure all the time is annoying, so you develop anxiety around asking. This solves the immediate problem of annoying your friends, but the underlying anxiety is still there. So at best, this creates more suffering than actually resolving the problem. But there may actually be a worse thing going on, where you're reinforcing the use of anxiety as the mechanism to change your behavior, instead of learning new ones.
This behavior is also what you would expect to happen if different people had different preferences. The quiet friend might worry about being too loud, because they would actually prefer to be even quieter. You could say that they're projecting their own preferences on you, which is an error, but a slightly different one. And I admit that this doesn't fit all of your examples.
Another issue. Why is there "loudest" in the title? Why would this only happen with loud alarms? Surely, minor alarms can be warped through the same mechanisms, right?
It's because the non-broken alarms, which also start out loud, get quieter throughout your life as they calibrate themselves, and as one's habits fix the situations that make them correctly go off. So given a random initial distribution of loudness, eventually the alarm that's loudest on average will probably be a broken one.
There are two distinct questions. First, are there quiet broken alarms. I propose that there are and your theory should agree - the person starts off with a broken alarm, modifies their behavior to the extreme, and the alarm stops coming up. Your text seemed to imply that it can't stop, but I don't think I understood why not.
Second, are there loud correct alarms. It easy to imagine a case where there would be. For example, an alcoholic might worry that his drinking is hurting his family. Or an introvert could worry that they're too boring to hang out with. Your suggestion that habits are easy to change is weird.
This is related to simultaneous overconfidence and underconfidence. The loudest alarms are very probably too loud. The most needed alarms are very probably too quiet. I don't think removing the habits-easy-to-change assumption really impacts this. Yeah, there are cases when your worst problem will stick out like a sore thumb because it is very bad. But, if there are a large number of possible problems and there's some noise in alarm calibration, it's unlikely.
I agree there are broken alarms that are quiet (including those that are broken in the direction of failing to go off, which leads to a blind spot of obliviousness!), and that there are people stuck in situations where there is a correct loud alarm that happens most of the time.
I said that habits are easier to change than alarms, not that they're easy in an absolute sense.
I'd be quite curious about more concrete examples of systems where there is lots of pressure in *the wrong direction*, due to broken alarms. (Be they minds, organisations, or something else.) The OP hints at it with the consulting example, as does habryka in his nomination.
I strongly expect there to be interesting ones, but I have neither observed any nor spent much time looking.
This is anecdotally true in my practice, as well. Though the explanation seems simple enough, I would not frame it as alarms: if you are overly sensitive ("high-gain" in the control theory terms) to a specific stimulus, you are bound to overreact to that stimulus to try to match the others in perceived intensity. This is a simple negative feedback loop. Works almost universally.
However, humans are a very stuck control system by the time they are adults (unlearning is orders of magnitude is more difficult than learning), and deliberately changing the forward and feedback gain is very difficult.
I think this post is a clear explanation of an important point in system dynamics, connected to personal psychology (and thus individual rationality). It connects to other important concepts, like the tails coming apart and Goodhart, while being distinct (in a way that I think helps clarify them all).
Using this space to ask a question unrelated to this post itself: How does frontpage/featured work? I'm confused about it. When someone posts, they can choose 'their blog' or 'frontpage' to post? But mods can move from 'blog' to frontpage? And then things on frontpage can be featured? Do people ever want to post to just 'their blog', or is it generally assumed that a post should go on the frontpage if eligible?
Since this is a meta comment, seemed best to link to this meta thread discussing it: https://www.lesserwrong.com/posts/TWdmm3Q4gouF7EETW/12-31-17-update-frontpage-redesign.
(We are working on another update to make it a bit clearer)
Meta-meta: Should I have gotten any kind of indication or notification of this reply? I didn't.
Also-meta, but I'm going to reply here since you already saw my original question and I don't think the post you linked answers it: People have the option to post either to personal blog or frontpage, right? Even people like me with not that much karma? So under what circumstances would a mod move a post from personal blog to frontpage? Might an author not object to that? I'm trying to understand whether I have a good model of the purpose of personal blog versus frontpage, and of what moderators can do, and what they normally do.
I've responded in a new meta post (in general, best if Meta discussion that you'd like an admin response on to go in the meta section)
https://www.lesserwrong.com/posts/LGAMb2Q76K9zXoRPX/1-9-2018-update-frontpage-views
I mostly second Vaniver's nomination. I've also found this post really useful when thinking about LessWrong as an organization, and how my own preferences might often be actively pushing things in the wrong direction.
I've been familiar with this pattern for a while, but I didn't explicitly connect it with the statistical explanation. My analysis was, rather, that at some point you learn "I need to work hard to get X right" -- at this point X is actually an important problem for you. So, you get into the habit of worrying about X, finding ways to be better at X, noticing details relevant to X... At some point, you cross the threshhold where X is not so important to work on, and maybe even negative to push any further in that direction. But, you don't automatically notice this point. By default, you keep on thinking X is a problem for you. Moreover, you've developed habits of thinking associated with pushing further in the direction of X; probably the feedback cues you're looking at are making you goodhart on X, and you haven't learned to pick up on the cues that would help you notice you've gone too far, because those would not have been helpful earlier.
So, you keep pushing further and further on X.
Therefore, if your life feels like a constant struggle to be nice, or clean, or punctual, or truthful, etc, it is possible that you're actually already one of the most nice/clean/punctual/truthful/etc people there is and you ought to stop worrying about it and maybe even adjust your habits a bit in the other direction.
Hmm. My biggest alarm is that I don't have enough friends and am not social enough. But this is accurate; I have not one single intellectual friend of the sort I want, and people ignore my writing. I think the reason I don't act to rectify the situation is that I don't know how.
Okay, but what about other alarms, ones I don't notice so much? Well, I have an intense desire to be right (and not just appear to be right in the eyes of anyone else), and be knowledgeable... and so I frequent LW. I spend an unusual amount of time fretting over the accuracy and fairness of things I've posted earlier and often edit them days, weeks, or years afterward to provide more nuance and accuracy. I do this even at the risk of sounding boring and producing less writing, even though I think one needs a lot of writing to gain a following; might this interfere with my social life? Could be. Another alarm says I took a wrong turn to end up working on oil & gas software. I donate to clean energy efforts as a form of "offsetting", but this has the disturbing opportunity cost of reducing my donations to other efforts. That alarm seems reasonable and I'm open to different opportunities.
But, maybe this post has exactly the problem you linked to: offering good advice to those who least need it.
That sounds pretty rough.
This is harsh and may be completely off the mark, but I was trying to call attention especially to alarms where those close to you disagree. If friends and family agree that you're not social enough, then that's probably a true alarm that you're facing.
Epistemic Status: Simple point, supported by anecdotes and a straightforward model, not yet validated in any rigorous sense I know of, but IMO worth a quick reflection to see if it might be helpful to you.
A curious thing I've noticed: among the friends whose inner monologues I get to hear, the most self-sacrificing ones are frequently worried they are being too selfish, the loudest ones are constantly afraid they are not being heard, the most introverted ones are regularly terrified that they're claiming more than their share of the conversation, the most assertive ones are always suspicious they are being taken advantage of, and so on. It's not just that people are sometimes miscalibrated about themselves- it's as if the loudest alarm in their heads, the one which is apt to go off at any time, is pushing them in the exactly wrong direction from where they would flourish.
Why should this be? (I mean, presuming that this pattern is more than just noise and availability heuristic, which it could be, but let's follow it for a moment.)
It's exactly what we should expect to happen if (1) the human psyche has different "alarms" for different social fears, (2) these alarms are supposed to calibrate themselves to actual social observations but occasionally don't do so correctly, and (3) it's much easier to change one's habits than to change an alarm.
In this model, while growing up one's inner life has a lot of alarms going off at various intensities, and one scrambles to find actions that will calm the loudest ones. For many alarms, one learns habits that basically work, and it's only in exceptional situations that they will go off loudly in adulthood.
But if any of these alarms don't calibrate itself correctly to the signal, then they eventually become by far the loudest remaining ones, going off all the time, and one adjusts one's behavior as far as possible in the other direction in order to get some respite.
And so we get the paradox, of people who seem to be incredibly diligently following the exact wrong advice for themselves, analogous to this delightful quote (hat tip Siderea) about system dynamics in consulting:
The funny thing about cognitive blind spots (and that's what we're looking at here) is that you can get pretty far into reading an article like this, hopefully enjoying it along the way, and forget to ask yourself if the obvious application to your own case might be valid.
If so, no worries! I developed this idea an embarrassingly long time before I thought to ask myself what would be the constant alarm going off in my own head. (It was the alarm, "people aren't understanding you, you need to keep explaining", which was a huge epiphany to me but blindingly clear to anyone who knew me.)
And the framing that helped me instantly find that alarm was as follows:
What do I frequently fear is going wrong in social situations, despite my friends' reliable reassurance that it's not?
That fear is worth investigating as a possibly broken alarm.