Motivated reasoning is a natural byproduct of any mind that tries to do anything to the outside world.

Consider an optimal temperature controller. It has thermometers and runs a Kalman filter to calculate the probability distribution of the temperature at each moment taking into account the model of the process and all the data available. What will be the expected value of the temperature?

The set point. Always, so long as the output isn't saturated. Because if the expected temperature were any lower than the setpoint it'd increase the heat until it isn't. If it were any higher, it would decrease the heat until it isn't. The temperature controller fundamentally works by expecting the temperature to be what it "wants", and then acting to maintain that expectation.

This is unavoidable, because if it's not acting to keep the expected value in line with the setpoint -- according to it's own model/data -- then it isn't functioning as a control system with respect to that setpoint, and will be better described as optimizing for something else.

When you analyze the system from the third person and think "Hm, p(it will achieve its goal of controlling the temp to 70f at all times) is low", then that's your hint to redesign the controller to stop expecting "70f" to be the temperature in the next timestep. Instead, program it to expect something that won't fail catastrophically (e.g. "The temperature will rise to 70f as fast as it safely can without burning out heating elements").

To bring it back to the human, it's not a "quirk of human biology" that your sportsball coach talks about how "You have to believe you can win!". It comes directly from the fact that you cant try to win without expecting to win, and that people want to win.

That doesn't mean you can't use more fault tolerant strategies like "score as many points as possible". The latter can be even more effective at winning, but that is a different plan and requires giving up on trying to win. Minds that can find these more fault tolerant plans no longer need to believe they'll win and therefore do less motivated cognition failures, so if you want to fail less due to motivated cognition then that's the way to do it. You'll still be expecting what you want to come true, just hopefully in a more realizable way.

Apologizing requires not being out of trouble, and you're trying to not be in trouble. Do you want to be in trouble, and face the consequences? If that's not appealing to you, of course you're going to try to not do it, and that involves expecting to not be in trouble. No wonder people come up with defensive justifications in those cases. When you want to face the music, because what you're drawn to is being a person of integrity, and "not being guilty" is something you recognize you cannot have, then you won't feel tempted.

The question that can tranform the former into the latter is "Can I get what I want? Can I be not guilty and stay out of trouble?". When you sit with that and "No, I can't" sinks in, the temptation to rationalize melts away.

Of course, that can be tricky to sit with too, because there are often temptations to flinch away from the question. There are reasons for that, and understanding them opens up paths to making it easier there too, but this comment has gotten long enough.
 

Not uncertainty, but rather high confidence of safety.

Phrased this way, it sounds all "spooky mysterious", in the way that needs a named theorem and explanation.

Phrased to be easily understood, due to the vast space and few tiny satellites, you are very unlikely to have a collision when blindly throwing darts at this dart board. 

I may be maximally ignorant about what tomorrows lottery numbers will be, but I can safely predict that I will not win.

You certainly should not become more confident that 2 satellites are safe, just because you added random noise to the measurements.  [...]However, giving each one a random push from its jets increases the actual variation in their paths, likely pushing them away from the previous point estimate of a collision, and thus does make them safer. 

And if you add random noise, you don't get more confident. Like, if you're cruising along on what appears to be a potential collision course, and then your sensor goes bad and starts giving noisy data, you don't get more confident that you're safe. You just get more scared more slowly, in the case that you're on track to collide.

If you never had any evidence that this unlikely event is about to take place, then of course you can't magically get to the answer without evidence. Maybe you're on a collision course and jinking would make you safer. More likely though, you're just on a "close call" course, and jinking could put you on an actual collision course.

As an outsider with special knowledge, you may be rooting for the satellite to make some random motions, or to not change anything. As an engineer working on a satellite with noisy instruments, you don't have the privilege of knowing the right answer ahead of time, and you will have no way of firing the rockets only when it's helpful. If you program your satellites to move randomly, all you will accomplish is wasted fuel.

Imagine if from the inside. You're handed a revolver and forced to play Russian roulette. Your wife saw the spin and knows whether it landed on a live round and is either praying that you spin again or praying that you don't. You don't know where the live round is, and can't see or hear her. Do you ask to spin the spin the cylinder again? Would it help anything if you did?

The take away is just that if you want to predict rare events, you need evidence. Rare events do happen, if rarely, and unless you collect the evidence you'll have no choice but to be surprised if it happens to you.

I think Trump's comments about Rob Reiner were not to remotely take his death seriously. 

Well of course. Can you honestly not see the difference?

LLMs give us a really useful tool for political conversations where "mind killing" is a real risk. We can now post the entire context to Claude and ask a neutral question like "Does it look like <username> is engaging with arguments that might change his mind, or defending against them?".

What do you predict Claude will say, and are these predictions validated by experiment?

Or rather, before running that check, what would the appropriate response be if one were to run such a check and not receive ego-syntonic feedback? If I run that check and Claude says "Jimmy's comments contain the following signs of cognitive dissonance:", what should I do with that?

These are all very different things -- both the attacks, and the responses to the attacks.

To get a real parallel to the Kirk situation, you'd have to have someone like Destiny getting publicly assassinated by someone who unironically claims "Destiny was spreading hate and can't be reasoned with" as justification, right wing media outlets getting caught 

cheering, and in that context someone like Ben Shapiro making jokes about how much blood gushed from Destiny's neck. Thankfully none of this has happened.

Failing to remember the senator shooting is a much different response to a much different situation. The analogy there would be failing to remember the more closely analogous case where the congressmen where shot by a left wing crazy (remember that one?).

The Paul Pelosi situation is different in a few important ways. The fact that Paul Pelosi is still alive and has recovered is pretty significant. So is the fact that the right wing response was to make fun of him for getting attacked by his gay lover, not to cheer.

If, instead of emphatically stating that he had no sympathy for the Trump supporter killed in the attempted assassination of Trump, Destiny had just made a joke that it was probably Trumps gay lover shooting at him, that would be a very different thing. And pretty funny, IMO.

The problem -- and I say this as a fan of Destiny, who has enough Destiny-like tendenciess that when I showed his videos to an ex, her response was "He's been hanging out with you too much" -- is that Destiny is losing track of the actual symmetries and how to break them. I laughed at his vampire joke (and was viscerally bothered by Kirk's killing), and he's right that you gotta tit when the other guy tats and saying "I'M ONLY TITTING IN RESPONSE TO YOUR TAT! HOW 'BOUT WE STOP!?" does break the symmetry.

But what he loses track of is that the other side also thinks they're titting in response to "the other guy started it", and the other side also thinks "BUT WE'RE OBJECTIVELY RIGHT THO!", and that when he's insisting that he has no sympathy for the Trump supporters who were killed, that isn't above the symmetry it's actually engaging in the thing (and escalation of the thing) that he wants to see less of (from the other side).

I'm glad that he's pushing back against the expectation of performative sympathy, and performative condemnation, in favor of (sometimes provocative) humor. I'd like to see the humor be more consistently genuine though, and for him to have some actual sympathy under there too instead of succumbing to the contempt attractor and stoking the flames.
 

Actually fighting back effectively can get you in big trouble, and often models many behaviors you don’t actually want. Whereas the techniques you would use against a real bully outside of school, that you’d want to use, don’t work.

In my experience, this isn't true because fighting back effectively stops escalation before it happens.

I wasn't bullied in school, but not for lack of attempts. When they threw a ball of paper at me, I'd throw it back. When they asked if I want to fight, I'd say "ok". This happened many times, and not once did I actually have to fight anyone and therefore never I got suspended. Even the much bigger kid a year ahead of me just didn't show up to the scheduled fight (thankfully).

Even as an adult, the same approach worked well the one time I was in a bizarre enough situation to need it in a literal sense. The only time I ever got in a real fight was when it was too scary to say "ok" and I tried to run away instead.

To give a more "adult" example, if a neighbor in your apartment complex starts yelling at you for "messing with" the jacuzzi heater, the right posture isn't to yell back, it's to just to tell him that you were fixing it, then turn your back on him and go back to enjoying your jacuzzi session until he realizes he's being a dick and apologizes. It's absolutely a useful skill that shows up in places that are worth being in.

That doesn't mean "fight back" won't get you in trouble, just that you have to make sure you're doing the right thing. The posture isn't "Fuck me? No fuck you!", or "I'm not gonna take your shit anymore!", it's "ok". The former can get you in a lot of trouble -- both with bullies and with the school, because it's actively escalating. The latter deflates attempts to bully really quickly because there's just no place to put them.

In hindsight, the one time I failed to avoid a fight it's because I didn't commit to either "ok" or running. I chose to run, and when they gave chase and started kicking at us I decided "I guess I'm gonna have to fight back" -- which they weren't expecting but by then they didn't have a graceful exit. Expectation management.

So yes, "just fight back!" is naive and dumb. And "opportunities to learn to stand up for yourself" are of no use if you don't manage to learn these things. At some point you just gotta pull your kid.

At the same time, there's something real there too. There is an opportunity, if you manage to rise to it. And it applies to adult life as well in far more subtle ways.

The point is that "maintaining sanity" is a (much) higher bar than "Don't flail around like a drama queen". Maintaining sanity requires you to actually update on the situation you find yourself in, and continue to behave in ways that make sense given the reality as it looks after having updated on all the information available. Not matching obvious tropes of people losing their mind is a start, but it is no safe defense. Especially since not all repeated/noticeable failure modes are active and dramatic, and not all show up in fiction.

For example, if there's something to David Gross's comment that the wretched journalist was actually giving you an opening because they saw importance in what you had to say about the situation, blowing off a genuine opening to influence the discourse on AI safety while calling it "doing nothing" would not be sane. Preemptive contempt has a purpose in bounded rationality, but it's still a form of pushing away from the information the journalist has to offer. It can make sense within a grand plan that weights this journalist low, but that requires a grand plan.

How do you actually orient to the world, now that we are what we are? Are you still working to bring about the good outcome? If so, what's the grand plan that ties everything together? Sharing that seems important for helping people retain sanity. Have you given up? If so, what is the overarching plan that drives how you choose to interact with the world? Because you still have to decide what to do with your time.

This is a hell of a problem to orient to, and I don't know that any of us get to say we're doing it sanely. It's a high bar to strive towards.

The trope that this post and comment match to me isn't one that shows up in science fiction. It's a real bitch to wrestle free from, because the whole premise has to do with protecting stability of sense making by pushing away from challenging updates with avoidance and contempt, and the whole project fails if it doesn't turn meta and resist awareness of the trope. I notice that even writing and rewriting this comment to be minimally threatening of stability without holding back content, it's going to be a tough one to engage with to the extent that there isn't a preexisting superstructure regulating contact with reality to maintain stability while minimizing the cost of missed updates.

Which is certainly a possibility. As is leveraging the skill of becoming genre savvy as new patterns emerge ("trope dodging").

So if this contempt provokes contempt quickly, I'm sorry. My best isn't always good enough, which is kinda the possibility we're all wrestling with here.

Any thoughts on what to do if "just explain it to someone" turns into a long back and forth dialog?

The distinction between what might be called "lying" and "bullshitting" is important here, because they scale with competence differently.

It was pretty interesting watching this develop in my kids. Saying "No!" to "Did you take the cookie from the cookie jar?" is the first thing you get, because it doesn't require a concept for "truth" at all. Which utterance postpones trouble? Those are the sounds I shall make!

Yet for a while my wife and I were in a situation where we could just ask our younger kid about her fight with our older kid, because the younger kid did not have a concept for fabricating a story in order to mislead. She was developed enough to say "No!" to things she knew she did, but not developed enough to form the intention of misleading.

The impression I get from LLM is that they're bullshitters. Predict what text comes next. Reward the one's that sound good to some dumb human, and what's gonna come out? We don't need to postulate an intent to mislead, we just need to notice that there is no robust intent to maintain honest calibration -- which is hard. Much harder than "output text that sounds like knowing the answer".

It takes "growing up" to not default to bullshit out of incompetence. Whether we teach them they'll be rewarded by developing skillful and honest calibration, or by intentionally blowing smoke up our ass is another question.

I like this post because it takes things you can only learn by "actually doing things", and then presents them in ways that can be generalized.

My above description is false, actually. I've been saying that you are trying to hit the limit without going over. Actually, fast drivers hover at the limit. They oscillate between a little bit under and a little bit over. [...]

They find the limit by probing for it, dancing at it.

This part in particular, because the default assumption is "Oh no, can't cross the limit!", yet this is true about a lot of things.

Also, even if you're just driving to visit your grandma and not pushing the limits of traction, a traction aware driver will drive differently than your average driver. For example, it's quite common to approach a red light at their current driving speed, only to start braking harder and harder at the end. Which is a foolish use of the safety margin, and also slower than the person who brakes gently and early, and therefore is more likely to still have momentum when the light turns green.

The problem with talking about things is that we don't really have a good shared ontology of how "preferences"/"desires"/"values"/etc work, and they don't work the way people think they do.

Basically everything is way more context dependent than anyone realizes -- as in, "I only wanted to go to the store because I thought it had the food I wanted", to give a trivial example. But that food you had a preference for is subject to change as your bodies needs for nutrients changes. Even things like people's identities as "asexual" or "straight" are prone to update with the evidence we come across.

So then you try to say "Well, that's 'tastes', when I talk about 'values' I mean things like 'autonomy'". Except that kind of thing is merely instrumental as well -- stabilized by motivated blind spots about how useless autonomy can be in the right contexts. And then the right contexts come along, and your "values" shift. Which can sound like "Oh no! Value drift!" from the outside, but once you get there, it's just "Oh no, that store was closed. It's my recognition of that which has changed".

Then you try to retreat to "Okay, but pain is bad. Like, by definition!". Except it isn't, because masochists. Which aren't even uncommon, with how many people like spicy food, and hard massages that "hurt in a good way".

The last step seems to be avoidance of suffering, saying "Ah, right, pain isn't suffering" but suffering is the definition of bad!. Except we choose that too! Suffering is what we choose in order to stave off the loss of hope. Often without realizing it, so we can get stuck with unproductive suffering which really is good to eliminate, but it's something we choose nonetheless. And becoming conscious of it can allow more deliberate choice between hopelessness and continued suffering.

The whole thing is hard to make sense of, so it's kinda "Of course people are going to use terms in unclear and conflicting ways". When you say people should talk about things like "Their own preferences", are you referring to their preference to go to the store, or to eat the food that they believe the store has for them? Or something upstream of that? When you talk about "normative values", what the heck is that, exactly? If it's "The thing that we should value", then what exactly is that 'should' being used to distract from? Do we have any shared and accurate idea of what this means, descriptively speaking?

I think we need more deliberate study of how human tastes/desires/wants/values/etc change or don't, before we're going to have smooth hiccup-free communication on the topic. I agree with you that these terms conflate things, but I don't think we have the option of not conflating things yet. So I'm nudging away from "Just use clear language and then everything will be clear" and towards "notice what your concepts might be hiding, and how much ambiguity is necessarily left".