Sometimes when I talk to people about how to be a strong rationalist, I get the impression they are making a specific error.

The error looks like this: they think that good thinking is good thinking irrespective of environment. If they just learn to avoid rationalization and setting the bottom-line first, then they will have true beliefs about their environment, and if there's something that's true and well-evidenced, they will come to believe it in time.

Let me give an extreme example.

Consider what a thoughtful person today thinks of a place like the Soviet Union under Stalin. This was a nation with evil running through their streets. People were vanished in the night, whole communities starved to death, the information sources were controlled by the powerful, and many other horrendous things happened every day.

Consider what a strong rationalist would have been like in such a place, if they were to succeed at keeping sane. 

(In reality a strong rationalist would have found their ways out of such places, but let us assume they lived there and couldn't escape.) 

I think such a person would be deeply paranoid (at least Mad-Eye Moody level), understanding that the majority of their world was playing power games and trying to control them. They'd spend perhaps the majority of their cognition understanding the traps around them (e.g. what games they were being asked to play by their bosses, what sorts of comments their friends would report them for, etc) and trying to build some space with enough slack to occasionally think straight about the narratives they had to live out every day. It's kind of like living in The Truman Show, where everyone is living a narrative, and punishing you / disbelieving you when you deviate. (Except with much worse consequences than what happened in that show.)

Perhaps this is too obvious to need elaborating on, but the cognition of a rationalist today who aims to come to true beliefs about the Soviet Union, and the cognition of a rationalist in the Soviet Union who aims to come to true beliefs about the Soviet Union, are not the same. They're massively different. The latter of them is operating in an environment where basically every force of power around you is trying to distort your beliefs on that particular topic – your friends, your coworkers, the news, the police, the government, the rest of the world.

(I mean, certainly there are still today many distortionary forces about that era. I'm sure the standard history books are altered in many ways, and for reasons novel to our era, but I think qualitatively there are some pretty big differences.)

No, coming to true beliefs about your current environment, especially if it is hostile, is very different from coming to true beliefs about many other subjects like mathematics or physics. Being in the environment can be especially toxic, depending on the properties of that environment and what relationship you have to it.

By analogy, I sometimes feel like the person I'm talking to thinks that they just practice enough fermi estimates and calibration training and notice rationalization in themselves and practice the principle of charity, then they'll probably have a pretty good understanding of the environment they live in and be able to take positive, directed action in it, even if they don't think carefully about the political forces acting upon them.

And man, that feels kinda naive to me.

Here's a related claim: you cannot get true beliefs about what are good actions to take in your environment without good accounting, and good record-keeping. 

Suppose you're in a company that has an accounting department that tells you who is spending money and how. This is great, you can reward/punish people for things like being more/less cost-effective. 

But suppose you understand one of the accounting people is undercounting the expenses of their spouse in the company. Okay, you need to track that. (Assume you can't fire them for political reasons.) Suppose another person is randomly miscounting expenses depending on which country the money is being spent. Okay, you need to track that. Suppose some people are filing personal expenses as money they spent supporting the client. Okay, now you need to distrust certain people's reports more-so.

At some point, to have accurate beliefs here, it is again not sufficient to avoid rationalization and be charitable and be calibrated. You need to build a whole accounting system for yourself to track reality.

[A]s each sheep passes out of the enclosure, I drop a pebble into a bucket nailed up next to the door. In the afternoon, as each returning sheep passes by, I take one pebble out of the bucket. When there are no pebbles left in the bucket, I can stop searching and turn in for the night. It is a brilliant notion. It will revolutionize shepherding.

—The Simple Truth

I sometimes see quite thoughtful and broadly moral people interact with systems I know to have many power games going internally. Moral Mazes, to some extent or another. The system outputs arguments and trades, and the person sometimes engages with the arguments and sometimes engages in the trade, and thinks things are going well. But I feel like, if they knew the true internal accounting mechanisms in that entity, then they would be notably more disgusted with the parts of that system they interacted with. 

(Imagine someone reading a scientific paper on priming, and seeking deep wisdom in how science works from the paper, and then reading about the way science rewards replications.)

Again, I occasionally talk to such a person, and they can't "see" anything wrong with the system, and if they introspect they don't find a trace of any rationalization local to the situation. And if they've practiced their calibration and fermis and charity, they think they've probably come to true beliefs and should expect that their behavior was net positive for the world. And yet there are instances I feel that it clearly wasn't.

Sometimes I try to tell the people what I can see, and that doesn't always go well. I'm not sure why. Sometimes they have a low prior on that level of terrible accounting, so don't believe me (slash think it's more likely that I'm attempting to deceive them). This is the overly-naive mistake.

More often I think they're just not that interested in building that detailed of a personal accounting system for the thing they're only engaging with some of the time and isn't hurting them very much. It's more work than it's worth to them, so they get kind of tired of talking about it. They'd rather believe the things around them are pretty good rather than kinda evil. Evil means accounting, and accounting is boooring. This is the apathetic mistake.

Anyway. All this is me trying to point to an assumption that I suspect some people make, an assumption I call "Context-Free Integrity", where someone believes they can interact with complex systems, and as long as they themselves are good and pure, their results will be good and pure. But I think it's required that you to actually build your own models of the internals of the complex systems before you can assess this claim.

...writing that down, I notice it's too strong. Eliezer recommends empirical tests, and I think you can get a broad overall sense of the morality of a system with much less cost than something like "build a full-scale replica accounting model of the system in google sheets". You can run simple checks to see what sorts of morality the people in the system have (do they lie often? do they silence attempts to punish people for bad things? do they systematically produce arguments that the system is good, rather than trying to simply understand the system?) and also just look at its direct effects in the world.

(In my mind, Zvi Mowshowitz is the standard-bearer on 'noping out' of a bad system as soon as you can tell it's bad. The first time was with Facebook, where he came to realize what was evil about it way in advance of me.)

Though of course, the more of a maze the system is, the more it will actively obscure a lot of these checks, which itself should be noted and listed as a major warning. Just as many scientific papers will not give you their data, only their conclusions, many moral mazes will not let you see their results, or tell you metrics that are confusing and clearly goodharted (again on science, see citation count).

I haven't managed to fully explain the title of this post, but essentially I'm going to associate all the things I'm criticizing with the name "Context-Free Integrity". 

Context-Free Integrity (noun): The notion that you can have true beliefs about the systems in your environment you interact with, without building (sometimes fairly extensive) models of the distortionary forces within them.