A Call for Constant Vigilance

katydee

Related to: What Do We Mean By "Rationality?"

Rationality has many facets, both relatively simple and quite complex. As a result, it can often be hard to determine what aspects of rationality you should or shouldn't stress.

An extremely basic and abstract model of how rationality works might look a little something like this:

Collect evidence about your environment from various sources
Update your model of reality based on evidence collected (optimizing the updating process is more or less what we know as epistemic rationality)
Act in accordance with what your model of reality indicates is best for achieving your goals (optimizing the actions you take is more or less what we know as instrumental rationality)
Repeat continually forever

A lot of thought, both on LessWrong and within the academic literature on heuristics and biases, has gone into improving epistemic rationality, and while improving instrumental rationality was less of a focus at first, recently the community has been focusing more on it. On the other hand, improving your ability to collect evidence has been relatively neglected-- hence the (in-progress as of this writing) Situational Awareness sequence.

But most neglected of all has been the last step, "repeat continually forever." This sounds like a trivial instruction but is in fact highly important to emphasize. All your skills and training and techniques mean nothing if you don't use them, and unfortunately there are many reasons that you might not use your skills.

You might be offended, angry, hurt, or otherwise emotionally compromised. Similarly, you might be sleepy, inebriated, hungry, or otherwise physically compromised. You might be overconfident in your ability to handle a certain type of problem or situation, and hence not bother to think of other ways that might work better.^[1] You might simply not bother to apply your skills because you don't think they're necessary, missing out on potential gains that you don't see at a glance-- or maybe even don't know exist. All in all, there are many times in which you may be missing out on the benefits that your skills can provide.

It may therefore be worthwhile to occasionally check whether or not you are actually applying your skills. Further, try to make this sort of check a habit, especially when encountering circumstances where people would typically be less than rational. If you find that you aren't using your skills as often as you'd expect, that may be cause for alarm, and at the very least is cause for introspection. After all, if rationality skills can be constantly applied to being successful in everyday life, we should be constantly on the watch for opportunities to apply them, as well as for potential lapses in our vigilance.

I indeed suspect that most LessWrong users would benefit more from being more vigilant in practicing and applying basic rationality skills than they would from learning cool advanced techniques.This principle is generally true in the martial arts, and both the inside and outside view strongly suggest to me that it is true for the art of rationality as well.

All in all, improving your rationality is a matter of serious practice and changing your mindset, not just learning cool new life hacks-- so next time you think about improving your rationality, don't look for new tricks, but new ways to truly integrate the principles you are already familiar with.

[1] The footnote to Humans Are Not Automatically Strategic describes several examples where this might apply.

Related to: What Do We Mean By "Rationality?"

Rationality has many facets, both relatively simple and quite complex. As a result, it can often be hard to determine what aspects of rationality you should or shouldn't stress.

An extremely basic and abstract model of how rationality works might look a little something like this:

Collect evidence about your environment from various sources
Update your model of reality based on evidence collected (optimizing the updating process is more or less what we know as epistemic rationality)
Act in accordance with what your model of reality indicates is best for achieving your goals (optimizing the actions you take is more or less what we know as instrumental rationality)
Repeat continually forever

[1] The footnote to Humans Are Not Automatically Strategic describes several examples where this might apply.

You might be offended, angry, hurt, or otherwise emotionally compromised. Similarly, you might be sleepy, inebriated, hungry, or otherwise physically compromised. You might be overconfident in your ability to handle a certain type of problem or situation, and hence not bother to think of other ways that might work better.[1]

This is in principle good advice, but I'd like to add a note of caution here - I feel that most "rationalists" actually follow it too closely, and end up losing (and rationalists should win).

Evolutionary process have produced a brain which has different specialized modules for dealing with different situations, and the "purpose" of these modules is more in line with instrumental rationality, not epistemic rationality. Consequentially, a good epistemic rationalist often must suppress the contribution of many of these modules (overconfidence, emotion, etc).

The instrumental rationalist, on the other hand, better play close attention to emotions and overconfidence. Don't forget Egan's law - given human cognitive limitations, someone who applies sound epistemic rationality to full effect is not going to behave too differently from the highly successful person next to them who does not care about epistemic rationalisty at all. In other words, subtracting within reasonable bounds the effects of luck and privilege, anyone who you'd gladly trade most aspects of your life with is a superior instrumental rationalist, regardless of intelligence or learning.

Although I do think overall, instrumental rationalism does improve when epistemic rationality improves, I think that some of the tensions between them have the unfortunate result of making strong epistemic rationalists err in systematic ways when it comes to instrumental rationality.

What does this mean practically? It means you have emotions for a reason. The parts of your brain which generate emotion are the ones which are calibrated for social behavior. If you feel yourself getting angry, it is likely that the behaviors which anger produces (confronting the aggressor) will in fact produce a positive result. Similarly, if you are sad, sad behaviors (crying, seeking support or temporarily withdrawing from the social scene, depending on the situation) will likely produce a positive result.

Same goes for cognitive biases. Fundamental attribution error produces positive results because it's better to assume that actions are innate to people rather than a result of random circumstances, since the latter don't hold any predictive value. The action resulting from overconfidence bias (risk taking) produces positive results as well. I can't even think of any biases that don't follow this pattern.

Behaviorally speaking, an instrumental rationalist should not correct a bias unless they have understood the reason the bias evolved and have adjusted the other variables accordingly. For example, if you are epistemically well calibrated in confidence, take care not to let that translate into instrumental underconfidence. I think the notion that the portions of your psyche which are useful when it comes to logic, reason, epistemic rationalisty, etc...will understand enough and react quickly enough to match the performance of systems which are specialized for this purpose is a bit misguided, and it is extremely important to let the appropriate systems guide behavior when comes to instrumental rationality.

Caveat - Of course, your brain is designed to make viable offspring in the ancestral environment. 1) The environment has changed and 2) your goal isn't necessarily to have offspring. But still - there is a good deal of overlap between the two utility functions.

subtracting within reasonable bounds the effects of luck and privilege

That sounds like an overwhelming exception to me.

0Viliam_Bur13y

If it increases the probability of winning like that highly successful irrational person, it's still worth doing. I mean, if an irrational person has a 20% chance of becoming highly successful, and a rationality training could increase it to 40%, then I would prefer to take that rationality training, even if the rewards for the "winners" in both categories are the same. But yes, we should remember that we use the human hardware, so we don't consistently overestimate the benefits of learning some rationality. Ideas which would work great for a self-improving AI may have less impressive impact on the sapient apes.