Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

The Martial Art of Rationality

35 Post author: Eliezer_Yudkowsky 22 November 2006 08:00PM

I often use the metaphor that rationality is the martial art of mind.  You don't need huge, bulging muscles to learn martial arts - there's a tendency toward more athletic people being more likely to learn martial arts, but that may be a matter of enjoyment as much as anything else.  Some human beings are faster or stronger than others; but a martial art does not train the variance between humans.  A martial art just trains your muscles - if you have the human-universal complex machinery of a hand, with tendons and muscles in the appropriate places, then you can learn to make a fist.

How can you train a variance?  What does it mean to train +2 standard deviations of muscle?  It's equally unclear what it would mean to train an IQ of 132.

But if you have a brain, with cortical and subcortical areas in the appropriate places, you might be able to learn to use it properly.  If you're a fast learner, you might learn faster - but the art of rationality isn't about that; it's about training brain machinery we all have in common.

Alas, our minds respond less readily to our will than our hands. Muscles are evolutionarily ancient subjects of neural control, while cognitive reflectivity is a comparatively more recent innovation.  We shouldn't be surprised that muscles are easier to use than brains.  But it is not wise to neglect the latter training because it is more difficult.  It is not by bigger muscles that the human species rose to prominence upon Earth.

If you live in an urban area, you probably don't need to walk very far to find a martial arts dojo.  Why aren't there dojos that teach rationality?  One reason, perhaps, is that it's harder to verify skill.  To rise a level in Tae Kwon Do, you might need to break a board of a certain width.  If you succeed, all the onlookers can see and applaud.  If you fail, your teacher can watch how you shape a fist, and check if you shape it correctly.  If not, the teacher holds out a hand and makes a fist correctly, so that you can observe how to do so. Within martial arts schools, techniques of muscle have been refined and elaborated over generations.  Techniques of rationality are harder to pass on, even to the most willing student.  It is also harder to give impressive public exhibitions of rationality.  This may partially explain why there are no rationality dojos as yet.

Very recently - in just the last few decades - the human species has acquired a great deal of new knowledge about human rationality.  The most salient example would be the heuristics and biases program in experimental psychology.  There is also the Bayesian systematization of probability theory and statistics; evolutionary psychology; social psychology.  Experimental investigations of empirical human psychology; and theoretical probability theory to interpret what our experiments tell us; and evolutionary theory to explain the conclusions.  These fields give us new focusing lenses through which to view the landscape of our own minds.  With their aid, we may be able to see more clearly the muscles of our brains, the fingers of thought as they move.  We have a shared vocabulary in which to describe problems and solutions.  Humanity may finally be ready to synthesize the martial art of mind: to refine, share, systematize, and pass on techniques of personal rationality.

Such understanding as I have of rationality, I acquired in the course of wrestling with the challenge of Artificial General Intelligence (an endeavor which, to actually succeed, would require sufficient mastery of rationality to build a complete working rationalist out of toothpicks and rubber bands).  In most ways the AI problem is enormously more demanding than the personal art of rationality, but in some ways it is actually easier.  In the martial art of mind, we need to acquire the realtime procedural skill of pulling the right levers at the right time on a large, pre-existing thinking machine whose innards are not end-user-modifiable.  Some of the machinery is optimized for evolutionary selection pressures that run directly counter to our declared goals in using it.  Deliberately we decide that we want to seek only the truth; but our brains have hardwired support for rationalizing falsehoods.  We can try to compensate for what we choose to regard as flaws of the machinery; but we can't actually rewire the neural circuitry.  Nor may a martial artist plate titanium over his bones - not today, at any rate.

Trying to synthesize a personal art of rationality, using the science of rationality, may prove awkward:  One imagines trying to invent a martial art using an abstract theory of physics, game theory, and human anatomy.  But humans are not reflectively blind; we do have a native instinct for introspection.  The inner eye is not sightless; but it sees blurrily, with systematic distortions.  We need, then, to apply the science to our intuitions, to use the abstract knowledge to correct our mental movements and augment our metacognitive skills.  We are not writing a computer program to make a string puppet execute martial arts forms; it is our own mental limbs that we must move.  Therefore we must connect theory to practice.  We must come to see what the science means, for ourselves, for our daily inner life.

And we must, above all, figure out how to communicate the skill; which may not be a matter for declarative statements alone.  Martial artists spar with each other, execute standard forms, and are watched throughout by their teachers.  Calculus students do homework, and check their answers.  Olympic runners continually try to beat their best previous time, as measured by a stopwatch.

How to communicate procedural skills of rationality, or measure them, is probably the single largest open issue that stands between humanity and rationality dojos - at least it's the part of the problem that most baffles me.  Meanwhile I lecture.  So does anyone out there have ideas?

Comments (38)

Sort By: Old
Comment author: Robin_Hanson2 22 November 2006 09:02:34PM 14 points [-]

To continue with this metaphor, it seems what we need is a good set of problems to test our rationality (i.e., ability to resist common biases). As with any cognitive test, the details of the test can't be known in advance, or people would just memorize answers instead of developing skills. So we need to collect a rather large set of good test questions, or create a generators to create sets of them automatically. Not easy, but perhaps worth doing.

Comment author: MichaelAnissimov 23 November 2006 01:14:29AM 2 points [-]

It seems that calibrating personal judgements using decision markets could in principle be a way to measure rationality. You make a guess, and if your guess is close to what the market predicts, you can be slightly more confident about yourself. Of course, for this to work, you'd have to avoid prior knowledge of the market, and skip participating in betting on topics that you want to consider unassisted first.

Great post btw.

Comment author: Pensans 23 November 2006 02:02:18AM -2 points [-]

Do we want "techniques" for finding the truth? I suppose if the truth is "the weight of this fruit for sale." But if the truth is more like "what I ought to be doing," there is something closer to the essence of personality involved. If my personality is gained as a technique at a truth dojo, then it does not feel like mine. The necessity of discovering certain truths personally rather through technical instruction is a limit on the usefulness of teaching.

Ultimately, the nature of rationality is a question that goes to essence of personality. If I am willing to receive only deductively derived formula as rational, then I have -- not only a technique of rationality -- but a particularly well-known and annoying personality. If I am willing to act on any emotional or intuitive vapor, then I have another well-known and annoying personality. We are all aware that the grounds upon which I am willing to believe and act are not just techniques of rationality -- they are part of our fundamental identities and determine the way that we form or do not form social relations with others.

Comment author: Robin_Hanson2 23 November 2006 02:07:34AM 0 points [-]

Michael, you can't have a prediction market without a way to pay off the bets, and if you have that you can measure personal accuracy directly, if you just wait.

Comment author: Eliezer_Yudkowsky 23 November 2006 03:51:14AM 0 points [-]

Robin, I agree that the main difficulty is figuring out how to pay off the bets, but it seems to me that - given such a measure - playing a prediction market around the measure makes the game more complex, and hopefully more of a lesson, and more socially involving and personally intriguing. In other words, it's the difference between "Guess whether it will rain tomorrow?" and "Bob is smiling evilly; are you willing to bet $50 that his probability estimate of 36.3% is too low?" Or to look at it another way, fewer people would play poker if the whole theme was just "Estimate the probability that you can fill an inside straight." I think Anissimov has a valid fun-amplifying suggestion here.

Comment author: Robin_Hanson2 23 November 2006 01:13:05PM 5 points [-]

Psychologists have developed a self-deception test, which includes questions like "Did you ever hate your mother," on the assumption that most everyone did but few want to admit it. See:

Paulhus, Delroy L. "Self-deception and Impression Management in Test Responses." In Angleitner, A. & Wiggins, J. S., Personality assessment Via Questionnaires. New York, NY: Springer, 1986, 143-165.

Perhaps questions like those might be a good part of a rationality test and practice regime .

Comment author: Joshua_Fox 23 November 2006 03:40:26PM 6 points [-]

> Why aren't there dojos that teach rationality?

They're called universities. My PhD adviser taught me to think rationally and caught many of the errors in my thinking, including those caused by biases. This eventually allowed me to catch many of my own errors.

No, it wasn't perfect. It was limited to one academic area and was not explicitly aimed at teaching rationality and overcoming biases. (Then again, martial-arts senseis do not teach their skills "explicitly", but by repetition and training.) Moreover, gaining subject-matter expertise took as much of the effort as learning rational thinking processes. But a PhD in a rigorous field from a good university is as close as you'll find to a "color belt" (dare I say "black belt"?) in rationality.

Comment author: Eliezer_Yudkowsky 23 November 2006 08:14:28PM 6 points [-]

Joshua, the thought had occurred to me, but with all due respect to universities, that's the same sort of training-in-passing that you get from reading "Surely You're Joking, Mr. Feynman" as a kid. It's not systematic, and it's not grounded in the recent advances in cognitive psychology or probability theory. If we continue with the muscle metaphor, then I would say that - if judged by the sole criterion of improving personal skills of rationality - then studying physics is the equivalent of playing tennis. If you actually want to do physics, of course, that's a whole separate issue. But if you want to study rationality, you really have to study rationality; just as if you wanted to study physics it wouldn't do to go off and study psychology instead.

Comment author: Steve_Sailer 24 November 2006 07:48:48PM 9 points [-]

I think you are accepting the advertising hooey of martial arts dojos too much at face value. Most people don't take martial arts for all the abstracted reasons that appear in the brochures, but because they want to feel they can win fights with other people.

Similarly, people don't want to learn how to be rational for the sake of being rational. Instead, you have to sell rationality for more human ends, such as being able to win arguments, or make money, or understand a particular field such as baseball statistics. You can learn a lot of general lessons about rationality from reading Bill James on baseball statistics, but it's not very exciting to study rationality for the sake of being rational.

Comment author: johnlawrenceaspden 09 October 2012 08:01:44PM 8 points [-]

I did judo as a child. At first it was because my parents thought I'd develop physical self-confidence that way. It worked (and judo's something I'd recommend to any boy), but it doesn't take much training before you can win fights against most people. After that you stay interested because you want to beat other people at judo, just like any other sport.

So you might sell your rationality dojos as being about winning arguments, but pretty soon I imagine that the practitioners might get more interested in being right.

Most people will get very competitive the minute they've got something they can measure. Is there any form of transport which isn't raced? And most non-competitive people will get very competitive the moment they're in a fight they can win.

Suddenly I'm imagining a room where the sensei is describing the Amanda Knox case, and the students are asking him questions and debating with each other, and after all the talking is done, people place and take bets on the result at various odds, and then the sensei reveals the actual answer and the correct reasoning. And ranking points are transferred accordingly, and there is a ladder on the wall where names are listed in order with the scores.

I already reckon I could seriously enjoy such a game. Who wants to play?

Comment author: [deleted] 09 October 2012 08:04:57PM 1 point [-]

I'm in as long as we don't discuss the Amanda Knox case.

Comment author: johnlawrenceaspden 10 October 2012 07:18:19PM 18 points [-]

I mentioned it only because it seems to have been a unique triumph for Less Wrong. I'd read about the case, and thought nothing of it particularly. And then people here started saying "Look at it from a probabilistic point of view", and so I did, and after a few hours head-scratching and diagram-drawing I realized that it was almost certainly a miscarriage.

I mentioned this to a few people I know, and they reacted pretty well as you'd expect to a middle-aged man suddenly getting a bee in his bonnet about a high-profile sex murder case involving pretty girls.

When she was eventually acquitted, various people said "How did you do that?". And the mathematically minded types were quite impressed with the answer, while the muggles think I've got some sort of incomprehensible maths-witchcraft thing that I can do to find out the truth.

Which is exactly the sort of thing you might want to sell, if you can find a way to teach it.

Comment author: Eliezer_Yudkowsky 10 October 2012 07:21:35PM 15 points [-]

Er, you have some sort of incomprehensible maths-witchcraft thing that you can do to find out the truth.

Comment author: [deleted] 10 October 2012 07:29:33PM 1 point [-]

Fair enough.

Comment author: shminux 10 October 2012 10:38:32PM 1 point [-]

I mentioned it only because it seems to have been a unique triumph for Less Wrong.

Why has it been so unique? Surely there are plenty of high-profile predictions one can make using the same Bayesian techniques? (Or one can simply ask gwern, who is apparently well calibrated after a thousand or so recorded predictions.)

Comment author: somervta 11 October 2012 09:28:07PM -1 points [-]

The key here was in applying Bayes, not in being especially calibrated.

Comment author: johnlawrenceaspden 12 October 2012 05:49:53PM 2 points [-]

Well actually I was just wondering about that.

What other claims like 'Amanda Knox is innocent' can we make, in the sense that (a) they're counter common thinking (b) we're pretty sure we're right (c) there's likely to be a resolution in our favour soon?

The Amanda Knox thing was a surprising prediction that came true. More of those would be neat.

Comment author: Morendil 10 October 2012 08:58:30PM 0 points [-]

cough GJP cough

Comment author: johnlawrenceaspden 11 October 2012 10:20:22AM 0 points [-]

GJP?

Comment author: Jayson_Virissimo 11 October 2012 10:23:00AM *  3 points [-]

GJP?

The Good Judgment Project is basically a prediction tournament. If The Sequences are like kata and PredictionBook is like randori, then prediction tournaments are like competitive Judo.

Comment author: johnlawrenceaspden 12 October 2012 05:50:34PM 0 points [-]

Thanks. Signed up for it.

Comment author: Morendil 11 October 2012 11:34:00AM 0 points [-]
Comment author: Paul_Gowder 24 November 2006 09:33:27PM 3 points [-]

Steve: Wasn't that the claim of the sophists? "We'll teach you how to win arguments so you can prevail in politics." The problem is that the skills for winning arguments aren't necessarily the schools for rationality in general. Probably the easiest way to learn the skills for winning arguments is to go to law school and "learn to think like a lawyer."

Comment author: pdf23ds 28 November 2006 02:18:56AM 4 points [-]

Steve, you're probably right, but I doubt that's really a big obstacle. Rationality has plenty of uses, and if anyone wants to teach it for money, they shouldn't have problems finding ways to promote it, in the long run. Besides, if people want to learn karate to beat people up, why don't dojos advertise based on that?

But I think that the analogy to martial arts is ill-advised for other reasons. Martial arts is a mostly anachronistic practice when evaluated against its original purpose. If you're really serious about self-defense nowadays, you get weapons. (Weapons have always been an advantage, but it's only been for a few hundred years now that they've become such a decisive advantage.)

Where are the tools for giving us a rational advantage? Sure, we have plenty of them for bringing knowledge to us quickly. The internet has been pretty successful at that. But what about other prostheses that specifically target our rational acuity?

Comment author: Dylan 22 July 2009 10:24:54PM 0 points [-]

Well,

I'll go with the martial arts metaphor, it has at least limited application, and is already the reference point for the discussion. As far as there being no rationality dojos, what about Buddhism, or any other mystic or contemplative path along religious lineages?

As far as measuring rationality or rational skills, the main problem is there is no generally agreed upon Truth or even agreement as to whether Truth even exists. This discussion is not about the actual content of truth per se but about the integrity of the approach to truth. This may be soundly based on the idea that we are only approaching knowledge of truth, but truth itself is still not known. So we cannot measure the soundness as a function of measurable results, since the results are either non-existent,unknown or not agreed upon.

It seems that if this is the case then any form of measurement cannot be objective or externally made, but must be subjective and internally assessed. So the question would become not what is true as fact, but what is true as honest. My vote would be for rational integrity as the context within which skill or understanding would be viewed.

Comment author: orthonormal 22 July 2009 10:44:21PM *  6 points [-]

Welcome to Less Wrong, Dylan! Check out the welcome thread and introduce yourself.

I agree that one can develop a pretty impressive Art based on destroying self-deception rather than on seeking truth, particularly when 'seeking truth' sounds like an exceptionally problematic phrase.

But it might not have to be that problematic. Read The Simple Truth if you want to know what Eliezer means by truth; it's much less naive than I'd expected when I was new around here.

Comment author: thomblake 11 May 2011 10:37:59PM 2 points [-]

Re-reading this brings to mind the history of the Logicians in ancient China. Apparently, they fell out of favor because all that their skills seemed to be good for was winning arguments, and others already had much more effective ways of winning arguments. Advertising rationality as a way to win arguments, as suggested by some of the original commenters, could cause rationality to fall into a similar trap.

I think this problem is avoided by your (later) idea of a Bayesian Conspiracy; if practitioners of beisudo were demonstrably better at the various instrumental tasks they undertook, then that might be enough advertising to attract people to rationality.

I also wonder if cargo-cult rationality would be good or bad. It seems like it could go either way.

Comment author: chatquitevoit 18 July 2011 04:08:46PM 0 points [-]

I wonder how much of a correlation there is between people who put effort into self-training in rationality (or communal training, a la Less Wrong) and those who actually train a martial art. And I don't mean in the "Now I'lll be able to beat up people Hoo-AH" three-week-course training - I mean real, long-term, long-rewards-curve training. I've done aikido on and off for years (my life's been too hectic to settle down to a single dojo, sadly), and it takes a similar sort of dedication, determination, and self-reflection as a serious foray into training your mind to rationality. And, I'd go so far as to say, a similar 'predilection of mind and preference' (and I'll let you LWers go to town on that one).

Comment author: Navanen 08 December 2012 04:52:16AM 0 points [-]

What are you meaning by correlation? Do you mean how similar the thinking of those who mindfully approach rationality training is to the thinking of people who seriously dedicate themselves to practicing a martial art? Or do you mean something else?

Comment author: AbyCodes 24 July 2011 11:23:11AM *  0 points [-]

Why aren't there dojos that teach rationality?

There is. Buddhist temples in the Himalayas (Bhutan and the neighbouring countries) and remote china. I lived in Bhutan as a child. All buddhist monks lived their lifes' in monasteries; meditating, contemplating, doing the daily chores, in introspection and following the teachings of Buddha. Isn't that a "dojo of rationality"?

How to communicate procedural skills of rationality, or measure them, is probably the single largest open issue that stands between humanity and rationality dojos - at least it's the part of the problem that most baffles me.

Discussion; as is with every other field.

Comment author: Arandur 13 August 2011 01:06:56AM -2 points [-]

Well, if we have a Bayesian Conspiracy in place, as it were, then I reckon the test would be derivation of hidden knowledge from available priors. This would require that the Conspiracy have access to answers reasoned out ahead of time, and also that the actual testing was kept secret. Perhaps an aspiring rationalist may only be tested once. If we have that strict a measure, though, we might consider allowing a certain amount of leniency... but then again, perhaps only those who strictly adhere to the Art deserve to move further into its ranks.

I am quite aware that my idea is based largely on a post you made about 16 months after this one; in that sense, I'm simply perpetuating a feedback loop, and may be acting in redundancy. But the more I think on this problem, the more convinced I am that it is a Good Idea: test, ceremony, Conspiracy, and all.

Comment author: witzvo 30 May 2012 03:03:26AM *  0 points [-]

Computer games, done properly, could be a good teacher of rationality. If you fail to observe the laws of nature that happen to apply in the game and act accordingly, you die / can't get to the goal. Also, debugging programs has been good for me: plenty of times I could not see an error in my algorithm, but knew there had to be one (modulo system bugs), and so I learned what does and does not constitute valid argument that way. Put these ideas together (games where you have to program things that work in the game universe) and it'd be interesting. Some AI competitions work that way, but we could probably build an iPad app for kids with the right ideas. For example, I'm not talking about teaching basic or some terrible "easy" language. Why not base the programming language of the game off some other Turing-complete system, like, say, cellular automata. The automata rules could change from time to time to keep them guessing. The game could be a sequence of problems: fill in these squares correctly in order to get the desired result and move to the next level. Actually, maybe Markov algorithms would be better. Or a version of the untyped lambda calculus (http://worrydream.com/AlligatorEggs/). Having a good baseline programming "world" is an important part of getting them to think about the right kinds of abstractions.

Comment author: RichardKennaway 30 May 2012 10:45:55AM 0 points [-]

Computer games, done properly, could be a good teacher of rationality. If you fail to observe the laws of nature that happen to apply in the game and act accordingly, you die / can't get to the goal.

The laws may carry their own message.

Space Invaders: However high you build your score, the alien horde (i.e. Death) crushes you in the end.

First Person Shooters: Killing your enemies is fun!

Tetris, Kingdom of Loathing, Evony, etc.: Clicky-clicky-clicky! Better than life!

Comment author: witzvo 01 June 2012 12:46:17AM *  0 points [-]

That's not quite what I had in mind, but, well, if we make our own game it can have it's own laws. For example, you gain power by understanding the rules of the game and acting accordingly, not just by reflexes or level grinding. If you understand the rules keenly, you act in subtly different ways that are vastly more effective, perhaps. Maybe you can collaborate to achieve more than you can individually, etc....

Here's a game that, at least, makes you figure out the game mechanics a bit to live: http://www.ludumdare.com/compo/ludum-dare-23/?action=preview&uid=7288 Variations on this requiring more clever tool building, say, would be interesting. But I think it is possible to build a game around a turing machine if we are clever about it.

Comment author: Desrtopa 01 June 2012 01:30:46AM 1 point [-]

We've had at least one discussion on video games as a rationality teaching tool in the past.

Comment author: witzvo 01 June 2012 03:31:10AM *  0 points [-]

Thanks. I also noticed Learn to code as related. I'm not sure I should continue here or there. I'll finish here with a link to Blocky maze which is a fairly vanilla example of making coding into a game using a simplified traditional visual programming language, rendered in javascript.