When I was practicing a (relatively new - think 1970s) form of karate, I discovered that there was a near-religious fervor surrounding the art. While I did see a lot of competent martial artists at the higher levels, they continuously insisted on the infallibility of the kata, slow-speeds sparring, and "situationals" that made up the bulk of their practice. I was repeatedly informed that the art was self-defense oriented, but was rarely subjected to any realistic practice. They claimed that they removed a lot of their sparring early on because it incited competitiveness, and there were strict rules about questioning the senseis. I ended up leaving my local school for a number of reasons, the most relevant being a desire for more realistic instruction in self-defense.
I did see a lot of value in this particular martial art, but none of that was in its ability to foster combat skills. Sure, the members who had been there ten years and gotten their black belts were better fighters than your average guy on the street. But at the lower levels, the value lay more in the discipline and personal development aspects than anything else. I would have stayed longer, probably, if the other practitioners had just admitted that, rather than insisting on complete infallibility in combat. Their cult-like devotion drove me away faster than mere honesty would have.
Judo is said to quite definitely a sport rather than a street-fighting art. And the Shoshin Shotokan karate I did was aimed at self-discipline, fitness, and a clear mind on Zen principles. It may well be that an alert mind will help you avoid combat and maximize your chances better than pure MMA- type fighting ability. Or maybe not.
Related: The Martial Art of Rationality
One principle in the martial arts is that arts that are practiced with aliveness tend to be more effective.
"Aliveness" in this case refers to a set of training principles focused on simulating conditions in an actual fight as closely as possible in training. Rather than train techniques in a vacuum or against a compliant opponent, alive training focuses on training with movement, timing, and energy under conditions that approximate those where the techniques will actually be used.[1]
A good example of training that isn't alive would be methods that focused entirely on practicing kata and forms without making contact with other practitioners; a good example of training that is alive would be methods that focused on verifying the efficacy of techniques through full-contact engagement with other practitioners.
Aliveness tends to create an environment free from epistemic viciousness-- if your technique doesn't work, you'll know because you won't be able to use it against an opponent. Further, if your technique does work, you'll know that it works because you will have applied it against people trying to prevent you from doing so, and the added confidence will help you better apply that technique when you need it.
Evidence from martial arts competitions indicates that those who practice with aliveness are more effective than others. One of the chief reasons that Brazilian jiu-jitsu (BJJ) practitioners were so successful in early mixed martial arts tournaments was that BJJ-- a martial art that relies primarily on grappling and the use of submission holds and locks to defeat the opponent-- can be trained safely with almost complete aliveness, whereas many other martial arts cannot.[2]
Now, this is not to say that one should only attempt to practice martial arts under completely realistic conditions. For instance, no martial arts school that I am aware of randomly ambushes or attempts to mug its students on the streets outside of class in order to test how they would respond under truly realistic conditions.[3]
Even in the age of sword duels, people would train with blunt weapons and protective armor rather than sharp weapons and ordinary clothes. Would training with sharp weapons and ordinary clothes be more alive than training with blunt weapons and protective armor? Certainly, but the trainees wouldn't be! And yet training with blunt weapons is still useful-- the fact that training does not fully approximate realistic conditions does not intrinsically mean it is bad.
That being said, generally speaking martial arts training that is more alive-- that better approximates realistic fighting conditions-- is more effective within reasonable safety margins. There is a growing consensus among students of martial arts who are looking for effective self-defense techniques that the specific martial art one practices is not hugely relevant, and that what matters more is the extent to which the training does or doesn't use aliveness.
Aliveness and Rationality
So, that's all well and good-- but how can we apply these principles to rationality practice?
While martial arts training has very clear methods of measuring whether or not skills work (can I apply this technique against a resisting opponent?), rationality training is much murkier-- measuring rationality skills is a nontrivial problem.
Further, under normal circumstances the opponent that you are resisting when applying rationality techniques is your own brain, not an external enemy.[4] This makes applying appropriate levels of resistance in training difficult, because it's very easy to cheat yourself. The best method that I have found thus far is lucid dreaming, as forcing your dreaming brain to recognize its true state through the various hallucinations and constructed memories associated with dreaming is no easy task.
That being said, I make no claims to special or unique knowledge in this area. If anyone has suggestions for useful methods of "live" rationality practice, I'd love to hear them.
[1] For further explanation, see Matt Thornton's classic video "Why Aliveness?"
[2] If your plan is to choke someone until they fall unconscious, it is possible to safely train for this with nearly complete aliveness by wrestling against an opponent and simply releasing the chokehold before they actually fall unconscious. By contrast, it is much harder to safely train to punch someone into unconsciousness, and harder still to safely train to break people's necks.
[3] The game of Assassins does do this, but usually follows rules that are constrained enough to make it a suboptimal method of training.
[4] There are some contexts in which rationality techniques are applied in order to overcome an external enemy. Competitive games and some sports are a good method of finding practice in this respect. For instance, in order to be a competitive Magic: The Gathering player, you need to engage many epistemic and instrumental rationality skills. Competitive poker can offer similar development.