While reading primary science literature, I've had the following experiences happen to me on multiple occasions.
1) Read a paper with a surprising result. Later discover it has critical flaws or didn't pass replication. I've learned to increase skepticism with increasingly surprising results. "This study is just wrong because of statistical issues or bad reporting" is now always one of the hypotheses in my mental arsenal, and I've found myself getting a bit better at predicting which results are just wrong using largely the heuristic of "this is too surprising to believe"
2) Form a hypothesis while reading. It gets verified (or falsified) via something you read later. Also, since one typically reads the methods before the results, one gets a lot of practice predicting results. (I don't formally make predictions but I find myself making them automatically as I read.)
Based on these experiences, I suggest that reading primary scientific literature is a good exercise in "alive" epistemic rationality training. The only drawback is that it takes a long time to get sufficiently acquainted with a field.
While I don't read scientific literature that much, I do make formal predictions pretty often. Typically any time I notice something I'm interested in that will be easy to check in the future.
Will I get to bed on time today? Will I be early for the meeting tomorrow? Etc.
I second the anecdotal evidence that this is a "live" exercise. Sidenote: it took me way too long to realize I needed to write all my predictions down. I spent a few weeks thinking I was completely excellent at predicting things.
Here's a prior that served me well for reading empirical literature:
1.) There is no effect (the null is true).
2.) If there is an effect but the data is observational, be very careful of any causal claims (they are most likely either due to modeling issues, bias due to confounding they missed, or getting the causal analysis wrong, or [a thousand more things]).
3.) If there is an effect and it is causal, I probably already heard about it, and there are lots of papers establishing it. Give the publication rate, and my reading rate, the chances of me stumbling on a genuinely new empirical result being reported for the first time is quite low.
4.) Conditional on me reading a paper, it's either related to what I do, or the authors are "good at the media," or (very rarely) it's actually a breakthrough!
5.) Most papers are crap, most wrong findings are not retracted (incentives).
Reminds me a bit about Feynman's story about Brazilian physics education, where the students were developing no idea how to apply the theoretical content they learned into actual application. I've also read a similar bit about French physics students, who are very good with formal methods, but tend to get stumped when they are given an open-ended tricky physics problem and need to figure the appropriate method to use themselves.
Also, the Street Fighting Mathematics book and course uses the practical martial arts metaphor.
Then there's the whole mess of teaching people software engineering, that seems to fail on both teaching theory (since we don't really have good theory for how to make software yet) and no aliveness training, since it's being taught at universities where the organization assumes that you have a theoretical subject you can lecture at people, not something where you'd need to basically do apprentice training to learn it properly.
It's apparently not just software engineering either:
We know that there are profound problems with university science teaching via traditional lectures and we are making some discoveries about how to improve it. For example, Nobel-winning physicist Carl Wieman decided to test how successfully he was teaching science. To his shock he discovered that the vast majority of his students did not understand what he was talking about. His graduate students would arrive in his lab but after 17 years of success ‘when they were given research projects to work on, they were clueless about how to proceed’ and ‘often it seemed they didn’t even really understand what physics was.’ However, after a few years in his lab they would be transformed into expert physicists. This was ‘a consistent pattern’ over many years.
From the Some notes on education paper.
Why do so many top martial arts, as for example Shotokan Karate, focus so strongly on kata, pre-set sparring exercises, and a little punches-pulled sparring?
The senseis are totally focused on their art, and have thought long and hard, and trained for decades, to make it as good as possible.
Possibly it is because they have goals for the practitioners beyond winning a MMA fight, like personal development, but I'm not sure that is the answer.
I can identify three fairly significant issues at fault here, all stemming from the fact that the original senseis actually fought and most of their successors don't:
Many martial arts-- despite originally being optimized for keeping you alive in combat-- have since been optimized for being accessible and easy to learn, in some cases for preteens. Accessibility and effectiveness are in many cases at cross purposes.
Many martial arts, without actual combat to point to as a metric of success or failure, have instituted systems of "point sparring" and so on in order to run tournaments. Thanks to Goodhart's Law, many of these martial arts have then become diluted by techniques and training optimized for scoring points in tournaments rather than defending yourself.
Many techniques that were once very practical are no longer so, and out of respect for tradition or simple lack of constant reevaluation are still being taught.
For instance, many traditional martial arts focus a lot on defending yourself against wrist grabs. I am told that this is because hundreds of years ago in Japan, wrist grabs were actually a common means of attack and important to know how to protect yourself against-- if someone walked up to you and grabbed your wrist, you couldn't draw your sword, which often meant you were about to die.
Nowadays, of course, people don't often initiate attacks by walking up and grabbing your wrist, so there is much less utility in such movements, but the curriculum of many traditional martial arts has not been updated to compensate.
When I was practicing a (relatively new - think 1970s) form of karate, I discovered that there was a near-religious fervor surrounding the art. While I did see a lot of competent martial artists at the higher levels, they continuously insisted on the infallibility of the kata, slow-speeds sparring, and "situationals" that made up the bulk of their practice. I was repeatedly informed that the art was self-defense oriented, but was rarely subjected to any realistic practice. They claimed that they removed a lot of their sparring early on because it incited competitiveness, and there were strict rules about questioning the senseis. I ended up leaving my local school for a number of reasons, the most relevant being a desire for more realistic instruction in self-defense.
I did see a lot of value in this particular martial art, but none of that was in its ability to foster combat skills. Sure, the members who had been there ten years and gotten their black belts were better fighters than your average guy on the street. But at the lower levels, the value lay more in the discipline and personal development aspects than anything else. I would have stayed longer, probably, if the other practitioners had just admitted that, rather than insisting on complete infallibility in combat. Their cult-like devotion drove me away faster than mere honesty would have.
Judo is said to quite definitely a sport rather than a street-fighting art. And the Shoshin Shotokan karate I did was aimed at self-discipline, fitness, and a clear mind on Zen principles. It may well be that an alert mind will help you avoid combat and maximize your chances better than pure MMA- type fighting ability. Or maybe not.
One thing that MMA has going for it as a competitive art is that it's well optimized for the specific type of fighting that takes place in an MMA ring. Some advocates will insist that the types of confrontations which take place in the ring are "real fighting," and anything which less effectively prepares one for the ring is not as effective in a "real fight," but it would be more accurate to say that MMA matches are a certain kind of fight, one which approximates only a certain subset of encounters one might have outside the ring.
For example, if one is hired as a bouncer at a club, it's certainly probable that one will eventually be involved in "real fights," where a person is seriously trying to hurt you without any rules to hold them back. But they'll overwhelmingly be untrained people with a lot of alcohol in them, who, at the very least for the sake of your job if not out of ethical concern, you should try to avoid injuring too badly. MMA training would be poorly optimized for these sort of encounters, as it's stripped of a lot of pain compliance techniques which would be handy in such a situation, and emphasizes many techniques which could result in your getting fired or possibly being sued or serving jail time.
If you're mugged, you'd be facing a violent encounter where your assailant almost certainly has a weapon, there may be multiple assailants involved, and facing you in a fight is not their top priority. Generically speaking, the standard advice for these situations is "give them your money," in which case no sort of martial arts training would leave you more prepared than any other, but some schools will teach not just techniques, but tactics for defending oneself in such a situation. An MMA school is unlikely to offer students such preparation, since that's time they could be spending learning to better kick the ass of one guy in the ring.
A law enforcement officer, again, is likely to face "real fights" with people who may be high on drugs, wielding weapons, in large groups, or some combination of the above, where the objective is generally to restrain and arrest, or if their own lives are threatened, to respond with lethal force using deadly weapons.
And so on and so forth. One on one brawls of the sort that MMA training optimizes for actually form a fairly small and avoidable portion of all violent encounters, but it's hard to arrange realistic comparative tests between martial artists of ability in any other kind of fight.
And the Shoshin Shotokan karate I did was aimed at self-discipline, fitness, and a clear mind on Zen principles.
That sounds very suspicious. Are they sure they didn't try to invent a combat art, fail, and invent that excuse afterward for continuing?
The human species is full of folly, but no. Rowers didn't try to invent a way to navigate lakes, failed, and ended up in those paper-thin useless boats. Rowing is a sport, like judo, and Shoshin Shotokan karate is a self-improvement practice with the goals I mentioned, only touching on sport and street-fighting.
Practitioners do claim that their style is effective in real fighting, though they recognize that styles without the layer of traditional Japanese discipline, like Krav Maga, have their own advantages (and disadvantages).
But avoiding a street fight is the best way to win it. The inspirational stories told among Shoshin Shotokan karate practitioners about the lineage of masters, people like Gichin Funakoshi and Hirokazu Kanazawa, tell not how they bloodied a roomful of thugs, but how they scared thugs off with self-confidence, or at most, restrained them with a testicle squeeze.
As far as I know, karate originated as unarmed fighting techniques used because possession of weapons like swords by ordinary people was very much illegal. So the beginning was quite practical.
However Shotokan is a bit of a special case because its founder was of the opinion that competitions and contests were the wrong approach and the point of karate was, basically, self-improvement. I don't know whether the founder tried and failed (at effective combat) or just didn't try.
In the West there is considerable social pressure to restyle martial arts as a non-aggressive system of exercises which aim to teach discipline (in particular, obedience to authority), provide some strength and aerobic training, and make you look cool.
Most judo is quite definitely taught as a sport rather than a street-fighting art, yes, but this is not strictly a fact about the art of Judo. Such is Kodokan Judo, which is by far the most common form, focused almost entirely on Judo as a grappling sport (to the point where many judoka don't even know that atemi-waza (striking techniques) exist, despite the fact that they're in the kata).
it's certainly possible to teach Judo as street-fighting, but most people who want that will go for jiu-jitsu. Judo is (deliberately) 'tamer' than jui-jitsu, so as street-fighting, it's really best suited for limited self-defense. It was quite common in police work in Japan, not sure if that's still true.
(This makes it very useful in situations where your response to assault is legally limited, which IIRC is why it was popular for police. It is much less natural to respond with excessive amounts of force when using judo)
Judo's kind of an interesting case, in that most of what calls itself "judo" in the modern day comes from a single lineage of Japanese grappling that was designed as a competitive martial art. The scope of Japanese grappling arts is much broader, and includes several less competitive and more self-defense oriented styles that could accurately have been called judo, but the Kodokan lineage has come to so dominate the field that anything not related to it now uses a different name (usually some form of "jujutsu"), to avoid confusion.
Judo is said to quite definitely a sport rather than a street-fighting art.
Judo is explicitly a sport version of jiu-jitsu. Jiu-jitsu is a street-fighting art and judo was created from it expressly to make it a sport. The major distinction is that a lot of dangerous techniques (e.g. strangulation) are outright forbidden in judo.
I've done Judo for several years, and yes it is a sport and doesn't pretend to be otherwise. A very fun sport that helps keep me physically fit to be sure, but in a hypothetical fight situation my best tactic remains "run away screaming".
I don't have any experience with judo. But this particular branch of karate (kokondo) advertised itself as strictly defense-oriented.
Why do so many top martial arts, as for example Shotokan Karate, focus so strongly on kata, pre-set sparring exercises, and a little punches-pulled sparring?
The simple answer is that kata develop muscle memory and submerge proper responses below the consciousness level. In a high-level fight you don't have time to think what to do -- you have to react "on instinct" and your instinct comes from (among other things) doing katas. Pre-set sparring does the same -- develops muscle memory and makes proper reactions "instinctive".
And sparring is "punches-pulled" because concussions, internal bleeding, and broken bones are bad for keeping and attracting students :-)
One other thing is that martial arts are quite different. The major relevant distinction here is between external and internal martial arts. Crudely speaking, external arts (e.g. taekwondo) focus on fairly straightforward application of physical force. Internal arts (e.g. tai chi) usually use less obvious and more complicated whole-body techniques but also stress the general energy/health aspect. It is often said that internal-arts masters live longer than external-arts masters.
Oh, and suburban-mall martial arts studios are generally a bad joke.
The simple answer is that kata develop muscle memory and submerge proper responses below the consciousness level. [...] And sparring is "punches-pulled" because concussions, internal bleeding, and broken bones are bad for keeping and attracting students :-)
That's accurate, but I'd like to note that an earlier focus on sparring will drill in working responses just as well or better. The problem is that they're usually not the optimal responses, and it's harder to train out an existing response than it is to train one in from nothing.
There's a quick-versus-good tradeoff here. If you start unstructured sparring early, your students will be able to adapt to serious self-defense correspondingly earlier; military-derived or highly self-defense oriented styles, like sambo and krav maga, usually take this approach. On the other hand, if your early focus is on kata, set self-defense techniques, and proper form, then your students won't be ready for serious self-defense for much longer -- possibly not until the early dan ranks -- but in theory they'll be better fighters after serious study. This approach is more common to traditional systems, especially those predating the Second World War.
Which one's better depends on what your goals are.
There's a quick-versus-good tradeoff here.
Yes, very much so. This is another thing the correlates well with the external vs. internal martial arts difference. External arts usually pick quick and internal arts usually pick good.
In Tai Chi, for example, you start by doing the form v-e-r-y s-l-o-w-l-y and it may take people years to get to the point where they can do the form quickly and correctly.
What criteria are you using her for "top martial arts?"
If we're talking in terms of "widely practiced," there's a strong selection pressure favoring schools which don't impose activities on their students which are highly painful, frightening, or conducive to injury. In the absence of strong feedback mechanisms where if the training doesn't work, the students find out and leave, the most popular martial arts will be ones which don't demand much of their students.
That's not to say that forms only persist because they allow students to believe they're practicing martial arts without engaging in more demanding activities; even arts with strong feedback mechanisms generally include some non-alive exercises, such as bagwork or shadowboxing, or the striking practice used in kendo.
It would not necessarily be correct to assume that the manner in which traditional martial arts are practiced today strongly reflects the training methods used by the traditional masters; the forms might be the same, but the emphasis on different types of training has shifted.
Non-alive training methods may be preferable when the techniques can't be practiced in alive training without an unacceptably high rate of injury, or when one is practicing basic movements at high levels of repetition to ingrain them into muscle memory, and involving a partner to use as a target would, even if it helped refine the movements, impose a great deal of boredom on the partner.
That's not to say that forms only persist because they allow students to believe they're practicing martial arts without engaging in more demanding activities; even arts with strong feedback mechanisms generally include some non-alive exercises, such as bagwork or shadowboxing, or the striking practice used in kendo.
Some of these exercises are also oriented towards muscular conditioning rather than technique practice.
If we're talking in terms of "widely practiced," there's a strong selection pressure favoring schools which don't impose activities on their students which are highly painful, frightening, or conducive to injury. In the absence of strong feedback mechanisms where if the training doesn't work, the students find out and leave, the most popular martial arts will be ones which don't demand much of their students.
A common adaptation seems to be to decouple the more painful and dangerous parts of practice from the less. I've studied a couple of arts where the harder forms of practice were taught in separate classes, at different times and not required for advancement in rank.
In both cases these were labeled "sparring class", but that's a little misleading; it seems common for them to incorporate more intense and painful conditioning techniques, or forms of partnered practice that aren't strictly sparring.
I agree with this but wouldn't necessarily recommend it as a go-to training method. See also Bet Your Friends to Be More Right.
How is aliveness different from "make beliefs pay rent in anticipated experience"?
Aliveness is making beliefs pay rent in actual experience.
One could turn that around and say that the process of anticipation needs to pay its rent.
Aliveness just seems to be beliefs-paying-rent on a meta-level.
A belief in the invisible garage dragon pays no rent: You don't anticipate anything. A belief that you can disarm someone holding a gun on you pays rent: You anticipate that if someone aims a gun at you, you'll be fine. A belief paying rent is not in principle relevant to whether or not it's true, though it does open it up to falsification.
Aliveness is a training paradigm designed to help create useful skills and ensure your beliefs about what those skills can do are accurate; making beliefs pay rent in anticipated experience is a technique for ensuring that your beliefs are meaningful.
For example, your beliefs may be meaningful but false because in training, you did not have accurate feedback while developing the skills that you use to form beliefs. Making beliefs pay rent is an example of a skill (or in this case, a habit) that is used to avoid meaningless beliefs. When you are training a skill like, say, calibration, it is important to have tight, realistic feedback loops on both your beliefs and the anticipated experiences they predict. Without the former, you will be anticipating correctly according to your beliefs without properly demonstrating the other skill you are training.
It's not obvious to me (with my rationalist hat on) how a belief may be false but meaningful.
I'm sorry to have such terse replies, but it's as simple as that.
I could believe that my 'O' and 'P' keys have been swapped, and thus anticipate that pressing my 'O' key will result in a 'P' and vice versa. This belief is clearly meaningful-- it means something-- but it also happens to be false. It passes the making beliefs pay rent test-- but that isn't a test for whether or not something is true.
Um, ok, one of us is using extremely nonstandard terminology without realizing it, and I don't think it's me. Take an example: "the earth is the closest planet to the sun". This belief is obviously false, but seems perfectly meaningful. Would you describe it as false and meaningless?
a belief in a celestial teapot is meaningless because it doesn't matter whether it's false or not: you have no evidence and no reasonable way of gaining evidence. Claiming to believe in it is just a waste of cognitive resources. A belief in, for example, the door opening inward or outward is meaningful: It will directly impact the action you take when you come to the door. This is regardless of whether the belief is false, and you push a pull door or the belief is true and you correctly pull the pull door.
Related: The Martial Art of Rationality
One principle in the martial arts is that arts that are practiced with aliveness tend to be more effective.
"Aliveness" in this case refers to a set of training principles focused on simulating conditions in an actual fight as closely as possible in training. Rather than train techniques in a vacuum or against a compliant opponent, alive training focuses on training with movement, timing, and energy under conditions that approximate those where the techniques will actually be used.[1]
A good example of training that isn't alive would be methods that focused entirely on practicing kata and forms without making contact with other practitioners; a good example of training that is alive would be methods that focused on verifying the efficacy of techniques through full-contact engagement with other practitioners.
Aliveness tends to create an environment free from epistemic viciousness-- if your technique doesn't work, you'll know because you won't be able to use it against an opponent. Further, if your technique does work, you'll know that it works because you will have applied it against people trying to prevent you from doing so, and the added confidence will help you better apply that technique when you need it.
Evidence from martial arts competitions indicates that those who practice with aliveness are more effective than others. One of the chief reasons that Brazilian jiu-jitsu (BJJ) practitioners were so successful in early mixed martial arts tournaments was that BJJ-- a martial art that relies primarily on grappling and the use of submission holds and locks to defeat the opponent-- can be trained safely with almost complete aliveness, whereas many other martial arts cannot.[2]
Now, this is not to say that one should only attempt to practice martial arts under completely realistic conditions. For instance, no martial arts school that I am aware of randomly ambushes or attempts to mug its students on the streets outside of class in order to test how they would respond under truly realistic conditions.[3]
Even in the age of sword duels, people would train with blunt weapons and protective armor rather than sharp weapons and ordinary clothes. Would training with sharp weapons and ordinary clothes be more alive than training with blunt weapons and protective armor? Certainly, but the trainees wouldn't be! And yet training with blunt weapons is still useful-- the fact that training does not fully approximate realistic conditions does not intrinsically mean it is bad.
That being said, generally speaking martial arts training that is more alive-- that better approximates realistic fighting conditions-- is more effective within reasonable safety margins. There is a growing consensus among students of martial arts who are looking for effective self-defense techniques that the specific martial art one practices is not hugely relevant, and that what matters more is the extent to which the training does or doesn't use aliveness.
Aliveness and Rationality
So, that's all well and good-- but how can we apply these principles to rationality practice?
While martial arts training has very clear methods of measuring whether or not skills work (can I apply this technique against a resisting opponent?), rationality training is much murkier-- measuring rationality skills is a nontrivial problem.
Further, under normal circumstances the opponent that you are resisting when applying rationality techniques is your own brain, not an external enemy.[4] This makes applying appropriate levels of resistance in training difficult, because it's very easy to cheat yourself. The best method that I have found thus far is lucid dreaming, as forcing your dreaming brain to recognize its true state through the various hallucinations and constructed memories associated with dreaming is no easy task.
That being said, I make no claims to special or unique knowledge in this area. If anyone has suggestions for useful methods of "live" rationality practice, I'd love to hear them.
[1] For further explanation, see Matt Thornton's classic video "Why Aliveness?"
[2] If your plan is to choke someone until they fall unconscious, it is possible to safely train for this with nearly complete aliveness by wrestling against an opponent and simply releasing the chokehold before they actually fall unconscious. By contrast, it is much harder to safely train to punch someone into unconsciousness, and harder still to safely train to break people's necks.
[3] The game of Assassins does do this, but usually follows rules that are constrained enough to make it a suboptimal method of training.
[4] There are some contexts in which rationality techniques are applied in order to overcome an external enemy. Competitive games and some sports are a good method of finding practice in this respect. For instance, in order to be a competitive Magic: The Gathering player, you need to engage many epistemic and instrumental rationality skills. Competitive poker can offer similar development.