LESSWRONG
LW

Comment Permalink
Eliezer Yudkowsky00

I thought the difference between religious and scientific teachings was that scientific teachings didn't have to worry about dilution?

More of a difference between things that are hard vs. easy to teach and measure. Businesses have the same problem with a great CEO trying to hire great employees, dilution of corporate culture, etc. - they have highly quantifiable output at the end of the day, but in the middle of the day and the middle steps of the process, it's not as easy to measure.

I anticipate a beginning period extending for at least several years when we don't have good metrics because we're still trying to develop them.

Reply
See in context
by Eliezer Yudkowsky
4 min read
186

68

Previously in series:  Whining-Based Communities

"But there is a reason why many of my students have achieved great things; and by that I do not mean high rank in the Bayesian Conspiracy.  I expected much of them, and they came to expect much of themselves." —Jeffreyssai

Among the failure modes of martial arts dojos, I suspect, is that a sufficiently dedicated martial arts student, will dream of...

...becoming a teacher and having their own martial arts dojo someday.

To see what's wrong with this, imagine going to a class on literary criticism, falling in love with it, and dreaming of someday becoming a famous literary critic just like your professor, but never actually writing anything.  Writers tend to look down on literary critics' understanding of the art form itself, for just this reason.  (Orson Scott Card uses the analogy of a wine critic who listens to a wine-taster saying "This wine has a great bouquet", and goes off to tell their students "You've got to make sure your wine has a great bouquet".  When the student asks, "How?  Does it have anything to do with grapes?" the critic replies disdainfully, "That's for grape-growers!  I teach wine.")

Similarly, I propose, no student of rationality should study with the purpose of becoming a rationality instructor in turn.  You do that on Sundays, or full-time after you retire.

And to place a go stone blocking this failure mode, I propose a requirement that all rationality instructors must have secret identities.  They must have a life outside the Bayesian Conspiracy, which would be worthy of respect even if they were not rationality instructors.  And to enforce this, I suggest the rule:

  Rationality_Respect1(Instructor) = min(Rationality_Respect0(Instructor), Non_Rationality_Respect0(Instructor))

That is, you can't respect someone as a rationality instructor, more than you would respect them if they were not rationality instructors.

Some notes:

• This doesn't set Rationality_Respect1 equal to Non_Rationality_Respect0.  It establishes an upper bound.  This doesn't mean you can find random awesome people and expect them to be able to teach you.  Explicit, abstract, cross-domain understanding of rationality and the ability to teach it to others is, unfortunately, an additional discipline on top of domain-specific life success.  Newton was a Christian etcetera.  I'd rather hear what Laplace had to say about rationality—Laplace wasn't as famous as Newton, but Laplace was a great mathematician, physicist, and astronomer in his own right, and he was the one who said "I have no need of that hypothesis" (when Napoleon asked why Laplace's works on celestial mechanics did not mention God).  So I would respect Laplace as a rationality instructor well above Newton, by the min() function given above.

• We should be generous about what counts as a secret identity outside the Bayesian Conspiracy.  If it's something that outsiders do in fact see as impressive, then it's "outside" regardless of how much Bayesian content is in the job.  An experimental psychologist who writes good papers on heuristics and biases, a successful trader who uses Bayesian algorithms, a well-selling author of a general-audiences popular book on atheism—all of these have worthy secret identities.  None of this contradicts the spirit of being good at something besides rationality—no, not even the last, because writing books that sell is a further difficult skill!  At the same time, you don't want to be too lax and start respecting the instructor's ability to put up probability-theory equations on the blackboard—it has to be visibly outside the walls of the dojo and nothing that could be systematized within the Conspiracy as a token requirement.

• Apart from this, I shall not try to specify what exactly is worthy of respect.  A creative mind may have good reason to depart from any criterion I care to describe.  I'll just stick with the idea that "Nice rationality instructor" should be bounded above by "Nice secret identity".

• But if the Bayesian Conspiracy is ever to populate itself with instructors, this criterion should not be too strict.  A simple test to see whether you live inside an elite bubble is to ask yourself whether the percentage of PhD-bearers in your apparent world exceeds the 0.25% rate at which they are found in the general population.  Being a math professor at a small university who has published a few original proofs, or a successful day trader who retired after five years to become an organic farmer, or a serial entrepreneur who lived through three failed startups before going back to a more ordinary job as a senior programmer—that's nothing to sneeze at.  The vast majority of people go through their whole lives without being that interesting.  Any of these three would have some tales to tell of real-world use, on Sundays at the small rationality dojo where they were instructors.  What I'm trying to say here is: don't demand that everyone be Robin Hanson in their secret identity, that is setting the bar too high.  Selective reporting makes it seem that fantastically high-achieving people have a far higher relative frequency than their real occurrence.  So if you ask for your rationality instructor to be as interesting as the sort of people you read about in the newspapers—and a master rationalist on top of that—and a good teacher on top of that—then you're going to have to join one of three famous dojos in New York, or something.  But you don't want to be too lax and start respecting things that others wouldn't respect if they weren't specially looking for reasons to praise the instructor.  "Having a good secret identity" should require way more effort than anything that could become a token requirement.

Now I put to you:  If the instructors all have real-world anecdotes to tell of using their knowledge, and all of the students know that the desirable career path can't just be to become a rationality instructor, doesn't that sound healthier?

 

Part of the sequence The Craft and the Community

Next post: "Beware of Other-Optimizing"

Previous post: "Whining-Based Communities"

Group Rationality
Personal Blog

68

Mentioned in
Load More (5/19)
Mandatory Secret Identities
0Insert_Idionym_Here
New Comment
186 comments, sorted by
top scoring
Click to highlight new comments since:
Some comments are truncated due to high volume. (⌘F to expand all)Change truncation settings
[-]Scott Alexander391

What does this post even mean? I don't have access to my own respect function, and I don't know if I'd mess with it this way even if I did.

If you were to say tomorrow "I've been lying about the whole AI programmer thing; I actually live in my parents' basement and have never done anything worthwhile in any non-rationality field in my entire life," then would I have to revise my opinion that you're a very good rationality teacher? Would I have to deny having learned really valuable things from you?

Or would I have to say, "Well, this guy named Eliezer taught me everything I know, he's completely opened my mind to new domains of knowledge, and you should totally read everything he's written - but he's not all that great and I don't have any respect for him and you shouldn't either" when referring people to your writing?

Or to put it another way...let's say there are two rationality instructors in my city. One, John, is a world famous physicist, businessman, and writer. The other, Mary, has no particular accomplishments outside her rationality instruction work. However, Mary's students have been observed to do much better at their careers than John's, and every time ... (read more)

Reply
[-]Eliezer Yudkowsky270

If you were to say tomorrow "I've been lying about the whole AI programmer thing; I actually live in my parents' basement and have never done anything worthwhile in any non-rationality field in my entire life," then would I have to revise my opinion that you're a very good rationality teacher? Would I have to deny having learned really valuable things from you?

But the fact that reality doesn't disentangle this way, is in a sense the whole point - it's not a coincidence that things are the way they are.

If we get far enough to have external real-world standards like those you're describing, then yes we can toss the "secret identity" thing out the window, so long as we don't have the problem of most good students wanting only to become rationality instructors themselves as opposed to going into other careers (but a teacher who raised their students this way would suffer on the 'accomplished students' metric, etc.). But on the other hand I still suspect that the instructors with secret identities would be revealed to do better.

Reply
[-]CronoDAS470

I've never seen anything from Eliezer that proves that he's done anything at all of value except be a rationality teacher. I know of two general criteria by which to judge someone's output in a field that I am not a part of:

1) Academic prestige (degrees, publications, etc.) and 2) Economic output (making things that people will pay money for).

Eliezer's institution doesn't sell anything, so he's a loss on part 2. He doesn't have a Ph.D or any academic papers I can find, so he's a loss on part 1, as well. Can SIAI demonstrate that it's done anything except beg for money, put up a nice-looking website, organize some symposiums, and write some very good essays?

To be honest, I'd say that his output matches the job description of "philosopher" than "engineer" or "scientist". Not that there's anything wrong with that. Many works that fall broadly under the metric of philosophy have been tremendously influential. For example, Adam Smith was a philosopher.

Eliezer seems to have talents both for seeing through confusion (and its cousin, bullshit) and for being able to explain complicated things in ways that people can understand. In other words, he'd be an amazing university professor. I just haven't seen him prove that he can do anything else.

Reply
[-]Paul Crowley250

Yes - in fact, the only thing that leads me to suspect that EY and SIAI are doing anything worth doing is the quality of EY's writings on rationality.

Reply
4badger
EY has a lengthy article in this volume if that counts as academic. As has been said, being a theoretician seems distinct enough from teaching that it should count as a day job. I still view Eliezer as more of a teacher than a theoretician, but I don't think Eliezer is saying teachers don't have to be completely divorced from their subject in their day job to avoid affective death spirals.
6Scott Alexander
Right. Our difference of opinion here is clearly nontrivial. I'll put it on the list of things to write posts about.
3arthurlewis
Are you saying that teachers who don't externally practice the thing they're teaching won't make good teachers? Or that they're not worthy of respect at all? If the former, I agree with Yvain and others that we have better metrics for determining teacher quality. If the latter, I'm not sure why this would be the case. The comparison to literary critics doesn't answer that question; it just accesses our assumed cached thoughts about literary critics. What's the problem with people wanting to be literary critics? The post proposes a required formula for respect, but it never explains what quantity that formula intends to maximize. What's the goal here?
[-]anonym210

Is the point about respect for instructors supposed to generalize to instructors of disciplines other than rationality?

If so, what do you make of Nadia Boulanger? Her accomplishments as a musician (or otherwise) are unimpressive relative to those of her students and peers, and yet she is regarded as one of the greatest music teachers ever, and is accorded correspondingly deep respect by music historians, composers, etc. Are they all wrong to respect her so much, or does it not apply to music or this case?

It seems to me that a better formula for determining respect would somehow reflect the respect given to her students which they say is significantly due to her influence as a teacher. For example, if Aaron Copland singles her out as an amazing teacher who profoundly affected his musical life & education, then she deserves some of the respect given to him. And likewise for her many other students who went on to do great things.

There seems to be an implicit underlying belief in this post that teaching is not (or should not be) an end in and of itself, or at least not a worthy one. I think Boulanger and teachers of her caliber show that that's just not the case.

Reply
8Eliezer Yudkowsky
I was thinking about that - a clause for respecting teachers with great students, should they have them. It still gives people the right incentives.
[-]AlanCrowe240

You've got things the wrong way round. It is the quality of the teacher's students that tell us whether we wish to study under her. The teachers own achievements are a proxy which we resort to because we need to decide now, we cannot wait to see the longer term effects on last years students.

Another proxy is the success of the teacher in getting her students through examinations. This is a proxy because we don't really want the certificate, we what the achievement that we think it heralds. We can assess the strength of this proxy checking whether success in the examinations really does herald success in real life.

I agree with the conclusion of the original post but find the argument for it defective. The key omission is that we don't have a tradition of rationality dojo's, so we do not yet have access to records of whose pupils went on to greatness. Nor do we have records that would validate an examination system.

Notice that the problems of timing are inherent. The first pupils, who went on to real world success, prove their teachers skill in an obvious way, but how did they choose their teacher? Presumably they took a risk, relying on a proxy that was available in time for the forced choice they faced.

Reply
0Annoyance
Yes, precisely. The issue isn't how we can become a better teacher, or find one to study under. The first question, that MUST be asked before all others, is: what does it mean to be a good teacher, and how can we define the relevant differences between teachers? Once that question has an answer, we can begin searching for ways to make ourselves better match that defined meaning, or in signal traits in others that indicate they're likely to match that definition well. Concepts like "has students that will accomplish great things" aren't useful for a variety of reasons. And once someone has developed a reputation for being a great teacher, they're likely to attract students with a lot of potential (assuming there are working metrics for potential that are actually consulted, as opposed to rich people simply buying a place for their talentless children). The reputation alone would result in the teacher's students doing better than most. Evaluating the teacher requires that we have some way of determining, or at least guessing, what a student's performance would have been without the teaching.
0gucciCharles
Isn't teaching itself a skill? So what that she was a bad musician, she was obviously a first rate teacher (independent of the subject that she taught).
0anonym
As long as we determine how much of their students' success is attributable to the teacher, it seems reasonable. It seems we could make those sorts of judgments by: * comparing the success of the students of a teacher with the success of students of other teachers having equally talented students (e.g., compare Boulanger's students' success with that of students of contemporaneous Fontainebleau teachers); or * when successful people have typically studied with many different teachers, asking them how much of their success they attribute to the influence of their various teachers.
7MBlume
I do find cases like this surprising, though. What was it that she was able to teach to her students that she could not put to use herself?
4gucciCharles
She gives a pattern of feedback that makes the students practice well? In the sense that she gives positive feedback she functions more as a motivator than as a teacher. Her skill is teaching, it's only happenstance that she teaches music; has she taught shoe polishing or finger painting she would have produced the best shoe polishers and the most skilled finger painters. Perhaps she doesn't have many complex skills but has strong fundamentals (think Tim Duncan of the NBA Spurs). She might make her students practice the fundamentals which will allow them to do more complex work as they get older. Finally, she might have knowledge more advanced than her skill. She might not have the hand eye coordination or the processing speed to play sophisticated music but she might know how it's done. Imagine a 5 foot tall jewish guy that loves basketball. He's not gonna make the NBA. It's simply not gonna happen. However, he might understand the game better than many NBA players. Likewise he might be the best basketball coach in the world even though his athleticism (and hence his basketball playing skills) is less than that of NBA players. Likewise the teacher might have had a strong theoretical understanding but not have had the ability to put her theoretical knowledge into practice.
2pre
The first thing that comes to mind is maybe she's able to teach students how to practice more in their youth than she did. That'd work at least.
1AlanCrowe
I was thinking of Nadia Boulanger, with Astor Paizzolla as the distinguished pupil. Piazzolla was trying to be a classical composer, but Boulanger said his classical music was lifeless, it was his tangos that had fire. Perhaps the multi-talented young pupil faces perilous choices about where to focus his energies. Whether the older great teacher can warn effectively against the common errors probably depends on having a breadth of experience, perhaps having put in many years as a mediocre teacher, following up pupils and noticing how things worked out. The teacher's own youthful errors might be uniformative even if severe.
1anonym
Yeah, definitely surprising, but genius in any form is surprising. There is an essay here by a former student that gives a sense of how she taught. And Philip Glass describes her as the decisive influence on him in this article, which also talks about her teaching a little. Here's an interesting passage from Nadia Boulanger: A Life in Music: It was Nadia's manner rather than her materials that was unique. Her intensity, her emotional involvement with her students, her broad knowledge of music in general, and her ability to project her own passionate enthusiasm for each detail as well as its over-all form, were the qualities that made her extraordinary. Her electric personality brought a distinctiveness to everything that Nadia did. In this, lies what one reviewer called "the difference between good teaching and great teaching," for in the latter "the student feels that the teaching enacts an extraordinarily intimate and demanding relation between the teacher and his subject, a relation such that the teacher's sense of his subject is indistinguishable from his sense of life."
[-]gwern170

Has this requirement been successfully implemented for CFAR instructors?

Reply
1Nisan
It's not a formal requirement, but I'm personally impressed by the prior accomplishments of some of the CFAR staff and some of the careers of CFAR consultants. And there is at least one person who somehow has a non-CFAR career while also working full-time for CFAR. EDIT: Read about them here.
[-]JulianMorrison160

  1. In the boost phase of a newly launched idea, it's actually a really good idea to train teachers. That gives you exponential growth.

  2. It's a fail if the discipline gets into a death spiral about teaching teachers to teach teachers, iff the recursion lacks a termination condition. (Suitable conditions left as an exercise.)

  3. Even in the cruise phase, an idea needs a teacher replacement rate of >= 1.

  4. In cruise phase, it's a fail if every student wants to teach. But I don't see how it's a fail if some students want to teach and proceed to do so. Nor do I see how it's a fail if they end up being most of the teachers.

  5. The intersection of two very rare categories of people is nobody.

  6. Aren't you the same guy who, just a few days ago, pointed out how much better a trained professional is at his job than some volunteer? Teaching is a nontrivial skill.

Reply
3Eliezer Yudkowsky
Most memes that grow exponentially do not manage to stay sane. What's best for the exponential growth of a meme (as though it were a bacterium with no identity other than itself) may not be best for the culturation of a cause. Exponential growth is good, I agree, but the fastest possible exponential growth... seems more doubtful. Exponential growth around friends giving friends copies of a fixed book, seems safer and shallower; the book won't change as it's passed around. The building-up of rationality dojos is a longer, slower endeavor which should be more carefully gotten right. Plus you can look for students who've already done something interesting with their lives and train them to be the teachers.
0JulianMorrison
Hmm. I'll concede a measured rate of startup, although I can't offhand think of any meme that got deranged by fast growth (and was sane to begin with). Perhaps, adopt the martial art idea of not giving out too many certificates-to-teach, and using lineage to check accreditation?
[-]Nominull150

No.

You're wasting huge amounts of optimization power, here, in two different ways. Firstly, you're saying that no one should focus his efforts on becoming a good rationality instructor, that any work he does on that is entirely meaningless unless he is at least as good at something else. Secondly, you're saying that no one should focus his efforts on instructing people in rationality, that they should spend most of their time on whatever other thing it is that makes them impressive. If you have someone who is naturally better at instructing people in rationality than in anything else, you are wasting most of the surplus you could have gained from him in these two ways.

I'm sympathetic to your concern, but surely there must be a way we can avoid throwing out the baby with the bathwater?

Reply
6Eliezer Yudkowsky
Well... go ahead and suggest a way to avoid throwing out the baby with the bathwater? I mean, we're talking about some pretty scary bathwater here.
[-]Nominull300

Personally I suspect that the bathwater only really gets dirty when you are teaching something that is essentially useless in modern society, like martial arts or literary criticism. Most people who study, say, engineering don't do so in the hopes of becoming teachers of engineering.

Now you might say that this is because teachers of engineering are expected to also do research, but firstly that doesn't explain the disparity between fields, and secondly, I don't think that the example of tertiary education is one to aspire to in this way. I seem to recall you are an autodidact, so you may not have the same trained gut reaction I do, but I have seen too many people who did not have the skill of teaching but were good researchers teaching horribly, and I remember one heartbreaking example of an excellent teacher denied tenure because the administrators felt his research was not up to snuff too well, to want to optimize rationality teachers on any basis other than their ability to teach rationality.

Reply
2Alex Flint
Martial arts seem to get an unreasonably bad rep on LW. It's at least as useful as painting or writing fiction, and I consider those to be fine personal development endeavours.
[-]Nornagest120

While I think martial arts are pretty useful by hobby standards (although their usefulness is broad enough that they might not be optimal for specialists in several fields), several historical and cultural factors in their practice have combined to create an unusually fertile environment for certain kinds of irrationality.

First, they're hard to verify: what works in point sparring might not work in full-contact sparring, and neither one builds quite the same skillset that's useful for, say, security work, or for street-level self-defense, or for warfare. It's difficult to model most of the final applications, both because they entail an unacceptably high risk of serious injury in training and because they involve psychological factors that don't generally kick in on the mat.

Second, they're all facets of a field that's too broad to master in its entirety in a human lifetime. A serious amateur student can, over several years, develop a good working knowledge of grappling, or of aikido-style body dynamics, or empty-hand striking, or one or two weapons. The same student cannot build all of the above up to an acceptable level of competence: even becoming sort of okay at the entire sp... (read more)

Reply
1Alex Flint
Thanks for a thoughtful reply! You could say much the same about painting/dancing/cooking/writing: There are many different sub-arts; it's hard to master all of them; practitioners can become unduly wedded to a single style; there are examples of styles that have "gone bonkers"; there are many factors in place that hurt the rationality of practitioners. These are all valid concerns, but I don't think they're particularly problematic within martial arts in comparison to other hobbies.
1Luke_A_Somers
You could say point 2 about those, but points 1 and 3 stand. If you are half-way decent at painting/dancing/cooking/writing and think you're pretty good, it is unlikely to get your face stove in the first time you try it seriously. This leads to your getting feedback and improving. You can watch serious, nothing-held-back demonstrations as public performances (or to take home and study, in the case of writing) for a nominal fee.
4Desrtopa
Really? I've always thought the opposite; that there's a common sense on this site that martial arts are a discipline worthy of taking seriously and investing far more attention in than I would have thought they merited with respect to their applications to rationality. I may be very interested in martial arts, but in most of my social outlets I don't have nearly as much of a sense of it being a shared interest.
1elharo
Painting and writing fiction produce items that can then be enjoyed by many other people who are neither writers nor painters. Martial arts produces almost nothing, aside from an occasional sports event.
2Eliezer Yudkowsky
Fair point. But this depends on things starting out healthy so that they stay healthy.
0MichaelHoward
[mistake] How about RatRespect1 = min(RatRespect0, sqrt(RatRespect0)^2 + (NonRatRespect0)^2))? [edit] Confound you, Pythagoras! What I meant to say was... RatRespect1 = min(RatRespect0, sqrt(RatRespect0 x NonRatRespect0) There's no sudden ceiling, but you still get wiped for neglecting the real world.
7AlexU
Do you people actually think in terms of equations like this? Once you begin throwing in exponents, I think the metaphorical/illustrative value of expressing things in math drops off quickly.
0MichaelHoward
Not very well in my case, it seems, my apologies. Exponents now thrown out again.
2AlexU
That wasn't meant as a criticism of you specifically. I've just noticed that people on this site like to use equations to describe thought processes, some of which might be better communicated using everyday language. I'd argue Eliezer's post is an even worse example -- why not just say "the lesser of the two quantities" or something?
[-]SoullessAutomaton100

To be fair, for people who are used to thinking in math, pseudo-mathematical notation is as readable as English, with advantages of brevity and precision.

"People used to thinking in math" currently describes a large portion of users on this site. Use of gratuitous mathematical notion is likely to help keep it that way.

Reply
6AlexU
"Use of gratuitous mathematical notion is likely to help keep it that way." Is that desirable? (Not saying you're implying it is.) The community could probably benefit from some smart humanities types.
3SoullessAutomaton
I was actually trying to imply that it isn't desirable, so yes, I agree fully.
0SystemsGuy
First post, so I'll be brief on my opinion. I would say "it depends". To communicate between people and even to clarify one's own thoughts, a formal language, with an appropriate lexicon and symbols, is a key facilitator. As for desirability of audience, the About page says "Less Wrong is an online community for discussion of rationality", with nothing about exclusivity. I would suggest that if a topic is of the sort that newbies and lay people would read, then English is better; if more for the theorists, then math is fine.
7Nebu
I personally find "min(A,B)" clearer than "the lesser of A and B", but I'm on the autistic spectrum.
5Eli Tyre
Oh. I thought that the use of min( ) here, was immediately readable and transparent to me. The meaning of "the lesser of the two quantities" is less obvious, and the phrase is longer to say.
2MBlume
hmmm...it's a little awkward reading the math without TeX, but I think assuming all variables real, that simplifies to RatRespect1=RatRespect0
0[anonymous]
How about RatRespect1 = min(RatRespect0, sqrt(RatRespect0)^2 + (NonRatRespect0)^2))? There's no sudden ceiling, but you still get wiped for neglecting the real world. HT to Pythagoras.
0[anonymous]
How about Rat_Respect1 = min(Rat_Respect0, sqrt(Rat_Respect0)^2 + (Non_Rat_Respect0)^2))? There's no sudden ceiling, but you still get wiped for neglecting the real world. HT to Pythagoras.
0outlawpoet
I agree with this comment vociferously. The upper bound isn't a terrible idea, but it would, for example, knock E.T. Jaynes out of the running as a desirable rationality instructor, as the only unrelated competent activity I can find for him is the Jaynes-Cumming Model of atomic evolution, which I have absolutely zero knowledge of.
[-]Eliezer Yudkowsky120

knock E.T. Jaynes out of the running

Dude, what on Earth are you talking about. E. T. Jaynes was a Big Damn Polymath. I seem to also recall that in his later years he was well-paid for teaching oil companies how to predict where to drill, though that's not mentioned in the biography (and wouldn't rank as one of his most significant accomplishments anyway).

Reply
1outlawpoet
Not something I was aware of, but good to know. I wasn't aware of anything from before his career as an academic, 1982-onward. His wikipedia article doesn't mention anything but the atom thing. But he certainly set out to be a Professor of rationality-topics.
8saturn
Regardless of the merits of E. T. Jaynes, we should place the activity of a rationality instructor in a separate mental bucket than a rationality theoretician. I would say that making a significant original intellectual advance counts as a real accomplishment.
[-]RobinHanson110

Well if we develop rationality tests, then you should rely on the teachers who help their students do better on tests. And if you can't develop tests, then I don't see why you'd think you had evidence that any particular person was good at teaching rationality. Relying on their ability to do something useful as a predictor of their ability to teach rationality seems nearly as bad as relying on their publication record, or their IQ, or wealth, etc. I say focus on developing tests.

Reply
[-]Eliezer Yudkowsky110

(Blinks.)

I wonder if this idea comes as a shock because everyone was planning on becoming rationality instructors, i.e., I should have warned everyone about this much earlier?

Is it offputting on some other level?

But I must also consider that it might really be that stupid. Damn, now I wish I knew the actual number of upvotes and downvotes!

Reply
5Scott Alexander
I don't think too many people are actually considering "rationality instructor" as a career path at this point - which reminds me - what exactly are your plans for this rationality dojo thing anyway? Is it just something you like to talk about, or something you plan to one day set up? Are you hoping people from Less Wrong will start the first ones, or that people from Less Wrong will be students in ones set up in some other way?
4Paul Crowley
When you (or anyone) says "rationality dojo", how literally is it meant? Is it specifically a physical meeting, rather than a web community? More literally, is this meant as a meeting of equals or an instructor with pupils? How much of the formalism of the dojo would you import? How would you change the relationship of sensei and pupil? I'm not so sure about wearing robes, and I draw the line at getting thwacked on the head with a stick. I am keen to increase the number of rational people, but there are a great many means by which a thing passes from mind to mind, and I'm not sure a dojo would be the first model I'd reach for - have I missed a post where this model is set out in more detail?
3Eliezer Yudkowsky
I don't think I can afford to divert my attention into setting one up, but I've heard others already discussing it, so it's worth placing some Go stones around it.
2Scott Alexander
Really? If it's not too private, who's been discussing it?
[-]AnnaSalamon120

I don't know if I'm part of who Eliezer heard, but I'm planning on trying to start a rationality training group on Saturdays in the SF bay area, for middle and high school students with exceptional mathematical ability. I want to create a community that thinks about thinking, considers which kinds of thinking work for particular tasks (e.g., scientific progress; making friends), and learns to think in ways that work. The reason I'm focusing on kids with exceptional mathematical ability is that I'm hoping some of them will go on to do the kind of careful science humanity needs, with the rationality to actually see what actually helps. The aim is not so much to teach rationality knowledge, since AFAICT the "art of human rationality" is mostly a network of plausible guesswork at this point, but to get people aiming, experimenting, measuring, and practicing in a community, sharing results, trying to figure out what works and actually trying the best ideas (real practice; community resistance to akrasia). With some mundane math teaching mixed in.

As to "day job" credentials, I've had unusual success teaching mathematical thinking (does this count as "day job"? at least math teaching success is measurable by, say, the students' performance on calculus exams), bachelor degrees in math and "great books", and two or three years' experience doing scientific research in various contexts. I don't know if this would put me above or below Eliezer's suggested bar to a stranger.

Reply
3TheOtherDave
How has this project been going?
2Eliezer Yudkowsky
You're focusing on easy-to-verify credentials of the sort you'd list on a resume to be hired by some skeptical HR person. You have a secret identity.
2AnnaSalamon
My secret identity just says that some combination of you and Michael Vassar thought I was worth taking a chance on. I was trying to do some analog of cross-validation, where we ask whether someone who was basically following your procedure but who didn't know me or have particular faith in your or Michael Vassar's judgment, would think it okay for me to try teaching. I was figuring that your focus on day job impressiveness was an attempt to get away from handed-down lineages of legitimacy / "true to the Real Teachings"ness, which Objectivism or martial arts traditions or religions sometimes degenerate into.
3Eliezer Yudkowsky
More of an attempt to make sure that people write instead of just doing literary criticism.
1AnnaSalamon
Got it. Sorry; I think I rounded you to the nearest cliche, maybe because of the emotional reaction you suggested some of us might be having.
3AnnaSalamon
FWIW, part of my own emotional reaction to it did come from that, though I noticed and have my reaction tagged as an emotionally contaminated thing to be wary of.
3MBlume
I had hoped to become a rationality instructor of some stripe, but with an apprentice period as an experimental physicist, in order to give concreteness to my teaching. So, no particular degree of shock here.
2SoullessAutomaton
Speaking only for myself--I am here, consciously and explicitly, to learn rationality for its own benefits. I have no overwhelming interest in teaching others and, all else equal, have other things I would prefer to be doing with my life. I didn't vote either way on the post because I am ambivalent to it. It felt underdeveloped compared to your usual material, and to some extent seems like you're getting ahead of yourself on this "teaching rationality" thing--the current understanding of applied rationality in this community here doesn't seem to justify raising the concern yet. Perhaps the idea would have been better presented in the context of one of your parables/short stories/&c.?
[-]gwern110

Thales, so the story goes, because of his poverty was taunted with the uselessness of philosophy; but from his knowledge of astronomy he had observed while it was still winter that there was going to be a large crop of olives, so he raised a small sum of money and paid round deposits for the whole of the olive-presses in Miletus and Chios, which he hired at a low rent as nobody was running him up; and when the season arrived, there was a sudden demand for a number of presses at the same time, and by letting them out on what terms he liked he realized a large sum of money, so proving that it is easy for philosophers to be rich if they choose, but this is not what they care about.

Reply
2jimrandomh
Huh? There is no way that knowledge of astronomy could possibly have told him about the olive crop. It seems more likely that his useful knowledge was of economics and business, but that he made up a story about astronomy to impress his peers.
[-]gwern140

"no way...could possibly..." (emphasis added)

This is a good example of what I meant over in the evolutionary psychology thread; coming up with evolutionary psychology explanations is a good practice to avoiding succumbing to 'arguments from incredulity', as I like to call this sort of comment.

"Oh, I couldn't think of how astronomy could possibly be useful in weather or crop forecasting, so I'll just assume the stories about Thales are a lie."

I'll leave this here for you.

" Forecasting Andean rainfall and crop yield from the influence of El Niño on Pleiades visibility", Nature 403, 68-71 (6 January 2000):

"Farmers in drought-prone regions of Andean South America have historically made observations of changes in the apparent brightness of stars in the Pleiades around the time of the southern winter solstice in order to forecast interannual variations in summer rainfall and in autumn harvests. They moderate the effect of reduced rainfall by adjusting the planting dates of potatoes, their most important crop1. Here we use data on cloud cover and water vapour from satellite imagery, agronomic data from the Andean altiplano and an index of El Niño va

... (read more)
Reply
0Paul Crowley
I read it as an injunction to focus on fixing my own rationality and making best use of it, and not to think about how to help other people be more rational. That runs entirely contrary to my own hopes for making the world a better place. If all you mean is "spread rationality, but keep the day job" then absolutely, I'm keeping the day job, it pays better. The idea of a rationality pressure group has crossed my mind, but if I were to work for such a thing it would not be in the role of instructor, and I could probably do more for such an organisation by keeping the day job and giving it money in any case.
0PhilGoetz
It's an idea that is common among writers (with respect to writing instructors). Not the secret identity part, though. Eliezer's idea is a bit different, because success in any area of life should indicate rationality. I don't understand the secret identity part. If one identity is secret, how are students supposed to know whether to respect the instructor for accomplishments under his/her non-instructor identity? (If you're a rationality instructor or practitioner, having a secret identity is probably a good idea anyway, so you're not the first against the wall when the religious-Luddite anti-transhuman pogrom begins.)
1MBlume
He's joking about the secret part -- think "day job"
0[anonymous]
The idea never occurred to me -- not when I was sincerely involved in martial arts, and not since becoming sincerely dedicated to rationality. I'd be quite surprised if it has occurred to more than a few people here.
-1[anonymous]
Perhaps few readers are thinking about becoming rationality instructors, so they feel it doesn't apply to them. That would likely diminish their estimation of its importance.
[-]LongInTheTooth100

Some thoughts from my experience in a martial arts dojo:

  1. We avoid lots of failure modes by making sure (as far as reasonably possible) that people are there to train first and everything else second. One consequence of this is that we don't attach a whole lot of our progress to any particular instructor; we're blessed with a number of people who are really good at aikido, and we learn from all of them, and from each other.

  2. On setting the bar too high for instructors: Most martial arts rely on a hierarchy of instructors, where the average dojo head is a reasonably normal person who is expert but not necessarily elite at the discipline. The "famous" people in the art travel around and deliver seminars to everybody else. Dojo head type people will also travel to attend more seminars than the average junior student, for obvious reasons.

All sorts of human enterprises work this way (although the formality of the hierarchy varies widely); everything from yoga to religions to Linux Users Groups. It's a good system.

Reply
[-]AnnaSalamon100

How much of what you're trying to do could be accomplished by largely tabooing the term "rationality" in rationality dojos, and having the community be really really attached to that tabooing? So that the dojos are for "finding ways of thinking that actually bring accurate beliefs" and "finding ways of thinking that actually help people reach their goals", with mostly no mention of a term like "rationality" that's easy to reify? If we talked like that, actual and prospective students and teachers might naturally look outward, to the evidence that various thinking processes were or weren't helping. Such evidence would be found partly in terms of the actual "day job" accomplishments (or lack of accomplishments) of the teacher, and also in terms of "day job" accomplishments of the students after vs. before joining the group, and also in terms of any measures that a group of active, experimentally minded rationality students could think up of whether they were actually becoming better at forming accurate beliefs.

Reply
2Vladimir_Nesov
You can taboo a word, or even a concept, but you can't taboo a meaningful regularity, pretend that it's not there. The problem with belief-in-belief-in-rationality is the same as with other lost concepts, one of the essential lessons to learn, not something to shoo away. If you can't attain even this, what aspirations?
3AnnaSalamon
I'm not proposing we pretend there's no regularity to "types of thinking that help us form accurate beliefs, across domains". Not at all. I'm proposing we stay attentive to the evidence as to what those types of thinking actually are and aren't, by spelling out our full goal as much as possible. If we use the term "rationality" as a shorthand instead of spelling out that we're after "types of thinking that actually help us form accurate beliefs", it's easy for the term "rationality" to become un-glued from the goal. So that "rationality" gets glued to "that thing we do in the rationality dojo" or to "whatever the Great Teacher said" or to "anything that sets me apart from others and lets me feel superior, like using long sentences and being socially awkward", instead of being a term for those meaningful regularities we're actually trying to study (the meaningful regularities in thinking methods that actually work). Well, yes, I agree that a rationality dojo should talk about lost purposes, about the trouble with belief in belief in general, and about what exactly goes wrong when people speak overmuch of "rationality" instead of keeping their eyes on the prize. Is this supposed to be in tension with the suggestion that we, as a community, build a strong norm against talking overmuch of "rationality" and for, instead, speaking of "kinds of thinking that help us form accurate beliefs / achieve our goals"? I'm imagining that it's precisely by having a really clear view of the standard "lost purposes" failure modes, and of their application to "rationality" learning, that we can maintain such a norm.
1Vladimir_Nesov
But for some reason we are talking about a specific failure mode, one that is not necessarily the single best case to demonstrate the general principles, and one that by itself is clearly insufficient. Investing disproportionally in this single case must have additional purposes. I can see two goals: * Safeguarding the movement on early stages, where it's easy to start in a wrong direction * Acting as a safety vent, compensating for the difficulty in certifying the sanity of the movement.
[-]gjm100

What work is the word "secret" doing in this post? It seems to me that you're talking about public identities, ones visible to outsiders, ones that potential students (not yet enrolled in the Conspiracy) can look at to evaluate would-be instructors. Are you using the phrase "secret identities" merely because it sounds cool?

Reply
[-]AlexU120

Ditto with "conspiracy." I'd argue that giving LW the language and trappings of a 12-year old boys' club is ultimately detrimental to its mission, but it looks like I'm in the minority.

Reply
3gjm
The business about the Bayesian Conspiracy is, I think, more an in-joke than anything else. Eliezer's written various bits of fiction set in a future world featuring an actual "Bayesian Conspiracy", and he's on record as saying that there's something to be said for turning things like science and rationality into quasi-mystery-religions (though I expect he'd hate that way of putting it) -- but he's not suggesting that we actually should, nor trying to do so. Dunno whether such things help or hinder the mission of LW. I think it would be difficult to tell.
3AlexU
It just seems at odds with the scientific ethos of cutting out the bullshit whenever possible. Instead, Eliezer seems bent on injecting bullshit back into the mix, which I'd argue comes at the expense of clarity, precision, and credibility. However, I do realize it's a calculated decision intended to give normally dry ideas more memetic potential, and I'm not in a position to say the trade-off definitely isn't worth it.
3JulianMorrison
Deliberately so. The original OB posts started with it as a thought experiment, "what if we kept science secret, so people would appreciate its Awesome Mysteries?"
1Paul Crowley
Despite that, I think that whole style is a tremendous mistake. It's an interesting thought experiment, but we should be clear that it runs completely counter to the things that actually bring about accurate results.
5Eliezer Yudkowsky
Ironic, rather. I considered "Mandatory Alternate Lives" but "alternate life" simply doesn't have the phrase-recognition impact of "secret identity". There is no phrase that means exactly what I want; so I use "secret identity" in an obviously inappropriate way.
[-]CronoDAS320

I think the phrase you want is "day job".

Reply
2JulianMorrison
If anything it's the teaching that ought to be under a nom de plume. I've heard more than once the complaint about universities, that they care more about getting an impressive name than whether he can teach.
[-]AnnaSalamon90

How much of Objectivism's failure was due to its teachers not having developed sufficient awesomeness elsewhere, and how much was due to the fact that it, say, tried to claim that it had the One True Method of Thought, instead of fostering an environment where all teachings were conjectural, teachers were facilitators of investigation instead of handers-down of The Answer, and everyone together tried to figure out what worked?

I mean, to what extent can we avoid similar failure modes by fostering a culture that doesn't reify anyone's teachings, but that instead tries to foster a culture of experimenting, thinking up new strategies, pooling data, and asking how we can tell what does and doesn't work?

Reply
[-]MrHen60

Among the failure modes of martial arts dojos, I suspect, is that a sufficiently dedicated martial arts student, will dream of...

...becoming a teacher and having their own martial arts dojo someday.

I do not think this analogy fits. Martial arts is a self-contained bubble. What else is there to do but teach? To use a variation on the analogy, if someone being trained in the United States Marine Corps were given the question of what a truly dedicated student of the USMC were to become, they would probably answer along the lines of someone who kills th... (read more)

Reply
[-]taelor20

Among the failure modes of martial arts dojos, I suspect, is that a sufficiently dedicated martial arts student, will dream of.. becoming a teacher and having their own martial arts dojo someday.

I think that accademia is also subject to this mode of failure. As an exercise, try to think of great literary figures who were also professors of literature at major universities. Off the top of my head, I can think of exactly one: Vladimir Nabokov, and he was notably contemptuous of his colleagues. Can anyone else think up anymore?

Unsurprisingly, Paul Graham h... (read more)

Reply
[-]Mike Bishop20

Obviously success in other realms is bayesian evidence that someone would make a better rationality instructor. But as many others have argued, in this post Eliezer exaggerates the importance of this type of evidence.

Reply
[-]JulianMorrison20

I have a question: why are you panicking about this now? It's not like we have a huge problem yet with too many teachers, or too many freshly founded schools.

Reply
6Eliezer Yudkowsky
So that, having written up my thoughts on the subject, I can vanish into an appropriately dark basement for 5 years and not find armies of deranged Objectivists when I peek out? I'm trying to write up now everything that needs to be written up, which includes a contingency in case The Book takes off (should it be written and sold).
3JulianMorrison
The traditional fix is to anoint some disciples to teach in your stead.
[-]SoullessAutomaton130

Yes, and my impression has been that annointed disciples are generally the instigators of things going subtly wrong in self-reinforcing ways. People with big, novel ideas are not necessarily good judges of character.

Reply
0[anonymous]
Especially if the interview lasts 5 minutes and the jackass winds up writing half your scripture. (no, I don't hold a grudge against Paul, what are you talking about?)
[-]Roko20

I've been expecting a deliberately daft post from Eliezer Yudkowsky and/or Robin Hanson to see whether we vote them up just based upon status.

I think this is it.

Reply
6Cyan
Eliezer has been very clear on OB that he doesn't write things with the intention of covertly testing or manipulating his audience. (Of course, anyone who did test or manipulate his audience might say the same thing...)
1Roko
And, of course, they wouldn't even admit to it afterwards, in all likelihood!
2Paul Crowley
If I were going to do this, I would write something that flattered my audience - this does the very opposite. Besides which, we know that EY is voted down based on status - there was a discussion of it in the March open thread.
[-]elharo10

Makes sense, though I will quibble with your opening line. What you say about martial arts dojos was probably true up until about twenty years ago, but today I suspect a sufficiently dedicated martial arts student is in fact dreaming of becoming a champion MMA fighter.

And you know, now that I think about it, even twenty years ago, I'm not sure anyone was dreaming of becoming a dojo owner. That was just what they could practically achieve. But they were dreaming of becoming a Dark Lord:

"Surely you've wanted to hurt people," said Professor Quirrell. "You wanted to hurt those bullies today. Being a Dark Lord means that people you want to hurt get hurt.

Reply
[-]marc10

I guess the failure mode that you're concerned with is a slow dilution because errors creep in with each successive generation and there's no external correction.

I think that the way we currently prevent this in our scientific efforts is to have both a research and a teaching community. The research community is structured to maximise the chances of weeding out incorrect ideas. This community then trains the teachers.

The benefits of this are that you get the people who are best at communicating doing the teaching and the people who are the best at research... (read more)

Reply
1Eliezer Yudkowsky
Hm. Arguably I should only be worried about fast dilution rather than slow dilution. But I'm also worried that the community grows slower if it's inward-looking, and hope for faster growth if it's involved with the outside world. Entirely possible. But I'm not sure I have so much faith in the system you describe, either. The most powerful textbooks and papers from which I get my oomph are usually not by people who are solely teachers - though I haven't been on the lookout for exceptions, and I should be.
2Paul Crowley
Er, I thought the difference between religious and scientific teachings was that scientific teachings didn't have to worry about dilution? It seems like you put a high probability on this community disappearing into a death spiral of some sort without you - I would have thought we should worry more that we're already in one which we haven't picked up on.
0Eliezer Yudkowsky
More of a difference between things that are hard vs. easy to teach and measure. Businesses have the same problem with a great CEO trying to hire great employees, dilution of corporate culture, etc. - they have highly quantifiable output at the end of the day, but in the middle of the day and the middle steps of the process, it's not as easy to measure. I anticipate a beginning period extending for at least several years when we don't have good metrics because we're still trying to develop them.
1marc
I think that you can legitimately worry about both for good reasons. Fast growth is something to strive for but I think it will require that our best communicators are out there. Are you concerned that rationality teachers without secret lives won't be inspiring enough to convert people or that they'll get things wrong and head into death spirals? From a personal perspective i don't have that much interest in being a rationality teacher. I want to use rationality as a tool to make the greatest success of my life. But I also find it fascinating and, in an ideal world, would stay in touch with a 'rational community' as both a guard against veering off into a solo death spiral and as a subject of intellectual interest. I'm sure that there must be other people like me that are more accomplished and could give inspiring lectures on how rationality helped them in their chosen profession. That would go some way to covering the inspiration angle. As an aside i appreciate why you care about this; I'm always a bit suspicious of self help gurus who's only measurable success is in the self help theory they promote. I wonder whether I'm selecting for people who effectively sell advice rather than effectively use advice.
[-]AshwinV00

The mini-intro to this post on the craft and community sequence page says that it was not well received. But the requirements that this write up recommends really act as beautiful safeguard against becoming pedantic. If I hadnt read this page quite early (before I got past the 25% mark on the sequences), I doubt I would have stopped myself from falling into a happy death spiral (I honestly still really struggle with that one all the time).

It's really hard for me even now to "not speak over much of the way" (though, I mostly think it to myself, ... (read more)

Reply
1AshwinV
Update: I'm over it now. :D
[-]Insert_Idionym_Here00

If you have "something to protect", if your desire to be rational is driven by something outside of itself, what is the point of having a secret identity? If each student has that something, each student has a reason to learn to be rational -- outside of having their own rationality dojo someday -- and we manage to dodge that particular failure mode. Is having a secret identity a particular way we could guarantee that each rationality instructor has "something to protect"?

Reply
3Nick_Tarleton
It's very easy to believe that you're being driven by something outside yourself, while primarily being driven by self-image. It's also very easy to incorrectly believe this about someone else.
1komponisto
Sometimes I wonder if the only people who aren't driven primarily by self-image/status-seeking are sociopaths (the closest human analogue of UFAI).
7JoshuaZ
Sociopaths care a lot about status, and the most extreme sociopaths respond to attempts to reduce their status with violence. I strongly suggest Jon Ronson's "The Psychopath Test" for a highly informative and amusing introduction to psychopathy/sociopathy and its symptoms.
6wedrifid
My understanding of sociopaths makes this seem like approximately the opposite of true. It is the drives other than seeking self-image and status that are under-functioning in sociopaths.
3komponisto
What then do you call someone like the Joker from Batman -- someone who cares not at all how they fit into or are perceived by human society, except as instrumental to gaining whatever (non-human-relationship-based) thrill or fix they are after?
8nshepperd
Fictional?
-5wedrifid
3DaFranker
Failure mode: My "something to protect" is to spread rationality throughout the world and to raise the sanity waterline, which is best achieved by having my own rationality dojo. Beware the meta.
0Insert_Idionym_Here
I agree. I think that failure mode might then be better avoided by restricting possible "somethings", as opposed to adding another requirement on to one's reasons for wanting to be rational.
2DaFranker
Yes, but that's an exercise implicitly left to the reader. Formulating it this way is somewhat intuitively easier to understand, and if you've read the other sequences this should be simple enough to reduce to something that pretty much fits (restriction of "things to protect") in beliefspace. Essentially, this article, the way I understand it, mostly points at an "empirical cluster in conceptspace" of possible failure modes, and proposes possible solutions to some of them, so that the reader can deduce and infer the empirical cluster of solutions to those failure modes. The general rule could be put as "Make rationality your best means, but never let it become an end in any way." - though I suspect that I'm making a generalization that's a bit too simplistic here. I've been reading the sequences in jumbled order, and I'm particularly bad at reduction, which is one of the Sequences I haven't finished reading yet.
Moderation Log
186Comments