The main problem is that a test tests ability to take the test, independently of what its makers intended. The more similar tests are to each other, the more taking the first is training for the second, and the easier it is to teach directly to the test rather than to the skill that inspired the test. The less similar the before and after tests are, the less comparable they are.
Rationality training is particularly tricky because one is to learn formal models of both straight and twisted thinking, recognize when real-life situations resemble those patterns, and then decide how much formal treatment to give the situation, as well as how much weight to give to one's formal model as against one's feelings, reflexive thoughts, and so on.
Traditional classroom tests are set up to best test the first bit, knowledge of the formal models, if one did solve the problems inherent in testing. Even to the extent one can ask people about how one ought to react in the field, e.g. when to use which sort of calculation, that is still a question with a correct answer according to a formal model and one is still not testing the ability to apply it!
These problems resemble those the military has faced in its training and testing. They use indoctrination, simulations, and field tests. Decision making is tested under uncomfortable conditions, ensuring probable good decision making under most circumstances. In general, knowing what they do is likely to be helpful.
The problems with tests are not intractable. One can limit the gain on the second test from having taken the first test by saturating the test taker with knowledge of the test before it is taken the first time, though few would be motivated. One can try to make a test similar to the skill tested, so ability at the test is well correlated with the skill one intends to test. One can try to devise very different sorts of tests that measure the same thing (I doubt that will work here).
One component of a useful classroom test might resemble the classic research on correspondence bias. In it, people judge individuals' support for positions based off an essay they supposedly wrote. Some subjects are told that the writer chose the thesis, others that the writer had it assigned. (The theses were either pro- or anti-Castro.) People inferred that the essay's author significantly agreed with the thesis even when they were told it was assigned to them. The quality of an essay a person produces is some evidence of what they believe, as is their willingness to write it at all, etc., but in general people overly infer others' dispositions from actions they take under social constraint, even when they know of the constraint.
Here is how the framework could translate into a useful rationality test: the test would give people some evidence for something they are biased to overly believe, and the quantity and quality of legitimate evidence in the test would vary widely. One would not be able to pass the test by simply detecting the bias and then declare oneself unmoved in that wrong direction, as one might be able to do for, say, sunk costs. Instead, the valid evidence and invalid inclination would be along the same vector such that one would have to distinguish the bias from the rest of the evidence in the environment.
This solves the problem of having a classroom test be an easy exercise of spotting the biased thought pattern and quashing it. Videos or essays of various people with known beliefs arguing for or against those beliefs could be used to train and test people in this. It's actually probably a skill one could learn without any idea of how one was doing it.
Expressed abstractly, the idea is to test for ability to quantify wrong thinking by mixing it with legitimate evidence, all of which increases confidence in a particular conclusion. This is hard to game because the hard part isn't recognizing the bias. The material's being media from real life prevents testers from imposing an unrealistic model that ignores actual evidence (e.g., a strongly pro-Castro person really might refuse to write an anti-Castro essay).
“I do not say this lightly... but if you're looking for superpowers, this is the place to start.”
--Michael Curzi, summer 2011 minicamp participant
Who: You and a class full of other aspiring rationalists and world-optimizers, from around the world.
What: Two 3-day weekend minicamps and one 8-day minicamp, filled with hands-on activities for applying rationality to your life, your goals, and the making of a better world. (See details in the FAQ.)
When and where: We're running three camps, so that we can do this for three sets of participants: May 11-13 and June 22-24 for the 3-day camps, and July 21-28 for the eight-day camp, all in the San Francisco Bay Area.
Why: Because you’re a social primate, and the best way to jump into a new way of thinking, make friends, and accomplish your goals is often to spend time with other primates who are doing just that.
Other reasons:
Instructors:
Cost: $650 for the three-day programs; $1500 for the week-long program. This includes lodging[1], meals, and tuition.
(Note that this *still* isn't quite enough to make running minicamps sustainable in the long-run; a lodging + meals at retreat centers start at around $90 per person per night, the "three-day camps" include four nights, and these workshops take a staff of about 5 full-time people for over a month each prior to each workshop, most of us at $3k/month, counting curriculum development time (plus miscellaneous expenses). We are trying to strike a compromise between "charge enough that we can run more camps" and staying affordable, especially for our start-up phase; costs will probably go up in following years.)
Three days (or a week) isn’t long enough to learn rationality, but it's long enough to learn how to learn rationality, and to get some momentum toward doing so.
Come meet us, and see what you can do.
Apply now.
Frequently Asked Questions:
1. I’m older. Should I still apply?
Yes! We're aiming for a more diverse crowd and would love to add your wider set of experiences and skills.
2. I’d like to come, but I’m not sure you’ll accept me. Should I still apply?
Absolutely! You can fill out our form in as little 10 minutes. What’s the harm?[2]
3. I’d like to come, but I can’t afford it. Should I still apply?
Yes, you should definitely apply. A limited number of scholarships will probably be available this time, and more may be available later.
(There's also an option on the application form if you want to apply but can't make any of the times - this just says that you want to be part of future minicamps and makes sure we have your application details.)
4. What will we do, exactly?
We're still working out the details. In our current model:
5. I’m new to all this. Will it make sense?
If you’ve read at least fifteen posts from the core sequences, yes it will. If you haven’t: why not read them now?
We’ll also aim for an atmosphere in which everyone is free to make mistakes and to try things, and in which people are receptive to a wide range of skill levels.
6. I’ve already read the Sequences seventeen times, and also I’m a self-made billionaire with three PhDs. Will I learn anything new?[3]
We hope so. We’re covering a good range of material, with much more of a focus on practice and exercise than in the Sequences, incorporating new lessons learned since the LW material was written, and with some instructors who've developed their own takes on rationality.
7. What evidence is there that I'll be glad I went?
After last year's minicamp, participants completed an anonymous exit survey. (With the instructions: "We're asking you these questions to learn how to run camps; please be honest; it'll help us more if you're accurate than if you're positive.") Here are their answers to the most relevant questions:
We also asked participants for testimonials -- statements designed to be shown to others, in case they wanted to recommend such camps. They wrote:
“This was an intensely positive experience. This was easily the most powerful change self-modification I've ever made, in all of the social, intellectual, and emotional spheres. I'm now a more powerful person than I was a week ago -- and I can explain exactly how and why this is true.
At mini-camp, I've learned techniques for effective self-modification -- that is, I have a much deeper understanding of how to change my desires, gather my willpower, channel my time and cognitive resources, and model and handle previously confusing situations. What's more, I have a fairly clear map of how to build these skills henceforth, and how to inculcate them in others. And all this was presented in such a way that any sufficiently analytical folk -- anyone who has understood a few of the LW sequences, say -- can gain in extreme measures.”
--Matt Elder / Fiddlemath
“I expected a week of interesting things and some useful tools to take away. What I got was 8 days of constant, deep learning, challenges to my limits that helped me grow. I finally grokked that I can and should optimize myself on every dimension I care about, that practice and reinforcement can make me a better thinker, and that I can change very quickly when I'm not constrained by artificial barriers or stress.
I would not recommend doing something like this right before another super-busy week, because I was learning at 100% of capacity and will need a lot of time to unpack all the things I learned and apply them to my life, but I came away with a clear plan for becoming better. It is now a normal and easy thing for me to try things out, test my beliefs, and self-improve. And I'm likely to be much more effective at making the world a better place as well, by prioritizing without fear.
The material was all soundly-researched and effectively taught, with extremely helpful supplemental exercises and activities. The instructors were very helpful in and out of session. The other participants were excited, engaged, challenging, and supportive.
I look forward to sharing what I've learned with my local Lesswrong meetup and others in the area. If that's even 1/4 as awesome as my time at the Mini-Camp, it will make our lives much better.”
--Ben Hoffman / Benquo
“I really can't recommend this camp enough! This workshop broke down a complex and intertwined set of skills labelled in my brain as "common sense" and distinguished each part so that I could work on them separately. Sessions on motivation, cognition, and what habits to build to not fool yourself were particularly helpful. This camp was also the first example that I've seen of people taking current cognitive science and other research, decoding it, and showing people what's been documented to work so that they can use it too. It feels to me now as though the coolest parts of the sequences have been given specific exercises and habits to build off of. This camp, and the people in it, have changed my path for the better.”
--David Jones / TheDave
You can also read the full testimonials from everyone who chose to give one.
Apply now
(You can totally fill out the application in just 10 minutes, so you might want to fill in the blanks right now -- we'd like to announce the first acceptances (for May) in the next week)
[1] More exactly, we provide a bed in a shared room at a house or retreat center rented by SIAI.
[2] Sometimes people say they’re “afraid of wasting our time” by sending in an application. In a word, no. If you’re interested in us, we’re interested in you. It takes just seconds to read someone’s form, and our experience shows that many of our highest-value people have been the ones who hesitated to apply.
[3] Okay, fine, this isn’t really a frequently asked question. But seriously, we’ll be covering a lot that isn’t in the sequences -- and the flesh-and-blood experience of meeting other aspiring rationalists is hard to duplicate.
ETA: CMR is still looking for good teachers and curriculum designers. If you're interested, please especially consider coming to a minicamp; we're hoping to find some good hires there.
ETA2: We will probably have answers to all applicants within about two weeks (i.e., by April 16 or so), with answers to the May folks probably earlier than the others. If for some reason you need your application processed *faster* than this, please shoot me an email: annasalamon at gmail.