Good information here that might be "hidden" due to the cute article title. Consider adding a descriptive subtitle, or posting the exercises as their own article.
Yup; I thought the article was spam from the title, but the content is good and does not deserve such an impression.
I'm fairly certain some good rationality drinking games could be thought up. Seriously, alcohol biases in predictable ways and it's both fun and informative to foreground that. (I'll try drinking vodka every day for a week some time soon, see if I can think of something. For science!)
In game B you choose random number between 0 and 1, and you win if the random number will be more tha[n] your confidence interval (0,9, for example).
I don't quite understand that. What does it mean for a number to be larger than a confidence interval, and which confidence interval for what, anyway? Can you (or someone who understands it) explain game B and why preferring it means you're overconfident in a bit more detail?
Thank you for your notice, there were not very clear description, I have edited it. Here is description for the both games:
In game A you will receive a money prize if your statement is true, in other words if the correct number is between your upper and lower bounds.
In game B you generate random number between 0 and 1, and you win if the random is between zero and your credence (0.9, for example). I can say that you win with probability equals to your credence.
If you prefer game A, you may be underconfident; if you prefer game B, you may be overconfident.
If you are overconfident, you may have 90% credence for some interval, but after repeated tests of your estimation the value you estimated will fall into your interval only in 80% or 70% or 60% of tests. So you will likely choose game B, because it has a higher chance of a payoff.
If you are underconfident, you will likely assign 70% credence, but after tests you will get 90% or 80% hit ratio. In this case you will likely choose game A, for the same reason.
Have I answered your question?
Nitpick: 10 rounds of the calibration and credence games isn't enough to conclude very much.
When we've done calibration exercises at our meetups (twice that I know of: once with Wits and Wagers questions, and once with properties of everyday objects), we went for 20-30 rounds, and we also did 50% and 90% confidence intervals for each round.
And we don't have bears playing balalaikas. Well, I would like to tell you about Moscow rationality community after all, not about some B movie featuring crazy Russians.
Moscow community have grown from 5 people on my first meetup to 17 on the last one. And I believe we have possibility to grow even more. Moscow is a big city and it must have many smart people who can start to study rationality.
Our story began in May 2012, when I gathered the people for the first time. Spring meetup announcements hadn't attracted many new members, so we gathered, discussed our site with Russian translations of LessWrong Sequences and made some plans. Our first venue was one of the Subway restaurants in Moscow.
The next milestone in our development was September meetup, when I started to use on-line form to collect information from potential members of our group. Or maybe for some other reason we had got new faces, and even recurring ones. I also told everyone than we should practice rationality skills doing some exercises. Of course we had a lot of theories and ideas to duscuss, but we had to be closer to the real world. That's why we started to practice our rationality skills. We have approximately 8-10 people on each meetup during this fall.
Soon enough this practice yielded good results, new members became heroes and started to improve our ways of training and create new exercises. In January, 2013 one member of our group proposed big and comfortable office room, and we moved there. Our meetups suddenly became more organized and more new members appeared — this year we have 13 people on average.
We have also started to design game that can teach group members some rationality skills. You can find some examples of interesting and fun games in the same guidelines I mentioned, but we want to develop games specially for the skills improvement. Of course even educational games have to produce fun, not only teach you something. We play Liar's dice for relaxation after exercises now.
And you can find some photos from our meetup here.
Appendix: Exercises
Organizer presents two block of questions, each block has 10 questions for the sake of easy results calculation.
In the first one I read questions which require 90% confidence intervals. For example, what is the wingspan of the last model Boeing 737? It is similar to the one from “How to Run a Successful Less Wrong Meetup” guidelines. After this block everyone can calculate their real confidence, for example, if the correct answers are inside your interval in 7 out of 10, your confidence interval is closer to 70% than to 90%. So calibration level still can be improved in this case.
The second block is similar to The Credence Game, it consists of true or false questions. Everyone needs to write down credence for each answer. The average expected credence is calculated, then the real average is calculated. For example, if someone has the following credence: 0.5, 0.7, 0.8, 0.9, 0.7, 0.6, 0.6, 0.8, 0.9, 0.8 the average expected credence will be 0.73. And if there are 6 correct, 3 incorrect and one answer with credence 50%, then it will be 6.5 real average: 0.5 for one answer with 0.5 credence and 6 for the correct answers. The two numbers are close and the person with that answers seems to be well calibrated.
And for the some reason everyone showed better result in the second block. I can conclude that a person has more difficulties with hitting into specified confidence interval than assigning confidence for own answers.
During the calibration session I present the following strategies to improve calibration, once a time:
1. Repetition and feedback. Take several tests in succession, assessing how well you did after each one and attempting to improve your performance in the next one.
2. Equivalent bets. For each estimate, set up the equivalent bet to test if that range or probability really reflects your uncertainty. It means that you should choose between two games. In game A you will receive a money prize if your statement is true, in other words if the correct number is between your upper and lower bounds. In game B you generate random number between 0 and 1, and you win if the random is between zero and your credence (0.9, for example). I can say that you win with probability equals to your credence. If you prefer game A, you may be underconfident; if you prefer game B, you may be overconfident.
3. Consider two pros and two cons. Think of at least two reasons why you should be confident in your assessment and two reasons you could be wrong.
4. Avoid anchoring. Think of range questions as two separate binary questions of the form “Are you 95% certain that the true value is over/under (pick one) the lower/upper (pick one) bound?”
5. Reverse the anchoring effect. Start with extremely wide ranges and narrow them with the “absurdity test” as you eliminate highly unlikely values.
I recommend to make at least one calibration session before any Fermi calculation sessions.
2. Tabooing, version 2
There is standard rationalists' taboo exercise, you just remove some word from your speech and try to talk about something. But I would like to propose another version.
You need to create some texts, or use existing texts from a book or a news article. You also need to find some words in each text that should obscure the meaning therefore tabooing will make it clearer. It may be some texts about politics or other controversial topic. Each text should be short, one or two paragraphs. You need to highlight words to taboo somehow, with italic font, separate colour or highlighter.
At the meetup ask the people to read the text tabooing the words you have selected.
If you are going to try this exercise, please let me know about the results, because I am still trying to improve it.
3. Reframing
First, define or find a statement you want to work with. The statement can be associated with some choice you want to make or it can be your interlocutor's phrase you want to make clear.
Second, do the reframing itself:
Check for you desire to maintain status quo. Do you see changes as bad things? Try to reverse changes direction.
Imagine, that you make a decision for every similar situation in the future.
Change unit of measurement, for example, convert time into money or vice versa.
Change time frame, into the past or the future.
Change the arena. Transfer your conditions into another country, even into imaginary place described in some book.
Imagine, that another person is faced with you issue. What will he or she decide or do?
Imagine yourself as an outside observer. What will you think about your own thoughts and deeds?
4. Value of information
Information can have an influence upon the utility and yield of your choices. If it can help you to make the choice with higher utility, then the difference is the value of the information for you. Take several daily situations when you need to make simple choices and estimate the impact of new information on these choices. For example, you need to buy some things and you may make random choice or look for detailed descriptions and specifications of this things. How much money, time and other resources you can save or earn in the future if you make informative choice instead of choice without the information?