I know. I self-committed to posting the article in Main yesterday but it out-grew a manageable size. I have extracted unneccessary and somewhat unresolved sidetracks esp. those on biases. I will post those parts later when they are ready.
It is downvoted because
And then it is not downvoted that bad. It got 8 up and two down.
I agree with it being moved to Discussions. I should have taken more time with this one. Or else built up to it with more general posts about rational games.
What I think I did right was:
scholarship ok
You seemed to have integrated a lot of links into the post. On the other hand as far as I see you didn't link to empirically validated claims.
Look at a statement like "Classical games with dice or shuffled cards surely build some intuition for probability theory which is present in most games in so far as some most games need some controlled random variables".
As a reader I don't really know what to do with the statements. On the one hand sounds reasonable. But it doesn't tell me anything that I didn't know beforehand.
On the other hand I don't really know whether there are significant effect sizes for the effect you are talking about.
I see you didn't link to empirically validated claims. That is because I didn't find any directly applicable references. So I settled for the LW links. Probably I should have stated this negative find. That might prompted someone to supply them should they be there.
I do have quite a few references about games and teaching concepts - but all of these address games in parenting, computer games, motor skills and literacy/numeracy. None of these are applicable here.
At best these would apply:
Quote from the study: “Providing children from low-income backgrounds with an hour of experience playing board games with consecutively numbered, linearly arranged, equal-size squares improved their knowledge of numerical magnitudes to the point where it was indistinguishable from that of children from uppermiddle-income backgrounds who did not play the games. Playing otherwise identical non-numerical board games did not have this effect.”
This supports the claim that games with clear concepts can quickly convey these concepts (albeit possibly only very simple concepts).
Deals with “the importance of home experiences in children’s acquisition of mathematics”. This supports the claim that concepts present in games significantly improve acquisition of complex concepts (albeit again simple ones).
Look at a statement like "Classical games with dice or shuffled cards surely build some intuition for probability theory which is present in most games in so far as some most games need some controlled random variables".
I wanted to draw the connection between the fact that probabilty theory is neccessary to explain those games theoretically (the "controlled random variable" part) and the ability to actually infer this in a game (the "intuition" part), But rereading it I agree that it comes across as either trivial or meaningless.
The only issue that stuck out to me (I can't tell if there are formatting issues) is that lose and losing were misspelled as loose and loosing.
A (slightly cognitively motivated) defense of Pokemon:
Now, there are two main areas you could be talking about, the card game and the video game. I'm going to focus on the video game, because that's what I'm more familiar with.
This game taught me and gave me an intuitive sense of expected value calculations, the use of standard deviation, and risk aversion. Each move has a certain power and accuracy. So, as an 8 year old, I was dealing with the question of, "Do I use a move with an attack of 90, but an accuracy of 80%, or one with an attack of 70, but 95% accuracy? And if I use a move which decreases their defense by 25%, and does the extra damage I will do later make up for the fact that I will spend one turn not attacking?
Now, the question is whether this is worth it. And the answer is sadly probably not, considering the number of hours I put into this game. Any focused game on teaching these concepts would do a much better job. But, it is not entirely useless.
I agree with this. But then there is always something to be learned. And as you say: You invested lots (!) of hours into it. And most went into attention to trademarks.
From the constant rumors of nonexistent Pokemon, I learned how to distrust the epistemological practices of my peers.
And most went into attention to trademarks.
I think that's a really bad way of thinking about it. It's not like you play 100 hours and 50 hours go into attention to trademarks 10 go into learning math etc.
Every hour that you play does multiple things at the same time.
It depends on what the alternative is. If the alternative is doing nothing or sitting in front of the TV then surely anything learned from a game is better. But here the alternative is to prefer games or in general interactions that have less attention diversion. For children that means playing and talking with them. Directing their attention to lasting topics.
The effect of media and marketing on children is addressed in multiple parenting guide. A scientific inquiry into this can be found here:
http://www.oxfordbibliographies.com/view/document/obo-9780199791231/obo-9780199791231-0006.xml
Discussing what is better wasn't my point. The point is that it's a bad idea to separate out hours as being about A or B. I don't think that's a useful way of thinking about an activity that does multiple things at the same time.
Zendo is often described as "Science: The Game." (More discussion here)
Lots of biases come up. You quickly learn to avoid positive bias if you play this often. You start to deal with confirmation bias and illusory correlation and neglect of sample size. Almost any bias that affects hypothesis generation and testing affects how well you play Zendo, and you can run through single rounds in as little as 10 or 15 minutes. I cannot recommend it enough.
If you're serious about using it for didactic purposes, have players work together, collaborating aloud. This way, you can cover some of the social biases, and have a clearer record of what people were thinking and when they thought it. (If you're really serious, record the play session, and show insight-generating clips as you go. When you play as the master, you get to see these all the time.)
I've wanted to try Zendo since hearing about it here on LW. Is there anywhere it can be played online? The obvious Google searches are failing me.
I like the idea of the players working together, too.
[ETA: A much simpler game that's good rationality training might be Clue. Drawing correct inferences on incomplete information gets surprisingly important if you're playing with reasonably good players, and it seems to me the skill should generalize.]
I'd actually argue that Zendo is simpler than Clue, just less familiar. Specifically, the gameplay mechanics themselves are about as simple as they can be, while still supporting the idea of "be a game of induction".
Is there anywhere it can be played online?
Actually, forums work out pretty well, and chat rooms (IRC, say) work excellently... because you can just use a family of text strings as koan space, instead of physical configurations. It's lacks some visual and tactile satisfaction, but it works online and is free. (Examples: on lw, on Board Game Geek, on the DROD forums.)
I always found Clue a bit pointless. The moving-around-the-board bit gets in the way of the otherwise straightforward logic puzzle.
Same for me. This mixes the clear concept of the logic puzzle with the movement and thereby draws attention away from the logic to other aspects. This reduces the chance that any deeper insights into the logic inferences occur (both by reducing the success of these inferences via the random element and via less concentration available at begin with).
Hrm. Suppose movement were fixed rather than random?
The movement does seem important to the game to me, if only because sometimes (particularly late in the game) you can infer things based on where your opponents appear to be heading. Or you can bluff people attempting to do the same.
Sure. And by doing so it could teach bluffing. But it really the other way around: You already have to know about bluffing to figure out that you could gain a small advantage by doing so. The games rewards such tactics too little. At least if you don't play it more often. And if you play it more often all this moving around and doing a 'straightforward logic puzzle' has long lost any appeal and just eats time. Except when you do it in the background and the socializing around the game moves into the forground. But then any game would do.
I would second Eleusis as a great game for training logical thinking. If you haven't played, at it's core, its basically the 2-4-6 game, with one of the players allowed to make up more complex rules. I've played several times with my friends, and you would be amazed at how difficult it is to tease out even some of the simpler rules. For instance, I once played a game, where a player went through almost 2 decks of cards before realizing the rule was "Alternate Red/Black".
Here is a simple game that requires only pen and paper. I've often played it with my young son and I think it helps teach some rationality skills.
Player 1 keeps writing down a different set of three numbers, and for each such set Player 2 says either "Yes" or "No" depending on whether the three numbers conform to a secret rule that Player 2 creates. The rule must be such that the average child in grade X (where X is specified) could easily apply it. If Player 1 keeps getting the same reply to different sets of numbers he can request a counterexample that would elicit the opposite reply. Player 1 eventually tries to guess the rule.
Possible rules: the numbers sum to >20, at least one number has 3 digits, all the numbers are odd, the second number is the biggest,...
This is Zendo, but with numbers instead of shapes. An analogous game can be played with words, e.g.:
Positive examples (these fit a particular rule)
Negative examples (these don't fit the rule):
Solution (rot13): Jbeqf gung raq jvgu gur anzr bs na navzny.
A related game I played with my 7 year old asks for all numbers included in a set. It is an extension of the game where you have to guess a single number and the child learns interval halving. He grew out of that some time ago. I didn't yet leave the natural numbers. As with most question and answer games we change roles. Changing roles has the advantage that there is no "teachers password" effect. And the child has to check the rule more often. After a few dozen rounds he was able to find "1, 3, 8". "all numbers > 0" "all divisible by 10 between -100 and +100", And he invented "only -100", "all divisible by 5 between -100 and 0". The advantage is that you don't need paper and pencil but can do this verbally.
The estimation game you describe sounds a lot like the party game Wits and Wagers, though with the added challenge of predicting what the other players may predict as well.
I like the idea behind your game though. One way you may be able to help it teach to calibrate your own confidence intervals is to have everyone also guess an X% confidence range for their guess (whatever you decide is a good range). Then, each time the answer falls within that range, award the player P points, and whenever it is outside the range, penalize them P/(1-X) points (e.g. confidence of 90%, give 1 point for correct range and -10 for incorrect). To keep the ranges tight, offer another bonus to whoever had the tightest correct bounds.
I discovered that Wits and Wagers was actually discussed here: http://lesswrong.com/lw/14u/wits_and_wagers/
I tried it out and it is much easier to play than my game and thus is somewhat more fun. But it also has less insights.
My experience is as follows:
It really is optimized for playability. I think it does some calibration of (over)confidence and it builds intuition for probability and risk-trade-offs.
But - and that is my main point - it doesn't have clear concepts. The concepts are all mingled up, skewed, hidden. You may gain intuition but it will not help you toward overcoming e.g. overconfidence bias or egocentric bias.
I still think for that the concept must be sufficiently present to be able to reflect and consciously use it.
Indeed. I wasn't aware of http://en.wikipedia.org/wiki/Wits_and_Wagers. There is no German match for it (yet). I actually considered using a confidence rating but discarded it because it was too easy to game the rules for it - but that was during the first test games that were about estimating your own performance via multiple estimates (called 'strict variant' in the post). When I switched to estimating the range of the other players guesses I didn't reconsider confidence ranges. Could be an interesting variant.
The current gameplay is really simple even though it includes a min-max range. There was a earlier and more complex version that had a 90% range (or more precisely an 1-1/N range) which earned N points if it excluded exactly 1 guess, 1 if it excluded none and 0 if more than 1. That was OK but didn't gain much. The most surprises were still registered by the min-max range.
See also this thread.
The group that spun out of that stalled very quickly, but I believe some members (including myself, certainly) are still interested in continuing the project. If anyone else wants to step in, perhaps we can revitalize it.
Just yesterday, my supervisor accepted my initial Master's thesis description, which basically says that I intend to implement that game as my thesis. So I should be finally getting back to it.
If you want a community that values friendliness and successfully uses social glue, then you should be a role model of that.
Moving some newbies post around without even a friendly note or helpful explanation could be seen as rude or at least as generally acceptable behavior.
I didn't take offense. I already noticed that you are efficient and driven by your goals and are still coming to terms with happiness. There must be leaders.
I wonder about 'still coping with happiness'. You mean you have passed beyond that? What is that like?
Damn. "to cope" doesn't mean what I thought it means. I meant "come to terms". And I wrote that because I saw a quote by him where he said exactly this. Alas I can't find it. I read a lot of his postings on happiness and the result is: Either he didn't say it or he said it a long time ago or he said something slightly differnt. Either way I'm wrong and I realized that I should't have said it in the first place. At least not publicely. I'm sorry.
The 'Estimation Game' sounds like a more complicated/formalized version of Wits and Wagers. The game has very similar questions to the ones you wrote; each player guesses an answer, then all guesses are revealed at the same time and players bet chips, at varying odds, on which guess(es) they think are closest to the actual number.
Sure. It is more formalized. That's because it intends to make the concept clear while still being easy (the explanation looks complicated but the game is not). Wits and Wagers seems to skew the bets to one side which obscures the concept. And Wits and Wagers has no differentiation between min-max range and majority-range.
But I already see that I do not get around to trying out Wits and Wagers. There are always some game mechanics one can learn from.
I highly recommend Hanabi. It's a cooperative game about common knowledge.
Rules are here: http://www.cocktailgames.com/upload/produit/hanabi/regles/regle_en_hanabi.pdf
In my experience Werewolf is a great game for skill building.
In the game you 8-24 players. At the beginning of the game every player gets a card that defines his role.
Every round of the game the living players discuss which player they believe to be a Werewolf and want to kill. Afterwards all living players vote to kill on of the player. Afterwards everybody closes his eyes and the players with the Werewolf role awake and open their eyes. Together they choose one of the non-Werewolves to die that night.
The game either ends when the Werewolves win and kill all the non-Werewolves or when all Werewolves are killed. For added complexity there are additional roles among the non-Werewolves with special abilities.
The game teaches you to read bodylanguage of other players. Is a player nervous because he wants to hide that he's a Werewolf? If you are a Werewolf you learn to lie convincingly. The game lives from constantly making evaluations about which players you can trust and whom you can't.
Yeah. I know that.game. It is a game that integrates many useful aspects - social skills, reasoning, reading body language, conforming to roles. It has no clear focus on a limited set of concepts but maybe a game packed with many concepts or skill areas does as well.
It has no clear focus on a limited set of concepts but maybe a game packed with many concepts or skill areas does as well.
Why should a game with focuses on fewer concepts be better than one that teaches multiple concepts at the same time?
Why should a game which focuses on fewer concepts be better than one that teaches multiple concepts at the same time?
If a game really teaches multiple concepts as might be in the Werewolf case there is no need to limit the game unneccessarily. If the concepts present in a game are not sufficiently clear or separated you may get some intuition about the concepts (system 1 knowlege in dual process theory*) but may fail to reflect and understand them (system 2). it may be that the concepts that are arguably present (for a knowledgable observer) fail to register for the player/student because:
The result of 2) and 3) is that the effect of the concepts lacks reproducibility which makes it diffcult to detect. Sure if you play the game long enough or at sufficiently competetive levels you will not get around to learning and understanding these concepts too.
But my focus is not only on games which pave the ground for concepts by building intuition for them. My objective it to actually teach the concepts (which requires build intuition for and reliable detection of the concepts). And not by playing the game to excess but as quickly as possible (more or less) while still being fun.
Remember that repeatability is a core part of the scientific method and having games allowing to successfully apply that tool (within the time and scope of the game) may in itself be an argument in favour of clear concepts.
If a game really teaches multiple concepts as might be in the Werewolf case there is no need to limit the game unneccessarily. If the concepts present in a game are not sufficiently clear or separated
Don't look at the number of concepts that are taught but how well the concepts are taught.
This might be about the same bias that came up when you said that a lot of hours of playing pokemon are about trademarks. The world just doesn't work that way. Multiple things happen at the same time.
get some intuition about the concepts (system 1 knowlege in dual process theory*) but may fail to reflect and understand them (system 2).
For most practical issues the intuition is much more valuable than having a mental concept.
Remember that repeatability is a core part of the scientific method and having games allowing to successfully apply that tool (within the time and scope of the game) may in itself be an argument in favour of clear concepts.
Calling for repeatability in a game means basically that you don't want that players that are in the game use their specific knowledge that they gathered outside of the game to teach that knowledge to other players.
I don't think that's a worthwhile goal. Especially when it comes to social games. I don't think that the benefit of enabling better use of the scientific method is worth that trade off.
For most practical issues the intuition is much more valuable than having a mental concept.
That is correct in general and for most games. But my main point was exactly that you also should teach the concepts - at least if you want to advance rationality. So here and for this goal it is more valuable.
I don't think that the benefit of enabling better use of the scientific method is worth that trade off.
The tradeoff with what? With supporting socializing? Again that is not the focus of this post.
That is correct in general and for most games. But my main point was exactly that you also should teach the concepts - at least if you want to advance rationality. So here and for this goal it is more valuable.
Take the problem of doctors interpreting medical results.
Who's more rational:
1) A doctor who doesn't know what the context of Bayes rule means but who has an intuition that allows him to be right about the probabilities of a treatment working with a given patient.
2) A doctor who can tell you what Bayes rule is and write down the numbers but who doesn't have the right intuition to be right about the probability when it comes to estamating which treatment works for a patient.
In the sense in which rationality is defined here, rationality is about picking winning strategies. Doctor 1) wins. He's the rational guy.
The tradeoff with what? With supporting socializing? Again that is not the focus of this post.
If I play Werewolf and someone else engages into bodylanguage that makes me conclude he's a Werewolf, I have to explain to my fellow players why that bodylanguage indicates that he's lying.
This means that I bring information about reading bodylanguage into the Werewolf game that I learned outside of it. As a result you lose repeatability.
Playing the game with different players has a different effect.
Ah. OK. Now I see your point clearly. You mean the 'high level' rationality in focus on LW. I agree that that is indeed not trained explicitly by the games I have in mind.
But then the question is: Are there games that address that kind of hight level rationality? Or does any game or rather a suitably high number of games or the extreme immersion in the game qualify?
It seems that my reference to rationality in the LW sense taken literally makes your argument justified.
Would you agree that the following goal would better match the game objectives I have described?:
"Games that train specific concepts helpful in understanding and reasoning about cognition, complex human behavior and strategies to aquired even more compelx concepts."
I'd take it that you'd prefer a Doctor who can do 1) and 2) to one that does only 1). And that is what I have in mind.
An interesting take is to have a game where programming is an integral part of solving the puzzles.
I played a game over the weekend that seemed pretty rationalist.
There were about 6 of us, and each person had to write a famous person (living, dead, or fictional) on a piece of paper and pass it to another person. That person then had to put that paper on their forehead and try to find out who they were by asking a yes or no question (e.g. "Am I a female?", "Was I alive in the past 50 years?").
I bet a game like this could be modified further to make it more rationalist.
Follow-up to: Fun and Games with Cognitive Biases
Related to: Rationality Games & Apps Brainstorming, Rationality and Video Games
Answering: Designing serious games - a request for help
Games on Lesswrong
Rational games or game ideas have been discussed or mentioned on lesswrong a few times:
Designing serious games - a request for help proposes to apply games for the rational cause. Fun and Games with Cognitive Biases plays (on a meta level) with concepts. Rationality Games & Apps Brainstorming urges for such games. Rationality and Video Games provides an example. Taboo is explicitly mentioned in http://lesswrong.com/lw/nu/taboo_your_words/. Doing this with scientific concepts could make for a nice casual game at a LW meetup. http://lesswrong.com/lw/3h/why_our_kind_cant_cooperate/ generallly urges for community building mechanism – and (social) games address this. A game that has been discussed extensively on lesswrong is Diplomacy
Introduction
I like to play games that are challenging and intellectually demanding. But I don't like to learn games that will gain nothing except the fun of learning and playing the game.
Games for rationalists should not only be fun. They should also train or teach something beyond the game. Games that don't are parasitic memes.
This post describe areas of rational thought addressed by games. By rationality I mean the LW view.
I list games suitable for getting acquainted with rational thinking. I list areas underrepresented by games and I give an elaborate example of a game suitable to such an area. I propose trying to invent games for these areas and sketch possible strategies.
To repeat I don't mean games that are just played frequently by rationalists or usually considered entertaining by rationalists (though they may). I mean games to train rationalists. Serious Games. To make it easy and fun to become a rationalist (or at least support such a cause).
Games supporting the acquisition of literacy and numeracy are out of scope here even though these are preconditions for more demanding concepts (and games). Also mostly out of scope are social and activity games (though these are be relevant to efficient group building) and games with a focus on motor ability (though such may make spatial or dynamical concepts more clear).
Outline
Using and Abusing Curiosity
To use natural curiosity and to direct it to 'worthy' objectives has got the newfangled name edutainment. Well known examples from TV are Sesame Street and Mammutland (which is based on The Way Things Work. Examples of educational games are given below.
The opposite, where natural curiosity is diverted into cognitive dead ends and where attention is collected in trademarked walled gardens is called Marketing. An example is Pokemon where the cute animals engage the curiosity of children which learn lots of names and attributes – but none of these have any lasting cognitive use and all the accumulated knowledge can primarily be used for status and thus directing attention to the trademark owner in the end.
Rational parents have to counter marketing when educating their children. Competing with media is hard because it is omnipresent and not in the best interest of education. Possibly we should lobby against maleducation but that will not help now. Instead we have to turn to games competetive with Pokemon.
Educational Game Theory
Educational Game Theory names three approaches to educational games:
creation from scratch by educators
integrating off-the-shelf games
creating from scratch by the players (e.g. children)
An example of the first is Ökolopoly (English Ecopolicy) which has a strong focus on grand picture of complex systems. I'd really recommend it for children grade 6 and up (but use the card board version as the simple 'game mechanics' (connected lookup-tables) are inspectable there).
I will provide an example which I invented and tested myself later.
Examples of the second kind are e.g. playing Taboo with math words or playing Liar's Dice and calculating expected values. Much more will be listed below with a focus on specific concepts.
That children invent games all the time is no surprise. Because one natural aspect of games is that they support or stimulate learning most games invented by children are educational by nature.
To illustrate this I'd like to describe a game my oldest son (9 years) invented:
He calls it 3D-computer and it consists of a 'user interface' made of paper and plastics, a number of 'input devices' and a large playing area made of taped together sheets of paper with a plan of a city. The player (usually one of his younger brothers) sits at the 'controls' and e.g. steers a car on a racing track thru the city. He has to press left and right (and say so loud) and my oldest son will act out the operation of the 3D-computer by moving the cars accordingly.
Obviously this is modeled after real computer games but it requires significantly more abstraction and cooperation of the players. To build the 'game' he had to plan it and build corresponding pieces. He created a 'user interface' consisting of menues and sub menues (drawn on paper) for the possible games and options that can be played with the 3D-computer. It makes the logic of the game – its concepts – clear both in the planning and realization and in the acting too.
Concepts in Games
Concepts in Games
Any normal game requires some rational thinking, but there are some areas of rational thought that are less covered by games than others.
Game Theory
This lends itself naturally to games with clear concepts. The simplified games analysed in game theory can be easily readapted to playable games or parts thereof. Some of these are directly playable for school children. I played lots of these in a math course.
Game theoy obviously applies to most games, but the basic min-max principle is very clearly present in games that have a measurable of advantage; I identify the following kinds
In these cases there may be strategic effects that outweigh the measurable but in the long run 'more is better' and a higher value predicts a win well.
Probability Theory
Classical games with dice or shuffled cards surely build some intuition for probability theory which is present in most games in so far as some most games need some controlled random variables (aka dice or shuffled cards) to support the game semantics.
It is also present when simulating (or testing) games; the game state changes due to the game rules are implied stochastic processes.
Probability theory is relevant as games with numerous regular random events satisfy criteria of theorems of large number.
Examples for dice games with simple rules and clear concepts are Cross And Circle, Snake and Ladders. These illustrate basic concepts and reflection about strategies using solid terminology can teach children a lot about probability.
Intuition about different distributions resulting from superposition and their likelihood is fairly clearly present in Yahtzee.
Intuition about the law of large numbers (and updating due to new information) can be found in an entertaining way in Liar's dice.
Anecdote: As a child I was a sore loser. Seems that at least one board flew through the room. Not that I'd lose that often, quite the contrary, but if...
I couldn't deal with chance playing tricks on me. Later I always tried to limit the effect of the dice. Both by hedging or by just changing the game rules beforehand. Once I printed an 'improved' list dice throws that were overly regular.
Decision theory
Games that are not based on skill necessarily involve decisions by the players. These fall into three categories
Decisions under certainty – games are seldom certain, but in some cases decisions can be modeled as if the game environment were fixed and then maximizing over multidimensional ratings of the game state (some measurables like win points)
Decisions under risk – when some decisions involve probabilistic losses or gains
Clear decision strategies or concepts are not so easy come by. Most games imply lots of decisions but effective strategies are seldom obvious.
Games where subjective valuations are rampant are trading cards (e.g. Pokemon or currently Star Wars) these are no simple trades but often involve a risk (gaining certain cards vs. losing some other cards due to a random or skill based process). What is unclear is whether there are any lessons learned from doing these games as the decisions are mostly individual and the conditions of the transactions vary from trade to trade so general insights are unlikely.
Cognitive Science
Cognitive biases can be harvested for game aspects worth to address. I will demonstrate this with the Overconfidence Effect and with miscalibration of probabilities particular which is the main focus of the 'estimation game' presented below. I will cover other biases in more detail in a separate post.
Humans are very good at detecting cheaters this is well known since the work of Cosmides and Tooby.This applies to real life as well as games. It would be interesting to develop games that try to move smoothly from obvious rules to abstract rules and try to train to perceive this as cheating.
Math and Logic
Math in general plays a role in most games. Correct logical inferences (even if done by intuition) train propositional logic (though a game which involves using modus ponens on unusual conditions would surely be a good idea).
A very nice and pure example using Set Theory even in its name is Set.
Looneylabs has a lot of games (or game material) suitable for logic. A very nice game building on this is Zendo.
Estimation Game 'Who guesses best'
Now I'd like to present an example of a game which was created to explicitly addresses a cognitive bias – the overconfidence bias – which is hard to overcome even for scientifically schooled persons explicitly briefed in this bias.
When I read chapter 21 by M. Alpertand H. Raiffa in Judgement under uncertainty - heuristics and biases 1982 by Kahneman et al (pages 294 – 305) see this Google Books link and this less wrong review.
I couldn't believe that it could be that hard to calibrate well and I decided to test this on myself and on others in a comparable setting (kind of privately reproducing the study). This developed into a game that was since played about 5 times with a total of about 20 person and 30 questions (thus much less than in the cited study). The results were as expected – basically. A significant trend was visible in the test games and also in the main game rounds during a large birthday party. I had improved the 'game chart' and scoring rules and this made the effect clearer and the calibration of most players improved quickly. The game is simple enough to be played by a smart eight year old.
The game chart can be downloaded here (on page 2 in english).
Estimation Game Rules
This is basically an estimation game – but with some extensions.
The Questions
First a number of questions about quantities to guess are needed.
Each quantity should have an exact value that has a believable source but is somewhat unusual such that most players will not know the value.
In a scholarly context the question should name a study or method and date of determination of the quantity. E.g. for the question “How many residents live in Hamburg, Germany?” the context could be: “from wikipedia: official census of 2012 determined via population register”. And for the trivial question “How many lentils are in this jar?” the context could be: “This morning I filled the jar with green lentils, weight the content, counted and weighed a sample and calculated the total number.”
Example questions:
Number of steps to the next underground station.
Number of lentils in a jar.
Amount of rain per square meter per year in Jerusalem in 2001.
Number of files on my PC as reported by ls -R this morning.
How long (days) was the first voyage of James cook into the south seas?
Total egg production in th U.S. In 1965. (this is from the Alpert Study)
Toll collections of the Panama Canal in 1967. (dito)
The questions can be provided by the host (which has the disadvantage that host cannot take part in the game for most questions). The better idea is to have each player provide one to three questions.
Guesstimation
Each question is now dealt with as follows:
The question is read out loud. If needed, detail questions about the context of the question are answered.
All players individually guesstimate the true value of the quantity in question. The number is written into the field in the middle below the green area. If the number is from one of the players he/she writes the correct number as his 'guess'.
Each player now considers what the other players might estimate. Values deemed to be definitely out of the range become the min/max values and written below the red/yellow border. Values deemed to be typical form the majority range and are written below the yellow/green border.
Points are awarded as follows:
For the best guess you get N points, for the next-best N/2, then N/4 (rounding down).... If multiple players are equally near, points are added and split. Numbers supplied by a players are excluded from scoring.
For the green area you get N/2 points if it contains exactly N/2 numbers (including your own) for each more or less you get one point less (rounding up).
For the yellow area the player with the smallest range which still contains all guesses gets N+1 points, the next-best N/2+1 (and so on, rounding down).
Numbers falling on a boundary are scored to the players advantage.
Example:
For the number of lentils in the jar you estimate 1300. And you guess that half of the guesses fall within 500 and 2000. You also assume that nobody will believe less then 100 or more than 10000.
It turns out that the other 6 players guessed 750, 1000, 1500, 2000, 2820, 4000.
You write 750, 1000, 1500 and 2000 into your green area. You write 2800 and 4000 into the right yellow area. Congratulations: No numbers fell into a red area.
The correct answer is revealed: 2800. You can circle it in the green area.
The player guessing 2000 (distance 820) gets 7 points, the player guessing 4000 (distance 1180) gets 3 points floor(7/2), the player guessing 1500 (distance 1320) gets 1 point floor(7/4). For your 1300 you and the remaining players get nothing (7/8<1).
You have got 4 numbers in your green area (you decide to include 2000). 4=ceil(7/2) is the optimum number of entries so you get 4 points. Had you excluded 2000 you'd got 3 points. If only your own number or if all 7 numbers were included you'd got 1 booby point.
All the numbers are in your min-max-range, so you get at least 1 point. Lets assume that your range of 9900 is the second smallest of those containing all numbers. This nets you an additional floor(7/2)=3 points.
May sound complicated. A summary of these rules is part of the game chart. The chart is mostly self-explanatory and even school children can fill-out the chart after one round of explanation.
Gaming the Rules
It is possible to game the rules by e.g. using absurd guesses to bomb the max-range of the other players. The scoring is chosen to make this a losing strategy. Good guesses and good majority bounds score higher than the points reaped from being the only one with a valid max-range. And players can hedge against bombing nonetheless (in a way these are black swan events and thus interesting in their own right).
Not quite Overconfidence
This game doesn't directly address the Overconfidence Bias. To do so the players would have to guess the range of their own guesses. Getting the range of ones own guesses via this kind of game takes much longer (times the number of players). The game works so well because it forces you to take the outside view. You have to consider what the other players might not know and then getting immediate feedback about ones own performance via
a) Distance from the truth of your own guess.
b) In extreme cases exclamation from the other players whose ranges were shredded.
It is critical to make the leap that your own guesses are as (over)confident as those of the others.
Variants
It is quite possible to play a stricter version where you guess your own ranges with the same game chart. The following differences are necessary:
Instead of the majority range you have to guess the range into which the true value will fall mostly. Mostly meaning on half of all questions this range should actually contain the value.
Instead of the maximum range you have to guess the range into which the true value will always fall.
Don't write the numbers of the other players into the colored area.
Use a singe color strip for all of your own guesses. Just make a mark in the area where the actual value lay in the end.
Points are awarded only at the end of the game.
This has the disadvantages of taking much longer and with the above simple rules it is too easy to game the rules (e.g. by using absurdly large or small ranges at the end of the game to ensure the required counts).
But this can be used after a few games to test whether you can make the leap to take the outside view on your own guesses. It can be done alone. Just think of a few arbitrary quantities and then research them later (on danger of choosing only quantities you can provide sensible estimates for).
Anchoring
Obviously your ranges are anchored to your guess. It is possible to add a small tweak to change the anchoring (disclaimer: I didn't try this): Allow the person knowing the correct answer to supply any example number and say it out load beforehand. This will provide another anchor (obviously anchored to the correct number somehow).
Motivation
The test-games were fun. OK. The game may not be fun for everybody. My acquaintances are mostly smart and well-educated. But even my 8 year old son liked it (and scored in the middle range of 10 players). It is fun especially if people come with their own individual questions. The resulting discussions about individual (mis)reasoning is also often insightful and a nice ice-breaker.
The performance of the players definitely improved.
Friends who are teachers asked me for the material and used it in highschool.
I hope this is is step into the right direction.
I'd like to close with an anecdote: I still try to win any game that I play and my friends know it. Sometimes, especially after I play unusual or when explaining tricks and then losing anyway I get remarks about it obviously not helping. To that my reply mostly is: I maximize my chances on infinitely many runs. So in the end I learned to lose.