Consider the following statement: two heads are better than one.

It seems obvious to me that several rationalists working together can, effectively, bring more precious brainpower to bear on a problem, than one rationalist working alone (otherwise, what would be the point of having a Less Wrong forum community? You might as well just leave it as a curated community blog and excise the discussion forums.). Further, due to various efforts (HPMOR especially) it appears that LW is inevitably growing. This makes it not only desirable to find ways to effectively get groups of rationalists to think together, but also increasingly necessary.

It is also desirable that methods of getting groups to think should be feasibly doable over the Internet. (I am aware that real-life meetups and stuff exist, but please be reminded that some people in the world do have to live in shitty little third-world countries and might not at all find it economically feasible to go to first-world countries with atrociously high costs-of-living)

So first, let us start with the current "best methods" of getting groups of Traditional Rationalists to coordinate and think, while avoiding groupthink effects that diminish our aggregate rationality. Hopefully, we can then use it as the basis of part of the art of rationalist group thinking. So I'll discuss:

• Disputation Arenas - procedures with centrifugal and centripetal phases
  1. Delphi Method - members secretly answer questions, non-member summarizer anonymizes answers, members read anonymized summary, repeat
  2. Prediction Market - members place stock bets on market, market settles on price, members react to price signal, repeat
  3. Nominal Group Technique - members write down ideas privately, non-member facilitator guides members in sharing ideas, non-member facilitator guides members in paring down and cleaning up ideas, members vote
• Conclusion - strengths and weaknesses of the various disputation arenas shown here

Disputation Arenas

One of my favorite SF authors, David Brin, talks a bit about what he calls "disputation arenas". I won't discuss his ideas here, since his concept of "disputation arena" is actually a relatively "new", raw, and relatively untested procedure - what I intend to discuss for now are things that have at least been studied more rigorously than just a bunch of blog posts (or personal website pages, whatever).

However, I do want to co-opt the term "Disputation Arena" for any process that tries to achieve the following:

• Avoid groupthink - actively search for information without settling too early on an option considered desirable by certain influential members
• Achieve consensus - designate some choice as the best, given current information known by the group members

We want our group rationality process to avoid groupthink (possibly at some expense of efficiency) because actual, real-world rationalists are not perfect Bayesian reasoners - two words: Robert Aumann. Because rationalists are not perfect, we do not expect a clear consensus to form after the end of the process (i.e. Aumann's does not necessarily apply), so the process we use must force some consensus to become visible.

One thing that David Brin discusses is the general division of the procedures into two "phases":

• Centrifugal phase - members of the group generate ideas separately.
• Centripetal phase - ideas are judged by the group together.

This seems to me to be a good way of labeling parts of any group-coordination process that attempts to avoid groupthink and achieve consensus.

In my personal research, I've found three things that attempt to achieve those two goals (avoid groupthink, achieve consensus) and which might (perhaps with a stretch) be considered as approximately having two phases (centrifugal and centripetal).

Delphi Method

Wikipedia:Delphi method

The Delphi Method was originally developed by RAND Corporation (Project RAND at that time, and no relation to Ayn Rand) in order to get better predictions on future military hardware. It is currently used to get better utilization of current human wetware. (^.^)v

Delphi Method: How To Do It!

Pen and paper version:

1. A panel of experts is chosen, and a questionnaire is prepared.
2. Experts answer the questionnaire, giving the answers and also justifications and reasons for those answers.
3. Summarizer provides anonymous summaries of the expert's answers and justifications.
4. Experts read the summary, and may revise their answers/justifications. The process is repeated (with the same experts and questionnaire) for a set number of rounds, or until everyone gets bored, or the military bunker everyone is in gets nuked.

Internet version (Wikipedia:Real-time Delphi):

1. Username/passwords are chosen and emailed, and a questionnaire is prepared. The questionnaire in the online version is somewhat restricted, however. Here are some ideas:
   • Use a "poll" question. Experts select one of the choices given.
   • Use "multiple-choice" questions. Experts then select from a range of (e.g.) 1 for (strongly disagree) to 10 (strongly agree) for each possible choice.
   • Use "multiple-choice" questions, and also give different aspects such as "feasibility", "desirability", "good side-effects", etc. Experts answer 1 to 10 for each combination of choice and aspect.
2. Experts answer the online questionnaire. Aside from the numerical or selection (quantitative) answer, experts should also supply a short sentence or two justifying each answer (qualitative).
3. After the expert submits his or her answers, he or she is shown the current averages (for scoring-type questionnaires) or current poll results - this is the quantitative group answer. Expert is also shown (or provided links to) randomly-sorted (and randomly-chosen, if groups are very large and the number of answers may overwhelm a typical human) qualitative answers for each poll item / choice / choice+aspect - the qualitative group answers - for each score or aspect.
   • IMPORTANT: individual qualitative answers should not show the username of the expert who gave them!
   • In effect, the average (or poll results) plus the randomized sample of qualitative answers are a simple, anonymous, machine-generated summary.
4. Experts may change their own quantitative and/or qualitative answers at any time, and see the current quantitative group answers and qualitative group answers at any time after they have submitted their own answers.
5. The questionnaire is kept open until some specified time, or somebody hacks the server to put LOLcats instead.

Delphi Method: Analysis

Delphi methods avoid groupthink largely by anonymity: this avoids the bandwagon effect, the halo effect, and the mind-killer. Anonymity and constant feedback also encourage people to revise their positions in light of new information from their peers (by reducing consistency pressure): in non-anonymous face-to-face meetings, people tend to stick to their previously stated opinions, and to conform to the meeting leader(s) or their own bosses in the meeting. A lot of those effects is reduced by anonymity. Pen-and-paper form makes anonymity much easier, since the summary gets the tone and language patterns of the summarizer; some amount of anonymity is lost in the online version (since language patterns might theoretically be analyzed) but hopefully the small sample size (just a short sentence or two) can make language pattern analysis difficult. Note that randomizing the order of the comments in the online version is important, as this reduces the effects of anchoring; sorting by time or karma may increase groupthink due to anchoring on earlier comments, but if each expert sees different "first comments", then this bias gets randomized (hopefully into irrelevancy).

Delphi methods achieve consensus by the summary (which often serves as the "final output" when the process is finished). Arguably, the pen-and-paper version is better at achieving consensus due to the "turn-based" arrival of the summary, which makes the expert pay more attention to the summary, compared to the real-time online system.

The Delphi method's centrifugal phase is the expert's private answering of the questionnaire: each expert makes this decision, and provides the justification, without other's knowledge or help.

The Delphi method's centripetal phase is the act of summarizing, and having the experts read the summary.

Delphi Method: Other Thoughts

I think that forum polls, in general, can be easily adapted into online real-time Delphis by adding the following:

• Members will be required to give a short sentence or two (probably limited to say 200 chars or so) justifying their poll choice.
• Members should be allowed to change their poll choice and their justification at any time until the poll closes or the forum is hacked by LOLcat vandals.
• Members should be able to click on a poll choice on the poll results page to get a random anonymous sampling of the justifications that the other members have made in choosing that poll choice.

The procedure says "experts" but I think that in something more democratic than the military you're supposed to read that as "anyone who bothers to participate".

Prediction Market

Wikipedia:Prediction market

Prediction markets are speculation markets built around bets on what things will happen in the future. They are also the core of Robin Hanson's Futarchy, and which you can see somewhere in the background of LW's favorite tentacle alien porn novella (O.o);;.

Prediction Market: How To Do It!

Pen and paper version:

1. Convince a trusted monetary institution to sell you "X is gonna happen" stock and "X is not gonna happen" stock for $1 a pair (i.e. $1 for a pair of contracts, one that says "If X happens, Monetary Institution pays you $1" and another that says "If X doesn't happen, Monetary Institution pays you $1", so the pair costs $1 total since X can't both happen and not happen). You may need to pay some sort of additional fee or something if the monetary institution is for-profit.
2. Sell the stock (i.e. the contract) you think is false for as high as you can get on the open market. Buy more stock of what you think is true from others who are willing to sell to you.
3. Just buy and sell stocks depending on what you think is the best prediction, based on what you hear on the news, gossip you hear from neighbors, and predictions from the tea leaves. Keep doing this until X definitely happens or X definitely does not happen (in which case you cash in your stock contracts, if you bet correctly on what happened), or a market crash results because someone discovers that the weak nuclear force actually allows you to make nuclear bombs out of orange juice, and Einstein and the gang were lying about it and distracting you by talking about dice-playing gods.

(what I described above is the simplest and most basic form I found; refer to the Wikipedia article for better elaborations)

Internet version:

1. Hack Intrade so that the topic you want to bet on is in their list of markets. Or better yet just hack Intrade and put one million dollars into your account.

Prediction Market: Analysis

Prediction markets avoid groupthink by utilizing the invisible hand. Someone selling you a stock might be an idiot who can't read the tea leaves properly. Or the seller might have knowledge you do not possess, so maybe buying the stock wasn't such a good idea after all? Remember: if you can't find who the sucker on the table is, that sucker is you!! You can't simply assume that what your neighbor says is true and you should sell as many stock of X as possible: maybe he or she is trying to take advantage of you to get your hard-earned cash. Groupthink in such a mistrusting environment gets hard to sustain. Prediction markets work better with very large groups of people, so that you get practical anonymity (although not perfect, in theory you or anyone else can keep track of who's selling to who; online versions are also likely to hide user identities). Anonymity in the prediction market also has the advantages previously described under Delphi Method above.

Prediction markets achieve consensus by utilizing the invisible hand. The price point of any sale serves as an approximate judgment of the epistemic probability of X occurring (or not occurring, depending on the contract that got sold). This gives a real-time signal on what the group of traders as a whole think the probability of X occurring is.

The prediction market's centrifugal phase is each individual trader's thought process as he or she considers whether to buy or sell stock, and at what price.

The prediction market's centripetal phase is any actual sale at any actual price point.

Prediction Market: Other Thoughts

Prediction markets are well-represented online; Intrade is just one of the more famous online prediction markets. Prediction markets appear to be the most popular and widely-known of the disputation arenas I've researched. These all tend to suggest that prediction markets are one of the better disputation arenas - but then remember that the Internet itself has no protection against groupthink.

Nominal Group Technique

Wikipedia: Nominal group technique

Nominal group technique is a group decision-making process, appropriate for groups of many different sizes. This procedure's pen-and-paper form is faster than the pen-and-paper forms of the other disputation arenas discussed here.

Nominal Group Technique: How To Do It!

Pen and paper version:

1. The facilitator informs the group of the issue to be discussed.
2. Silent idea generation: group members are provided pen and paper, and are told to write down all ideas they can think of about the issue on the paper. They are given a fixed amount of time to do this (usually 10 to 15 minutes).
   • IMPORTANT: members are not allowed to discuss, show, or otherwise share their ideas with others during this stage. There's a reason it's called "silent".
3. Idea sharing: the facilitator asks group members, one at a time, to discuss their own ideas, until all members have shared their ideas. The facilitator writes the shared ideas into a whiteboard, or a similar location visible to all members.
   • IMPORTANT: debate is not allowed at this stage; only one member at a time can speak at this stage.
   • Group members may also add additional ideas and notes to their written-down ideas while waiting for their turn.
4. Group discussion: members may ask for clarification or further details about particular ideas shared by other group members. The group may agree to split ideas, or merge ideas, or group ideas into categories.
   • IMPORTANT: the facilitator must ensure that (1) all members participate, and (2) no single idea gets too much attention (i.e. all ideas must be discussed).
   • The discussion should be as neutral as possible, avoiding judgment or criticism.
   • The final result of this stage should be a set of options to be chosen among.
5. Ranking: members secretly rank the options from 1 (best) to N (worst), where N is the number of options generated in the previous stage. The facilitator then tallies the (anonymous) rankings (by adding the rankings for each option) and declares the option with the lowest total as the group consensus.

Unlike the previous procedures, which have been extrapolated into Internet versions, there is currently no Internet version of nominal group technique.

Nominal Group Technique: Analysis

Nominal group techniques avoid groupthink by the two "secret" steps: silent idea generation, and the secret ranking. Having members write down their ideas in the silent idea generation step helps them precommit to those ideas in the idea sharing step, even though more influential group members may present opposite or incompatible ideas. Although the group discussion step disallows explicit criticism of ideas, those criticisms are implicitly expressed during the secret ranking step (i.e. if you have a criticism of an idea, then you should rank it lower).

Nominal group techniques achieve consensus by the idea sharing, group discussion, and ranking steps. In particular, tallying of option rankings is the final consensus-achieving step.

The nominal group technique's centrifugal phase is largely the silent idea generation step, and is the most explicit centrifugal phase among the disputation arenas discussed here.

The nominal group technique's centripetal phase is largely the rest of the procedure.

Nominal Group Technique: Other Thoughts

A modified form of nominal group technique eats up a quarter of my recently-finished novel, Judge on a Boat, which I talked about on LessWrong here, and whose latest raw text source you can read online. Yes, this entire article is just a self-serving advertisement to garner interest in my novel o(^.^o)(o^.^)o.

Compared to the other procedures here, nominal group technique is more complicated and much more dependent on the centralized facilitator; the extreme dependency on the facilitator makes it difficult to create an automated online version. On the other hand, a small group of say 5 to 10 people can finish the nominal group technique in 1 to 2 hours; the other procedures tend to work better when done over several days, and are largely impossible (in pen-and-paper form) to do in a similar time frame. Even the real-time online versions of the other procedures are difficult to do within 2 hours. Prediction markets in particular tend to fail badly if too thin (i.e. not enough participants); for small groups with tight schedules, nominal group technique tends to be the best.

Conclusion

For small groups that need to make a decision within one or two hours, use nominal group technique. It's relatively unwieldy compared to the other disputation arenas, and is less ideal (it has fewer protections against groupthink, in particular), but is fast compared to the others. Also, one might consider parallelizing nominal group technique: split a large group into smaller, randomly-selected sub-groups, have each perform the procedure independently, and then have them send a representative that performs the idea sharing, group discussion, and ranking steps with other representatives (i.e. each sub-group's nominal group technique serves as the silent idea generation of the super-group of representatives). This tends to bias the super-group towards the agendas of the chosen representatives, but if speed is absolutely necessary for a large group, this may be the best you can do.

As mentioned above, prediction markets tend to fail badly if there are too few participants in the speculation market; use it only for extremely large groups that are impossible to coordinate otherwise. In addition, using prediction markets for policy decisions is effectively futarchy; you may want to see the (defunct?) futarchy_discuss Yahoo! group's message archives. In particular the earlier messages in the archive tend to discuss the general principles of prediction markets. Prediction markets are the most famous of the disputation arenas here, but remember that the Internet is not a decent disputation arena.

The Delphi methods seem to be a "dark horse" of sorts. I don't see much discussion online about Delphi methods; I'm not sure whether it's because it's been tried and rejected, or if it simply isn't well known enough to actually be tried by most people. I tend to suspect the latter, since if the universe were in the former case I would at least see some "Delphi Methods suck!!" blog posts.

Both prediction markets and Delphi methods are continuously repeated methods. At any time, the procedure may be stopped or repeated in order to make decisions. However, unlike the nominal group techniques, both are targeted more towards generating advice for decision-makers, rather than making actual decisions themselves.

It may be possible to organize a large, hierarchical group (say a company) with a prediction market for the rank-and-file, some key experts (who should be aware of the prediction market's results) running a Delphi method, and the key decision-making individuals (who read the Delphi method's report) at the top who form a decision using nominal group technique. For more democratic processes, a "poll-style" real-time online Delphi method by itself may work.