What, in a broad sense, does the future look like? We don't know, and while many have historically made predictions, the track record for such predictions is less than impressive. I have noted that there appear to be two main types of view about the future-- the "new future" and the "business-as-usual future." In order to simplify this discussion, let's restrict it only to the coming century-- the period between 2013 and 2113.

The "new future" is, generally speaking, the idea that the coming century is going to be very different from the present; the "business-as-usual future" is, generally speaking, the idea that the coming century is going to be very similar to the present.

Here are some characteristics of the new future:

• Some large-scale event occurs that alters human experience forever-- an intelligence explosion leading to a technological singularity, existential risks leading to human suppression or extinction, global climate change on a massive scale, etc.
• Society changes a lot, and in fundamental ways that are difficult to understand. Daily life is vastly altered.
• If future history even exists after the dramatic change, it views the coming century as being a critical moment where everything became vastly different, on par with or exceeding the significance of the development of agriculture.

Here are some characteristics of the business-as-usual future:

• The intelligence explosion doesn't happen. AI continues to advance in much the same way that it has for the last several decades. More human-capable tasks become automated, but in slow and predictable ways. Intelligence amplification doesn't happen or doesn't yield generally useful results.
• The world doesn't end. Global warming ends up being just another doomsday scare. Perhaps a lot of people die in the Third World, but the rest of the world adapts and keeps going much like it has for the last ever. Yellowstone doesn't explode. No asteroids hit the earth. There isn't a nuclear war.
• Society doesn't change very much except in superficial ways. Daily life is more or less the same.
• Wars might happen. Nations might collapse. But wars have been happening and nations have been collapsing for thousands of years. By and large, the coming century is viewed by future history as not particularly unlike those that came before.

Reference class forecasting seems to indicate that the business-as-usual future is quite likely. But as we know, this is far from a textbook case of reference class forecasting, and applying such techniques may not be helpful. What, then, is a good method of establishing what you think the future will look like?

Note: This is a draft; so far, about the first half is complete. I'm posting it to Discussion for now; when it's finished, I'll move it to Main. In the mean time, I'd appreciate comments, including suggestions on style and/or format. In particular, if you think I should(n't) try to post this as a sequence of separate sections, let me know.

Summary: You want to find the truth? You want to win? You're gonna have to learn the right way to vote. Plurality voting sucks; better voting systems are built from the blocks of approval, medians (Bucklin cutoffs), delegation, and pairwise opposition. I'm working to promote these systems and I want your help.

Contents: 1. Overblown¹ rhetorical setup ... 2. Condorcet's ideals and Arrow's problem ... 3. Further issues for politics ... 4. Rating versus ranking; a solution? ... 5. Delegation and SODA ... 6. Criteria and pathologies ... 7. Representation, Proportional representation, and Sortition ... 8. What I'm doing about it and what you can ... 9. Conclusions and future directions ... 10. Appendix: voting systems table ... 11. Footnotes

1.

This is a website focused on becoming more rational. But that can't just mean getting a black belt in individual epistemic rationality. In a situation where you're not the one making the decision, that black belt is just a recipe for frustration.

Of course, there's also plenty of content here about how to interact rationally; how to argue for truth, including both hacking yourself to give in when you're wrong and hacking others to give in when they are. You can learn plenty here about Aumann's Agreement Theorem on how two rational Bayesians should never knowingly disagree.

But "two rational Bayesians" isn't a whole lot better as a model for society than "one rational Bayesian". Aspiring to be rational is well and good, but the Socratic ideal of a world tied together by two-person dialogue alone is as unrealistic as the sociopath's ideal of a world where their own voice rules alone. Society needs structures for more than two people to interact. And just as we need techniques for checking irrationality in one- and two-person contexts, we need them, perhaps all the more, in multi-person contexts.

Most of the basic individual and dialogical rationality techniques carry over. Things like noticing when you are confused, or making your opponent's arguments into a steel man, are still perfectly applicable. But there's also a new set of issues when n>2: the issues of democracy and voting. For a group of aspiring rationalists to come to a working consensus, of course they need to begin by evaluating and discussing the evidence, but eventually it will be time to cut off the discussion and just vote. When they do so, they should understand the strengths and pitfalls of voting in general and of their chosen voting method in particular.

And voting's not just useful for an aspiring rationalist community. As it happens, it's an important part of how governments are run. Discussing politics may be a mind-killer in many contexts, but there are an awful lot of domains where politics is a part of the road to winning.² Understanding voting processes a little bit can help you navigate that road; understanding them deeply opens the possibility of improving that road and thus winning more often.

2. Collective rationality: Condorcet's ideals and Arrow's problem

Imagine it's 1785, and you're a member of the French Academy of Sciences. You're rubbing elbows with most of the giants of science and mathematics of your day: Coulomb, Fourier, Lalande, Lagrange, Laplace, Lavoisier, Monge; even the odd foreign notable like Franklin with his ideas to unify electrostatics and electric flow.

One day, they'll put your names in front of lots of cameras (even though that foreign yokel Franklin will be in more pictures)

And this academy, with many of the smartest people in the world, has votes on stuff. Who will be our next president; who should edit and schedule our publications; etc. You're sure that if you all could just find the right way to do the voting, you'd get the right answer. In fact, you can easily prove that, or something like it: if a group is deciding between one right and one wrong option, and each member is independently more than 50% likely to get it right, then as the group size grows the chance of a majority vote choosing the right option goes to 1.

But somehow, there's still annoying politics getting in the way. Some people seem to win the elections simply because everyone expects them to win. So last year, the academy decided on a new election system to use, proposed by your rival, Charles de Borda, in which candidates get different points for being a voters first, second, or third choice, and the one with the most points wins. But you're convinced that this new system will lead to the opposite problem: people who win the election precisely because nobody expected them to win, by getting the points that voters strategically don't want to give to a strong rival. But when people point that possibility out to Borda, he only huffs that "my system is meant for honest men!"

So with your proof of the above intuitive, useful result about two-way elections, you try to figure out how to reduce an n-way election to the two-candidate case. Clearly, you can show that Borda's system will frequently give the wrong results from that perspective. But frustratingly, you find that there could sometimes be no right answer; that there will be no candidate who would beat all the others in one-on-one races. A crack has opened up; could it be that the collective decisions of intelligent individual rational agents could be irrational?

Of course, the "you" in this story is the Marquis de Condorcet, and the year 1785 is when he published his Essai sur l’application de l’analyse à la probabilité des décisions rendues à la pluralité des voix, a work devoted to the question of how to acheive collective rationality. The theorem referenced above is Condorcet's Jury Theorem, which seems to offer hope that democracy can point the way from individually-imperfect rationality towards an ever-more-perfect collective rationality. Just as Aumann's Agreement Theorem shows that two rational agents should always move towards consensus, the Condorcet Jury Theorem apparently shows that if you have enough rational agents, the resulting consensus will be correct.

But as I said, Condorcet also opened a crack in that hope: the possibility that collective preferences will be cyclical. If the assumptions of the jury theorem don't hold — if each voter doesn't have a >50% chance of being right on a randomly-selected question, OR if the correctness of two randomly-selected voters is not independent and uncorrelated — then individually-sensible choices can lead to collectively-ridiculous ones. 

What do I mean by "collectively-ridiculous"? Let's imagine that the Rationalist Marching Band is choosing the colors for their summer, winter, and spring uniforms, and that they all agree that the only goal is to have as much as possible of the best possible colors. The summer-style uniforms come in red or blue, and they vote and pick blue; the winter-style ones come in blue or green, and they pick green; and the spring ones come in green or red, and they pick red.

Obviously, this makes us doubt their collective rationality. If, as they all agree they should, they had a consistent favorite color, they should have chosen that color both times that it was available, rather than choosing three different colors in the three cases. Theoretically, the salesperson could use such a fact to pump money out of them; for instance, offering to let them "trade up" their spring uniform from red to blue, then to green, then back to red, charging them a small fee each time; if they voted consistently as above, they would agree to each trade (though of course in reality human voters would probably catch on to the trick pretty soon, so the abstract ideal of an unending circular money pump wouldn't work).

This is the kind of irrationality that Condorcet showed was possible in collective decisionmaking. He also realized that there was a related issue with logical inconsistencies. If you were take a vote on 3 logically related propositions — say, "Should we have a Minister of Silly Walks, to be appointed by the Chancellor of the Excalibur", "Should we have a Minister of Silly Walks, but not appointed by the Chancellor of the Excalibur", and "Should we in fact have a Minister of Silly Walks at all", where the third cannot be true unless one of the first is — then you could easily get majority votes for inconsistent results — in this case, no, no, and yes, respectively. Obviously, there are many ways to fix the problem in this simple case — probably many less-wrong'ers would suggest some Bayesian tricks related to logical networks and treating votes as evidence⁸ — but it's a tough problem in general even today, especially when the logical relationships can be complex, and Condorcet was quite right to be worried about its implications for collective rationality.³

And that's not the only tough problem he correctly foresaw. Nearly 200 years later and an ocean away, in the 1960s, Kenneth Arrow showed that it was impossible for a preferential voting system to avoid the problem of a "Condorcet cycle" of preferences. Arrows theorem shows that any voting system which can consistently give the same winner (or, in ties, winners) for the same voter preferences; which does not make one voter the effective dictator; which is sure to elect a candidate if all voters prefer them; and which will switch the results for two candidates if you switch their names on all the votes... must exhibit, in at least some situation, the pathology that befell the Rationalist Marching Band above, or in other words, must fail "independence of irrelevant alternatives".

Arrow's theorem is far from obvious a priori, but proof is not hard to understand intuitively using Condorcet's insight. Say that there are three candidates, X, Y, and Z, with roughly equal bases of support; and that they form a Condorcet cycle, because in two-way races, X would beat Y with help from Z supporters, Y would beat Z with help from X supporters, and Z would beat X with help from Y supporters. So whoever wins in the three-way race — say, X — just remove the one who would have lost to them — Y in this case — and that "irrelevant" change will change the winner to be the third — Z in this case.

Summary of above: Collective rationality is harder than individual or two-way rationality. Condorcet saw the problem and tried to solve it, but Arrow saw that Condorcet had been doomed to fail.

3. Further issues for politics

So Condorcet's ideals of better rationality through voting appear to be in ruins. But at least we can hope that voting is a good way to do politics, right?

Not so fast. Arrow's theorem quickly led to further disturbing results. Alan Gibbard (and also Mark Satterthwaite) extended that there is no voting system which doesn't encourage voting strategy. That is, if you view an voting system as a class of games where the finite players and finite available strategies are fixed, no player is effectively a dictator, and the only thing that varies are the payoffs for each player from each outcome, there is no voting system where you can derive your best strategic vote purely by looking "honestly" at your own preferences; there is always the possibility of situations where you have to second-guess what others will do.

Amartya Sen piled on with another depressing extension of Arrows' logic. He showed that there is no possible way of aggregating individual choices into collective choice that satisfies two simple criteria. First, it shouldn't choose pareto-dominated outcomes; if everyone prefers situation XYZ to ABC, that they don't do XYZ. Second, it is "minimally liberal"; that is, there are at least two people who each get to freely make their own decision on at least one specific issue each, no matter what, so for instance I always get to decide between X and A (in Gibbard's⁴ example, colors for my house), and you always get to decide between Y and B (colors for your own house). The problem is that if you nosily care more about my house's color, the decision that should have been mine, and I nosily care about yours, more than we each care about our own, then the pareto-dominant situation is the one where we don't decide our own houses; and that nosiness could, in theory, be the case for any specific choice that, a priori, someone might have labelled as our Inalienable Right. It's not such a surprising result when you think about it that way, but it does clearly show that unswerving ideals of Democracy and Liberty will never truly be compatible.

Meanwhile, "public choice" theorists⁵ like Duncan Black, James Buchanan, etc. were busy undermining the idea of democratic government from another direction: the motivations of the politicians and bureaucrats who are supposed to keep it running. They showed that various incentives, including the strange voting scenarios explored by Condorcet and Arrow, would tend open a gap between the motives of the people and those of the government, and that strategic voting and agenda-setting within a legislature would tend to extend the impact of that gap. Where Gibbard and Sen had proved general results, these theorists worked from specific examples. And in one aspect, at least, their analysis is devastatingly unanswerable: the near-ubiquitous "democratic" system of plurality voting, also known as first-past-the-post or vote-for-one or biggest-minority-wins, is terrible in both theory and practice.

So, by the 1980s, things looked pretty depressing for the theory of democracy. Politics, the theory went, was doomed forever to be a worse than sausage factory; disgusting on the inside and distasteful even from outside.

Should an ethical rationalist just give up on politics, then? Of course not. As long as the results it produces are important, it's worth trying to optimize. And as soon as you take the engineer's attitude of optimizing, instead of dogmatically searching for perfection or uselessly whining about the problems, the results above don't seem nearly as bad.

From this engineer's perspective, public choice theory serves as an unsurprising warning that tradeoffs are necessary, but more usefully, as a map of where those tradeoffs can go particularly wrong. In particular, its clearest lesson, in all-caps bold with a blink tag, that PLURALITY IS BAD, can be seen as a hopeful suggestion that other voting systems may be better. Meanwhile, the logic of both Sen's and Gibbard's theorems are built on Arrow's earlier result. So if we could find a way around Arrow, it might help resolve the whole issue.

Summary of above: Democracy is the worst political system... (...except for all the others?) But perhaps it doesn't have to be quite so bad as it is today.

4. Rating versus ranking

So finding a way around Arrow's theorem could be key to this whole matter. As a mathematical theorem, of course, the logic is bulletproof. But it does make one crucial assumption: that the only inputs to a voting system are rankings, that is, voters' ordinal preference orders for the candidates. No distinctions can be made using ratings or grades; that is, as long as you prefer X to Y to Z, the strength of those preferences can't matter. Whether you put Y almost up near X or way down next to Z, the result must be the same.

Relax that assumption, and it's easy to create a voting system which meets Arrow's criteria. It's called Score voting⁶, and it just means rating each candidate with a number from some fixed interval (abstractly speaking, a real number; but in practice, usually an integer); the scores are added up and the highest total or average wins. (Unless there are missing values, of course, total or average amount to the same thing.) You've probably used it yourself on Yelp, IMDB, or similar sites. And it clearly passes all of Arrow's criteria. Non-dictatorship? Check. Unanimity? Check. Symmetry over switching candidate names? Check. Independence of irrelevant alternatives? In the mathematical sense — that is, as long as the scores for other candidates are unchanged — check.

So score voting is an ideal system? Well, it's certainly a far sight better than plurality. But let's check it against Sen and against Gibbard.

Sen's theorem was based on a logic similar to Arrow. However, while Arrow's theorem deals with broad outcomes like which candidate wins, Sen's deals with finely-grained outcomes like (in the example we discussed) how each separate house should be painted. Extending the cardinal numerical logic of score voting to such finely-grained outcomes, we find we've simply reinvented markets. While markets can be great things and often work well in practice, Sen's result still holds in this case; if everything is on the market, then there is no decision which is always yours to make. But since, in practice, as long as you aren't destitute, you tend to be able to make the decisions you care the most about, Sen's theorem seems to have lost its bite in this context.

What about Gibbard's theorem on strategy? Here, things are not so easy. Yes, Gibbard, like Sen, parallels Arrow. But while Arrow deals with what's written on the ballot, Gibbard deals with what's in the voters head. In particular, if a voter prefers X to Y by even the tiniest margin, Gibbard assumes (not unreasonably) that they may be willing to vote however they need to, if by doing so they can ensure X wins instead of Y. Thus, the internal preferences Gibbard treats are, effectively, just ordinal rankings; and the cardinal trick by which score voting avoided Arrovian problems no longer works.

How does score voting deal with strategic issues in practice? The answer to that has two sides. On the one hand, score never requires voters to be actually dishonest. Unlike the situation in a ranked system such as plurality, where we all know that the strategic vote may be to dishonestly ignore your true favorite and vote for a "lesser evil" among the two frontrunners, in score voting you never need to vote a less-preferred option above a more-preferred option. At worst, all you have to do is exaggerate some distinctions and minimize others, so that you might end giving equal votes to less- and more-preferred options.

Did I say "at worst"? I meant, "almost always". Voting strategy only matters to the result when, aside from your vote, two or more candidates are within one vote of being tied for first. Except in unrealistic, perfectly-balanced conditions, as the number of voters rises, the probability that anyone but the two a priori frontrunner candidates is in on this tie falls to zero.⁷ Thus, in score voting, the optimal strategy is nearly always to vote your preferred frontrunner and all candidate above at the maximum, and your less-preferred frontrunner and all candidates below at the minimum. In other words, strategic score voting is basically equivalent to approval voting, where you give each candidate a 1 or 0 and the highest total wins.

In one sense, score voting reducing to approval OK. Approval voting is not a bad system at all. For instance, if there's a known majority Condorcet winner — a candidate who could beat any other by a majority in a one-on-one race — and voters are strategic — they anticipate the unique strong Nash equilibrium, the situation where no group of voters could improve the outcome for all its members by changing their votes, whenever such a unique equilibrium exists — then the Condorcet winner will win under approval. That's a lot of words to say that approval will get the "democratic" results you'd expect in most cases.

But in another sense, it's a problem. If one side of an issue is more inclined to be strategic than the other side, the more-strategic faction could win even if it's a minority. That clashes with many people's ideals of democracy; and worse, it encourages mind-killing political attitudes, where arguments are used as soldiers rather than as ways to seek the truth.

But score and approval voting are not the only systems which escape Arrow's theorem through the trapdoor of ratings. If score voting, using the average of voter ratings, too-strongly encourages voters to strategically seek extreme ratings, then why not use the median rating instead? We know that medians are less sensitive to outliers than averages. And indeed, median-based systems are more resistant to one-sided strategy than average-based ones, giving better hope for reasonable discussion to prosper. That is to say, in a simple model, a minority would need twice as much strategic coordination under median as under average, in order to overcome a majority; and there's good reason to believe that, because of natural factional separation, reality is even more favorable to median systems than that model.

There are several different median systems available. In the US during the 1910-1925 Progressive Era, early versions collectively called "Bucklin voting" were used briefly in over a dozen cities. These reforms, based on counting all top preferences, then adding lower preferences one level at a time until some candidate(s) reach a majority, were all rolled back soon after, principally by party machines upset at upstart challenges or victories. The possibility of multiple, simultaneous majorities is a principal reason for the variety of Bucklin/Median systems. Modern proposals of median systems include Majority Approval Voting, Majority Judgment, and Graduated Majority Judgment, which would probably give the same winners almost all of the time. An important detail is that most median system ballots use verbal or letter grades rather than numeric scores. This is justifiable because the median is preserved under any monotonic transformation, and studies suggest that it would help discourage strategic voting.

Serious attention to rated systems like approval, score, and median systems barely began in the 1980s, and didn't really pick up until 2000. Meanwhile, the increased amateur interest in voting systems in this period — perhaps partially attributable to the anomalous 2000 US presidential election, or to more-recent anomalies in the UK, Canada, and Australia — has led to new discoveries in ranked systems as well. Though such systems are still clearly subject to Arrow's theorem, new "improved Condorcet" methods which use certain tricks to count a voter's equal preferences between to candidates on either side of the ledger depending on the strategic needs, seem to offer promise that Arrovian pathologies can be kept to a minimum.

With this embarrassment of riches of systems to choose from, how should we evaluate which is best? Well, at least one thing is a clear consensus: plurality is a horrible system. Beyond that, things are more controversial; there are dozens of possible objective criteria one could formulate, and any system's inventor and/or supporters can usually formulate some criterion by which it shines.

Ideally, we'd like to measure the utility of each voting system in the real world. Since that's impossible — it would take not just a statistically-significant sample of large-scale real-world elections for each system, but also some way to measure the true internal utility of a result in situations where voters are inevitably strategically motivated to lie about that utility — we must do the next best thing, and measure it in a computer, with simulated voters whose utilities are assigned measurable values. Unfortunately, that requires assumptions about how those utilities are distributed, how voter turnout is decided, and how and whether voters strategize. At best, those assumptions can be varied, to see if findings are robust.

In 2000, Warren Smith performed such simulations for a number of voting systems. He found that score voting had, very robustly, one of the top expected social utilities (or, as he termed it, lowest Bayesian regret). Close on its heels were a median system and approval voting. Unfortunately, though he explored a wide parameter space in terms of voter utility models and inherent strategic inclination of the voters, his simulations did not include voters who were more inclined to be strategic when strategy was more effective. His strategic assumptions were also unfavorable to ranked systems, and slightly unrealistic in other ways. Still, though certain of his numbers must be taken with a grain of salt, some of his results were large and robust enough to be trusted. For instance, he found that plurality voting and instant runoff voting were clearly inferior to rated systems; and that approval voting, even at its worst, offered over half the benefits compared to plurality of any other system.

Summary of above: Rated systems, such as approval voting, score voting, and Majority Approval Voting, can avoid the problems of Arrow's theorem. Though they are certainly not immune to issues of strategic voting, they are a clear step up from plurality. Starting with this section, the opinions are my own; the two prior sections were based on general expert views on the topic.

5. Delegation and SODA

Rated systems are not the only way to try to beat the problems of Arrow and Gibbard (/Satterthwaite).

Summary of above:

6. Criteria and pathologies

do.

Summary of above:

7. Representation, proportionality, and sortition

do.

Summary of above:

8. What I'm doing about it and what you can

do.

Summary of above:

9. Conclusions and future directions

do.

Summary of above:

10. Appendix: voting systems table

Compliance of selected systems (table)

The following table shows which of the above criteria are met by several single-winner systems. Note: contains some errors; I'll carefully vet this when I'm finished with the writing. Still generally reliable though.

"Yes/No", in a column which covers two related criteria, signifies that the given system passes the first criterion and not the second one.

11. Footnotes

¹ When I call my introduction "overblown", I mean that I reserve the right to make broad generalizations there, without getting distracted by caveats. If you don't like this style, feel free to skip to section 2.

² Of course, the original "politics is a mind killer" sequence was perfectly clear about this: "Politics is an important domain to which we should individually apply our rationality—but it's a terrible domain in which to learn rationality, or discuss rationality, unless all the discussants are already rational." The focus here is on the first part of that quote, because I think Less Wrong as a whole has moved too far in the direction of avoiding politics as not a domain for rationalists.

³ Bayes developed his theorem decades before Condorcet's Essai, but Condorcet probably didn't know of it, as it wasn't popularized by Laplace until about 30 years later, after Condorcet was dead.

⁴ Yes, this happens to be the same Alan Gibbard from the previous paragraph.

⁵ Confusingly, "public choice" refers to a school of thought, while "social choice" is the name for the broader domain of study. Stop reading this footnote now if you don't want to hear mind-killing partisan identification. "Public choice" theorists are generally seen as politically conservative in the solutions they suggest. It seems to me that the broader "social choice" has avoided taking on a partisan connotation in this sense.

⁶ Score voting is also called "range voting" by some. It is not a particularly new idea — for instance, the "loudest cheer wins" rule of ancient Sparta, and even aspects of honeybees' process for choosing new hives, can be seen as score voting — but it was first analyzed theoretically around 2000. Approval voting, which can be seen as a form of score voting where the scores are restricted to 0 and 1, had entered theory only about two decades earlier, though it too has a history of practical use back to antiquity.

⁷ OK, fine, this is a simplification. As a voter, you have imperfect information about the true level of support and propensity to vote in the superpopulation of eligible voters, so in reality the chances of a decisive tie between other than your two expected frontrunners is non-zero. Still, in most cases, it's utterly negligible.

⁸ This article will focus more on the literature on multi-player strategic voting (competing boundedly-instrumentally-rational agents) than on multi-player Aumann (cooperating boundedly-epistemically-rational agents). If you're interested in the latter, here are some starting points: Scott Aaronson's work is, as far as I know, the state of the art on 2-player Aumann, but its framework assumes that the players have a sophisticated ability to empathize and reason about each others' internal knowledge, and the problems with this that Aaronson plausibly handwaves away in the 2-player case are probably less tractable in the multi-player one. Dalkiran et al deal with an Aumann-like problem over a social network; they find that attempts to "jump ahead" to a final consensus value instead of simply dumbly approaching it asymptotically can lead to failure to converge. And Kanoria et al have perhaps the most interesting result from the perspective of this article; they use the convergence of agents using a naive voting-based algorithm to give a nice upper bound on the difficulty of full Bayesian reasoning itself. None of these papers explicitly considers the problem of coming to consensus on more than one logically-related question at once, though Aaronson's work at least would clearly be easy to extend in that direction, and I think such extensions would be unsurprisingly Bayesian.

I've seen Richard Hamming's classic talk You And Your Research referenced several times on LessWrong and figured I would post the full version. The introduction is reproduced below:

The title of my talk is, ``You and Your Research.'' It is not about managing research, it is about how you individually do your research. I could give a talk on the other subject - but it's not, it's about you. I'm not talking about ordinary run-of-the-mill research; I'm talking about great research. And for the sake of describing great research I'll occasionally say Nobel-Prize type of work. It doesn't have to gain the Nobel Prize, but I mean those kinds of things which we perceive are significant things. Relativity, if you want, Shannon's information theory, any number of outstanding theories - that's the kind of thing I'm talking about.

Now, how did I come to do this study? At Los Alamos I was brought in to run the computing machines which other people had got going, so those scientists and physicists could get back to business. I saw I was a stooge. I saw that although physically I was the same, they were different. And to put the thing bluntly, I was envious. I wanted to know why they were so different from me. I saw Feynman up close. I saw Fermi and Teller. I saw Oppenheimer. I saw Hans Bethe: he was my boss. I saw quite a few very capable people. I became very interested in the difference between those who do and those who might have done.

When I came to Bell Labs, I came into a very productive department. Bode was the department head at the time; Shannon was there, and there were other people. I continued examining the questions, ``Why?'' and ``What is the difference?'' I continued subsequently by reading biographies, autobiographies, asking people questions such as: ``How did you come to do this?'' I tried to find out what are the differences. And that's what this talk is about.

I consider this talk good and useful not only for those interested in research, but for those interested in achieving much of anything. Check it out!

I'm finally getting around to reading "Thinking, Fast and Slow". Much of it I had already learned on LW and elsewhere. Maybe that's why my strongest impression from the book is how accessible it is. Simple sentences, clear and vivid examples, easy-to-follow exercises, a remarkable lack of references to topics not explained right away.

I caught myself thinking "This is a book I should have read as a kid". In my first language, I think I could have managed it as early as 11 years old. Since measured IQ is strongly influenced by habits of thinking and cognitive returns can be reinvested, I'm sure I would be smarter now if I had.

So I have decided to buy a stack of these books and give them to kids on their, say, 12th birthdays. Then maybe Dan Dennett's "Intuition Pumps" a year later - and HPMOR a year after that? I would like to see more suggestions from you guys.

It should be obviously better to start even earlier. So how do you teach rationality to a nine-year-old? Or a seven-year-old? Has anybody done something like that? Please name books, videos or web sites.

If such media are not available, creating them should be low-hanging fruit in the quest to raise the global IQ and sanity waterline. ELI5 writing is very learnable, after all, and ELI5 type interpretations of, say, the sequences, might be helpful for adults too.

There's a recent science fiction story that I can't recall the name of, in which the narrator is traveling somewhere via plane, and the security check includes a brain scan for deviance. The narrator is a pedophile. Everyone who sees the results of the scan is horrified--not that he's a pedophile, but that his particular brain abnormality is easily fixed, so that means he's chosen to remain a pedophile. He's closely monitored, so he'll never be able to act on those desires, but he keeps them anyway, because that's part of who he is.

What would you do in his place?

Really new to this site, I'm hitting a problem I've experienced in other aspects of my life as a student, employee and comedic performer: I'm extremely rude. I don't realize it at the time, thinking that I'm just being blunt, forceful and direct. In the sense that those can all be definitions of similar concepts, then yeah! Well, confidence in myself is a great asset, and I've turned it to positive effect, especially when I need to intimidate someone with a roguish smile and a calm, iron-hard assertion backed up by a blistering intensity (I sound like a Marty-Stu right now. Draco will want his leather pants back.)

But making rationally sound argument should not be about winning. It should be about accuracy, clarity and sanity. If you disagree with me on something I need to remind myself not to automatically fight. What good is being alpha when I'm ignoring my confusion and avoiding my embarrassment at possibly being mistaken. Not to mention that this is an internet forum, so limitations of the medium means attempting to look like a tough guy winds up hollow and sad, like a chocolate Easter bunny that's gone off.

Okay, I'm not one for similes, but I am one for trying to make myself more sane.

I think it would be a useful rationality exercise to take something that millions of people love, that you have contempt for, and sincerely try to appreciate it. The purpose would be to get practice imagining a different point of view, and to see whether you're able to do so.

50 Shades of Grey might not be the best choice for this exercise. I haven't read it. Maybe a better choice would be The da Vinci Code, NASCAR, or professional wrestling. But a book has a definite length that you have to get through to be allowed to say that you honestly tried.

The idea is to start a thread for people to discuss 50 Shades, or something else perceived as trashy, and try--not to find what you might like in it, but to get inside someone else's head and imagine why they might like it. (Of course it could backfire, and leave you less open-minded than before, if you always conclude that your contempt was simply right in the first place.)

I think the biggest problem is that if people succeed at finding something to appreciate in it, they would feel terribly embarrassed to say so. So this can be done with alternate accounts.

Anyone interested? What do you hold in contempt that you might be willing to take a closer look at?

In an attempt to encourage more people to actually do awesome things (a la instrumental rationality), I am proposing a new monthly thread (can be changed to bi-weekly, should that be demanded). Your job, should you choose to accept it, is to comment on this thread explaining the most awesome thing you've done this month. You may be as blatantly proud of you self as you feel. You may unabashedly consider yourself the coolest freaking person ever because of that awesome thing you're dying to tell everyone about. This is the place to do just that.

Remember, however, that this isn't any kind of progress thread. Nor is it any kind of proposal thread.This thread is solely for people to talk about the awesomest thing they've done all month. not will do. not are working on. have already done. This is to cultivate an environment of object level productivity rather than meta-productivity methods.

So, what's the coolest thing you've done this month?

Article Prerequisite: Self-Improvement or Shiny Distraction: Why Less Wrong is anti-Instrumental Rationality

Introduction

The goal of this post is to explore the idea of rationality training, feedback and ideas are greatly appreciated.

Less Wrong’s stated mission is to help people become more rational, and it has made progress toward that goal. Members read and discuss useful ideas on the internet, get instant feedback because of the voting system, and schedule meetups with other members. Less Wrong also helps attract more people to rationality.

Less Wrong helps with sharing ideas, but it fails to help people put elements of epistemic and instrumental rationality into practice. This is a serious problem, but it would be hard to fix without altering the core functionality of Less Wrong.

Having separate websites for reading and discussing ideas and then actually using those ideas would improve the real world performance of the Less Wrong community while maintaining the idea discussion, “marketing”, and other benefits of the Less Wrong website.

How to create a useful website for self improvement

1. Knowledge Management

When reading blogs, people only see recent posts and those posts are not significantly revised. A wiki would allow for the creation of a large body of organized knowledge that is frequently revised. Each wiki post would have a description, benefits of the topic described, resources to learn the topic, user submitted resources to learn the topic, and reviews of each resource. Posts would be organized hierarchically and voted on for usefulness to help readers effectively improve what they are looking for. Users could share self-improvement plans to help others improve effectiveness in general or in a specific topic as quickly as possible.

2. Effective Learning

Resources to learn topics should be arranged or written for effective skill acquisition, and there may be different resource categories like exercises for deliberate practice or active recall questions for spaced repetition.

3. Quality Contributors

Contributors would, at the very least, need to be familiar with how to write articles that supported the skill acquisition process agreed upon by the entire community. Required writing and research skills would produce higher quality work. I am not sure if being a rationalist would improve the quality of articles.

Problems

1. Difficult requirements

The number of prerequisites necessary to contribute to and use the wiki would really lower the amount of people who will be able to benefit from the wiki. It's a trade off between effectiveness and popularity. What elements should be included to maximize the effectiveness of the website?

2. Interest

There has to be enough interest in the website, or else a different project should be started instead. How many people in the Less Wrong community, and the world at large, would be interested in self improvement and rationality? 

3. Increasing the effectiveness of non altruistic people

How much of the target audience wants to improve the world? If most do not, then the wiki would essentially be a net negative on the world. What should the criteria be to view and contribute to the wiki? Perhaps only Less Wrong members should be able to view and edit the wiki, and contributors must read a quick start guide and pass a quick test before being allowed to post.

The other rationality quotes thread operates under the rule:

Do not quote from Less Wrong itself, Overcoming Bias, or HPMoR.

Lately it seems that every MIRI or CFAR employee is excempt from being quoted.

As there are still interesting quotes that happen on LessWrong, Overcoming Bias, HPMoR and MIRI/CFAR employee in general, I think it makes sense to open this thread to provide a place for those quotes. 

We recently established a successful Useful Concepts Repository.  It got me thinking about all the useless or actively harmful concepts I had carried around for in some cases most of my life before seeing them for what they were.  Then it occurred to me that I probably still have some poisonous concepts lurking in my mind, and I thought creating this thread might be one way to discover what they are.

I'll start us off with one simple example:  The Bohr model of the atom as it is taught in school is a dangerous thing to keep in your head for too long.  I graduated from high school believing that it was basically a correct physical representation of atoms.  (And I went to a *good* high school.)  Some may say that the Bohr model serves a useful role as a lie-to-children to bridge understanding to the true physics, but if so, why do so many adults still think atoms look like concentric circular orbits of electrons around a nucleus?  

There's one hallmark of truly bad concepts: they actively work against correct induction.  Thinking in terms of the Bohr model actively prevents you from understanding molecular bonding and, really, everything about how an atom can serve as a functional piece of a real thing like a protein or a diamond.

Bad concepts don't have to be scientific.  Religion is held to be a pretty harmful concept around here.  There are certain political theories which might qualify, except I expect that one man's harmful political concept is another man's core value system, so as usual we should probably stay away from politics.  But I welcome input as fuzzy as common folk advice you receive that turned out to be really costly.

A new Mr. Money Mustache article for those who enjoyed my sequence on financial planning and extreme early retirement.

When the Back of the Napkin can be Worth Millions

From EY's Facebook page, there were two posts that got me thinking about fiction and how to work it better and make it stronger:

It would have been trivial to fix _Revenge of the Sith_'s inadequate motivation of Anakin's dark turn; have Padme already in the hospital slowly dying as her children come to term, not just some nebulous "visions". (Bonus points if you have Yoda lecture Anakin about the inevitability of death, but I'd understand if they didn't go there.) At the end, Anakin doesn't try to choke Padme; he watches the ship with her fly out of his reach, away from his ability to use his unnatural Sith powers to save her. Now Anakin's motives are 320% more sympathetic and the movie makes 170% more sense. If I'd put some serious work in, I'm pretty sure I could've had the movie audience in tears.

I still feel a sense of genuine puzzlement on how such disastrous writing happens in movies and TV shows. Are the viewers who care about this such a tiny percentage that it's not worth trying to sell to them? Are there really so few writers who could read over the script and see in 30 seconds how to fix something like this? (If option 2 is really the problem and people know it's the problem, I'd happily do it for $10,000 a shot.) Is it Graham's Design Paradox - can Hollywood moguls just not tell the difference between competent writers making such an offer, and fakers who'll take the money and run? Are the producers' egos so grotesque that they can't ask a writer for help? Is there some twisted sense of superiority bound up with believing that the audience is too dumb to care about this kind of thing, even though it looks to me like they do? I don't understand how a >$100M movie ends up with flaws that I could fix at the script stage with 30 seconds of advice.

A helpful key to understanding the art and technique of character in storytelling, is to consider the folk-psychological notion from Internal Family Systems of people being composed of different 'parts' embodying different drives or goals. A shallow character is then a character with only one 'part'.

A good rule of thumb is that to create a 3D character, that person must contain at least two different 2D characters who come into conflict. Contrary to the first thought that crosses your mind, three-dimensional good people are constructed by combining at least two different good people with two different ideals, not by combining a good person and a bad person. Deep sympathetic characters have two sympathetic parts in conflict, not a sympathetic part in conflict with an unsympathetic part. Deep smart characters are created by combining at least two different people who are geniuses.

E.g. HPMOR!Hermione contains both a sensible young girl who tries to keep herself and her friends out of trouble, and a starry-eyed heroine, neither of whom are stupid. (Actually, since HPMOR!Hermione is also the one character who I created as close to her canon self as I could manage - she didn't *need* upgrading - I should credit this one to J. K. Rowling.) (Admittedly, I didn't actually follow that rule deliberately to construct Methods, I figured it out afterward when everyone was praising the characterization and I was like, "Wait, people are calling me a character author now? What the hell did I just do right?")

If instead you try to construct a genius character by having an emotionally impoverished 'genius' part in conflict with a warm nongenius part... ugh. Cliche. Don't write the first thing that pops into your head from watching Star Trek. This is not how real geniuses work. HPMOR!Harry, the primary protagonist, contains so many different people he has to give them names, and none of them are stupid, nor does any one of them contain his emotions set aside in a neat jar; they contain different mixtures of emotions and ideals. Combining two cliche characters won't be enough to build a deep character. Combining two different realistic people in that character's situation works much better. Two is not a limit, it's a minimum, but everyone involved still has to be recognizably the same person when combined.

Closely related is Orson Scott Card's observation that a conflict between Good and Evil can be interesting, but it's often not half as interesting as a conflict between Good and Good. All standard rules about cliches still apply, and a conflict between good and good which you've previously read about and to which the reader can already guess your correct approved answer, cannot carry the story. A good rule of thumb is that if you have a conflict between good and good which you feel unsure about yourself, or which you can remember feeling unsure about, or you're not sure where exactly to draw the line, you can build a story around it. I consider the most successful moral conflict in HPMOR to be the argument between Harry and Dumbledore in Ch. 77 because it almost perfectly divided the readers on who was in the right *and* about whose side the author was taking. (*This* was done by deliberately following Orson Scott Card's rule, not by accident. Likewise _Three Worlds Collide_, though it was only afterward that I realized how much of the praise for that story, which I hadn't dreamed would be considered literarily meritful by serious SF writers, stemmed from the sheer rarity of stories built around genuinely open moral arguments. Orson Scott Card: "Propaganda only works when the reader feels like you've been absolutely fair to other side", and writing about a moral dilemma where *you're* still trying to figure out the answer is an excellent way to achieve this.)

Character shallowness can be a symptom of moral shallowness if it reflects a conflict between Good and Evil drawn along lines too clear to bring two good parts of a good character into conflict. This is why it would've been hard for Lord of the Rings to contain conflicted characters without becoming an entirely different story, though as Robin Hanson has just remarked, LotR is a Mileu story, not a Character story. Conflicts between evil and evil are even shallower than conflicts between good and evil, which is why what passes for 'maturity' in some literature is so uninteresting. There's nothing to choose there, no decision to await with bated breath, just an author showing off their disillusionment as a claim of sophistication.

I was wondering if we could apply this process to older fiction, Great Literature that is historically praised, and excellent by its own time's standards, but which, if published by a modern author, would seem substandard or inappropriate in one way or another.

Given our community's propensity for challenging sacred cows, and the unique tool-set available to us, I am sure we could take some great works of the past and turn them into awesome works of the present.

Of course, it doesn't have to be a laboratory where we rewrite the whole damn things. Just proprely-grounded suggestions on how to improve this or that work would be great.

P.S. This post is itself a work in progress, and will update and improve as comments come. It's been a long time since I've last posted on LW, so advice is quite welcome. Our work is never over.

EDIT: Well, I like that this thread has turned out so lively, but I've got finals to prepare for and I can't afford to keep participating in the discussion to my satisfaction. I'll be back in July, and apologize in advance for being such a poor OP. That said, cheers!

A recent paper in Cortex describes how caloric vestibular stimulation (CVS), i.e., rinsing of the ear canal with cold water, reduces unrealistic optimism. Here are some bits from the paper:

Participants were 31 healthy right-handed adults (15 men, 20–40 years)...

Participants were oriented in a supine position with the head inclined 30° from the horizontal and cold water (24 °C) was irrigated into the external auditory canal on one side (Fitzgerald and Hallpike, 1942). After both vestibular-evoked eye movements and vertigo had stopped, the procedure was repeated on the other side...

Participants were asked to estimate their own risk, relative to that of their peers (same age, sex and education), of contracting a series of illnesses. The risk rating scale ranged from −6 (lower risk) to +6 (higher risk). ... Each participant was tested in three conditions, with 5 min rest between each: baseline with no CI (always first), left-ear CI and right-ear CI (order counterbalanced). In the latter conditions risk-estimation was initiated after 30 sec of CI, when nystagmic response had built up. Ten illnesses were rated in each condition and the average risk estimate per condition (mean of 10 ratings) was calculated for each participant. The 30 illnesses used in this study (see Table 1) were selected from a larger pool of illnesses pre-rated by a separate group of 30 healthy participants.Overall, our participants were unrealistically optimistic about their chances of contracting illnesses at baseline ... and during right-ear CI. ...Post-hoc tests using the Bonferroni correction revealed that, compared to baseline, average risk estimates were significantly higher during left-ear CI (p = .016), whereas they remained unchanged during right-ear CI (p = .476). Unrealistic optimism was thus reduced selectively during left-ear stimulation.

(CI stands for caloric irrigation which is how CVS was performed.)

It is not clear how close the participants came to being realistic in their estimates after CVS, but they definitely became more pessimistic, which is the right direction to go in the context of numerous biases such as the planning fallacy.

The paper:

Vestibular stimulation attenuates unrealistic optimism

• Ryan McKay, 
• Corinne Tamagni, 
• Antonella Palla, 
• Peter Krummenacher, 
• Stefan C.A. Hegemann, 
• Dominik Straumann, 
• Peter Brugger

I've read about a quarter of the sequences, but I'm not sure if this topic has been addressed on LessWrong before. If it has, let me know.

The Upward Scaling Importance of Rationality goes like this:

The more influence your thought process and decisions have, the more important it is that you're rationalist. In the grand scheme of things, it is relatively unimportant that a barback at a restaurant is a rationalist, and I say this having done that. It is extremely important that a leader of a highly influential company, or a president of a university or country is a rationalist. Their decisions affect thousands if not millions of people.

The more influential you are, the more your decisions have potential to screw over other people. Influence doesn't necessarily have to be in a management position: elementary school teachers and police officers are highly influential, even though they aren't in control of an organization. Influence can even be by virtue of the people you reach out to. A famous person with a large fanbase or a parent of a child prodigy, both have the capacity to influence the world with their decisions.

Though arguably, this can  be extended to anyone who votes.

So rationality scales upward: the more influential someone is, the more important is it they're rationalists. Neglecting this can have bad consequences.

In the book "How to Measure Anything" D. Hubbard presents a step-by-step method for calibrating your confidence intervals, which he has tested on hundreds of people, showing that it can make 90% of people almost perfect estimators within half a day of training.

I've been told that the Less Wrong and CFAR community is mostly not aware of this work, so given the importance of making good estimates to rationality, I thought it would be of interest.

(although note CFAR has developed its own games for training confidence interval calibration)

The main techniques to employ are:

Equivalent bet:

For each estimate imagine that you are betting $1000 on the answer being within your 90% CI. Now compare this to betting $1000 on a spinner where 90% of the time you win and 10% of the time you lose. Would you prefer to take a spin? If so, your range is too small and you need to increase it. If you decide to answer the question your range is too large and you need to reduce it. If you don’t mind whether you answer the question or take a spin then it really is your 90% CI.

Absurdity Test:

Start with an absurdly large range, maybe from minus infinity to plus infinity, and then begin reducing it based upon things you know to be highly unlikely or even impossible.

Avoid Anchoring:

Anchoring occurs when you think of a single answer to the question and then add an error around this answer; this often leads to ranges which are too narrow. Using the absurdity test is a good way to counter problems brought on by anchoring; another is to change how you look at your 90% CI. For a 90% CI there is a 10% chance that the answer lies outside your estimate, and if you split this there is a 5% chance that the answer is above your upper bound and a 5% chance that the answer is below your lower bound. By treating each bound separately, rephrase the question to read ‘is there a 95% chance that the answer is above my lower bound?’. If the answer is no, then you need to increase or decrease the bound as required. You can then repeat this process for the other bound.

Pros and cons:

Identify two pros and two cons for the range that you have given to help clarify your reasons for making this estimate.

Once you have used these techniques you can make another equivalent bet to check whether your new estimate is your 90% CI.

 

To train yourself, practice making estimates repeatedly while using these techniques, until you reach 100% accuracy.

To read more and try sample questions, read the article we prepared on 80,000 Hours here.

Like Eliezer, I "do my best thinking into a keyboard." It starts with a burning itch to figure something out. I collect ideas and arguments and evidence and sources. I arrange them, tweak them, criticize them. I explain it all in my own words so I can understand it better. By then it is nearly something that others would want to read, so I clean it up and publish, say, How to Beat Procrastination. I write essays in the original sense of the word: "attempts."

This time, I'm trying to figure out something we might call "tacit rationality" (c.f. tacit knowledge).

I tried and failed to write a good post about tacit rationality, so I wrote a bad post instead — one that is basically a patchwork of somewhat-related musings on explicit and tacit rationality. Therefore I'm posting this article to LW Discussion. I hope the ensuing discussion ends up leading somewhere with more clarity and usefulness.

Three methods for training rationality

Which of these three options do you think will train rationality (i.e. systematized winning, or "winning-rationality") most effectively?

1. Spend one year reading and re-reading The Sequences, studying the math and cognitive science of rationality, and discussing rationality online and at Less Wrong meetups.
2. Attend a CFAR workshop, then spend the next year practicing those skills and other rationality habits every week.
3. Run a startup or small business for one year.

Option 1 seems to be pretty effective at training people to talk intelligently about rationality (let's call that "talking-rationality"), and it seems to inoculate people against some common philosophical mistakes.

We don't yet have any examples of someone doing Option 2 (the first CFAR workshop was May 2012), but I'd expect Option 2 — if actually executed — to result in more winning-rationality than Option 1, and also a modicum of talking-rationality.

What about Option 3? Unlike Option 2 or especially Option 1, I'd expect it to train almost no ability to talk intelligently about rationality. But I would expect it to result in relatively good winning-rationality, due to its tight feedback loops.

Talking-rationality and winning-rationality can come apart

I've come to believe... that the best way to succeed is to discover what you love and then find a way to offer it to others in the form of service, working hard, and also allowing the energy of the universe to lead you.

Oprah Winfrey

Oprah isn't known for being a rational thinker. She is a known peddler of pseudoscience, and she attributes her success (in part) to allowing "the energy of the universe" to lead her.

Yet she must be doing something right. Oprah is a true rags-to-riches story. Born in Mississippi to an unwed teenage housemaid, she was so poor she wore dresses made of potato sacks. She was molested by a cousin, an uncle, and a family friend. She became pregnant at age 14.

But in high school she became an honors student, won oratory contests and a beauty pageant, and was hired by a local radio station to report the news. She became the youngest-ever news anchor at Nashville's WLAC-TV, then hosted several shows in Baltimore, then moved to Chicago and within months her own talk show shot from last place to first place in the ratings there. Shortly afterward her show went national. She also produced and starred in several TV shows, was nominated for an Oscar for her role in a Steven Spielberg movie, launched her own TV cable network and her own magazine (the "most successful startup ever in the [magazine] industry" according to Fortune), and became the world's first female black billionaire.

I'd like to suggest that Oprah's climb probably didn't come merely through inborn talent, hard work, and luck. To get from potato sack dresses to the Forbes billionaire list, Oprah had to make thousands of pretty good decisions. She had to make pretty accurate guesses about the likely consequences of various actions she could take. When she was wrong, she had to correct course fairly quickly. In short, she had to be fairly rational, at least in some domains of her life.

Similarly, I know plenty of business managers and entrepreneurs who have a steady track record of good decisions and wise judgments, and yet they are religious, or they commit basic errors in logic and probability when they talk about non-business subjects.

What's going on here? My guess is that successful entrepreneurs and business managers and other people must have pretty good tacit rationality, even if they aren't very proficient with the "rationality" concepts that Less Wrongers tend to discuss on a daily basis. Stated another way, successful businesspeople make fairly rational decisions and judgments, even though they may confabulate rather silly explanations for their success, and even though they don't understand the math or science of rationality well.

LWers can probably outperform Mark Zuckerberg on the CRT and the Berlin Numeracy Test, but Zuckerberg is laughing at them from atop a huge pile of utility.

Explicit and tacit rationality

Patri Friedman, in Self-Improvement or Shiny Distraction: Why Less Wrong is anti-Instrumental Rationality, reminded us that skill acquisition comes from deliberate practice, and reading LW is a "shiny distraction," not deliberate practice. He said a real rationality practice would look more like... well, what Patri describes is basically CFAR, though CFAR didn't exist at the time.

In response, and again long before CFAR existed, Anna Salamon wrote Goals for which Less Wrong does (and doesn't) help. Summary: Some domains provide rich, cheap feedback, so you don't need much LW-style rationality to become successful in those domains. But many of us have goals in domains that don't offer rapid feedback: e.g. whether to buy cryonics, which 40-year investments are safe, which metaethics to endorse. For this kind of thing you need LW-style rationality. (We could also state this as "Domains with rapid feedback train tacit rationality with respect to those domains, but for domains without rapid feedback you've got to do the best you can with LW-style "explicit rationality".)

The good news is that you should be able to combine explicit and tacit rationality. Explicit rationality can help you realize that you should force tight feedback loops into whichever domains you want to succeed in, so that you can have develop good intuitions about how to succeed in those domains. (See also: Lean Startup or Lean Nonprofit methods.)

Explicit rationality could also help you realize that the cognitive biases most-discussed in the literature aren't necessarily the ones you should focus on ameliorating, as Aaron Swartz wrote:

Cognitive biases cause people to make choices that are most obviously irrational, but not most importantly irrational... Since cognitive biases are the primary focus of research into rationality, rationality tests mostly measure how good you are at avoiding them... LW readers tend to be fairly good at avoiding cognitive biases... But there a whole series of much more important irrationalities that LWers suffer from. (Let's call them "practical biases" as opposed to "cognitive biases," even though both are ultimately practical and cognitive.)

...Rationality, properly understood, is in fact a predictor of success. Perhaps if LWers used success as their metric (as opposed to getting better at avoiding obvious mistakes), they might focus on their most important irrationalities (instead of their most obvious ones), which would lead them to be more rational and more successful.

Final scattered thoughts

• If someone is consistently winning, and not just because they have tons of wealth or fame, then maybe you should conclude they have pretty good tacit rationality even if their explicit rationality is terrible.
• The positive effects of tight feedback loops might trump the effects of explicit rationality training.
• Still, I suspect explicit rationality plus tight feedback loops could lead to the best results of all.
• I really hope we can develop a real rationality dojo.
• If you're reading this post, you're probably spending too much time reading Less Wrong, and too little time hacking your motivation system, learning social skills, and learning how to inject tight feedback loops into everything you can.

I'm shopping for a car, and I've spent many hours this past month reading user reviews of cars. There are seven things American car buyers have cared and complained about consistently for at least the past ten years.  In roughly decreasing importance:

• Performance
• Gas mileage
• Frequency and expense of repairs
• Smoothness of ride
• Exterior and interior styling
• Cup-holders
• Cargo space

Six of these things are complicated design trade-offs. For a good design, increasing any one of them makes most of the other five take a hit.

Cup-holders are not a complicated design trade-off. This should be a solved problem: Put two large, sturdy cup-holders somewhere accessible from the driver's seat. There is nothing to be gained from saving a few centimeters on cup-holder space that could be worth the millions of buyers who will walk away from a $50,000 car because they don't like its cup-holders.

Seriously, build the cup-holders first and design the rest of the interior around them. They're that important.

In the 1970s, no one had cup-holders or knew that they needed them. Things began changing in the 1980s, perhaps due to the expansion of Starbucks, perhaps due to the sudden increase in commute lengths. Today I like to have at least two and preferably three drinks with me for my 1-hour morning commute: A hot coffee to wake up, cold water for when I burn myself with the coffee, and a soda or tea for variety.

But car manufacturers were glacially slow to respond. I've been looking at used Jaguar XJs. These cars originally cost about $100,000 in today's money. Their owners complained continually about the cheap tiny plastic folding cup-holders that couldn't hold cups. They posted do-it-yourself fixes in online forums. Jaguar didn't even begin to address this until 2004, at least fifteen years into the cup-holder crisis, when they made the cup-holders slightly (but not much) less-crappy, and large enough to hold a small coffee (but not a medium).

Most new cars today finally have two cup-holders up front, and the collapsible cup-holders that enraged drivers for years by (predictably) collapsing are finally gone, but many cup-holders still aren't large enough to hold a Starbucks venti.

What the cup-holder paradox implies is that there are many multi-billion dollar care companies that spend hundreds of millions of dollars on product development every year without ever assigning a single summer intern to take one day to read some of the many thousands of user reviews available for free on cars.com, autotrader.com, and other websites. If they had, they'd have realized the depth of America's anger at shoddy cup-holders.

Or perhaps they read the reviews and dismiss them, because their customers are obviously morons who don't appreciate good auto design. Even today, auto manufacturers post photos of the interiors of all their new cars on their websites, but never in a dozen photos give you a clear view of the cup-holders, which makes me lean toward this view.

Or perhaps the cup-holders aren't even considered during design, but are added on at the last minute, because cars didn't used to have cup-holders at all and so that's not part of the design process. Perhaps automakers have internalized their process of producing and selling cars, and they can't conceive of adding a new element to that process, at least not until all the old automakers die out.

My priors say that it's more likely that I'm imagining the whole thing, that I selectively remember reviews complaining about cup-holders because of my own preferences, than that there has been a massive, systematic cognitive failure on the part of all the world's auto-makers, spanning 20 years, during which many of them somehow failed to observe, comprehend, or address this trivially-simple complaint of their customers, despite the billions of dollars at stake.

Am I?

The chapter on judgment under uncertainty in the (excellent) new Oxford Handbook of Cognitive Psychology has a handy little section on recent critiques of the "heuristics and biases" tradition. It also discusses problems with the somewhat-competing "fast and frugal heuristics" school of thought, but for now let me just quote the section on heuristics and biases (pp. 608-609):

The heuristics and biases program has been highly influential; however, some have argued that in recent years the influence, at least in psychology, has waned (McKenzie, 2005). This waning has been due in part to pointed critiques of the approach (e.g., Gigerenzer, 1996). This critique comprises two main arguments: (1) that by focusing mainly on coherence standards [e.g. their rationality given the subject's other beliefs, as contrasted with correspondence standards having to do with the real-world accuracy of a subject's beliefs] the approach ignores the role played by the environment or the context in which a judgment is made; and (2) that the explanations of phenomena via one-word labels such as availability, anchoring, and representativeness are vague, insufficient, and say nothing about the processes underlying judgment (see Kahneman, 2003; Kahneman & Tversky, 1996 for responses to this critique).

The accuracy of some of the heuristics proposed by Tversky and Kahneman can be compared to correspondence criteria (availability and anchoring). Thus, arguing that the tradition only uses the “narrow norms” (Gigerenzer, 1996) of coherence criteria is not strictly accurate (cf. Dunwoody, 2009). Nonetheless, responses in famous examples like the Linda problem can be reinterpreted as sensible rather than erroneous if one uses conversational or pragmatic norms rather than those derived from probability theory (Hilton, 1995). For example, Hertwig, Benz and Krauss (2008) asked participants which of the following two statements is more probable:

[X] The percentage of adolescent smokers in Germany decreases at least 15% from current levels by September 1, 2003.

[X&Y] The tobacco tax in Germany is increased by 5 cents per cigarette and the percentage of adolescent smokers in Germany decreases at least 15% from current levels by September 1, 2003.

According to the conjunction rule, [X&Y cannot be more probable than X] and yet the majority of participants ranked the statements in that order. However, when subsequently asked to rank order four statements in order of how well each one described their understanding of X&Y, there was an overwhelming tendency to rank statements like “X and therefore Y” or “X and X is the cause for Y” higher than the simple conjunction “X and Y.” Moreover, the minority of participants who did not commit the conjunction fallacy in the first judgment showed internal coherence by ranking “X and Y” as best describing their understanding in the second judgment.These results suggest that people adopt a causal understanding of the statements, in essence ranking the probability of X, given Y as more probable than X occurring alone. If so, then arguably the conjunction “error” is no longer incorrect. (See Moro, 2009 for extensive discussion of the reasons underlying the conjunction fallacy, including why “misunderstanding” cannot explain all instances of the fallacy.)

The “vagueness” argument can be illustrated by considering two related phenomena: the gambler’s fallacy and the hot-hand (Gigerenzer & Brighton, 2009). The gambler’s fallacy is the tendency for people to predict the opposite outcome after a run of the same outcome (e.g., predicting heads after a run of tails when flipping a fair coin); the hot-hand, in contrast, is the tendency to predict a run will continue (e.g., a player making a shot in basketball after a succession of baskets; Gilovich, Vallone, & Tversky, 1985). Ayton and Fischer (2004) pointed out that although these two behaviors are opposite - ending or continuing runs - they have both been explained via the label “representativeness.” In both cases a faulty concept of randomness leads people to expect short sections of a sequence to be “representative” of their generating process. In the case of the coin, people believe (erroneously) that long runs should not occur, so the opposite outcome is predicted; for the player, the presence of long runs rules out a random process so a continuation is predicted (Gilovich et al., 1985). The “representativeness” explanation is therefore incomplete without specifying a priori which of the opposing prior expectations will result. More important, representativeness alone does not explain why people have the misconception that random sequences should exhibit local representativeness when in reality they do not (Ayton & Fischer, 2004).

My thanks to MIRI intern Stephen Barnes for transcribing this text.

My paper "General purpose intelligence: arguing the Orthogonality thesis" has been accepted for publication in the December edition of Analysis and Metaphysics. Since that's some time away, I thought I'd put the final paper up here; the arguments are similar to those here, but this is the final version, for critique and citation purposes.

General purpose intelligence: arguing the Orthogonality thesis

STUART ARMSTRONG
stuart.armstrong@philosophy.ox.ac.uk
Future of Humanity Institute, Oxford Martin School
Philosophy Department, University of Oxford

In his paper “The Superintelligent Will”, Nick Bostrom formalised the Orthogonality thesis: the idea that the final goals and intelligence levels of artificial agents are independent of each other. This paper presents arguments for a (narrower) version of the thesis. It proceeds through three steps. First it shows that superintelligent agents with essentially arbitrary goals can exist in our universe – both as theoretical impractical agents such as AIXI and as physically possible real-world agents. Then it argues that if humans are capable of building human-level artificial intelligences, we can build them with an extremely broad spectrum of goals. Finally it shows that the same result holds for any superintelligent agent we could directly or indirectly build. This result is relevant for arguments about the potential motivations of future agents: knowing an artificial agent is of high intelligence does not allow us to presume that it will be moral, we will need to figure out its goals directly.

Keywords: AI; Artificial Intelligence; efficiency; intelligence; goals; orthogonality

1                       The Orthogonality thesis

Scientists and mathematicians are the stereotypical examples of high intelligence humans. But their morality and ethics have been all over the map. On modern political scales, they can be left- (Oppenheimer) or right-wing (von Neumann) and historically they have slotted into most of the political groupings of their period (Galois, Lavoisier). Ethically, they have ranged from very humanitarian (Darwin, Einstein outside of his private life), through amoral (von Braun) to commercially belligerent (Edison) and vindictive (Newton). Few scientists have been put in a position where they could demonstrate genuinely evil behaviour, but there have been a few of those (Teichmüller, Philipp Lenard, Ted Kaczynski, Shirō Ishii).

This is the introduction to a paper I started writing long ago, but have since given up on. The paper was going to be an overview of methods for improving human rationality through cognitive change. Since it contains lots of handy references on rationality, I figured I'd publish it, in case it's helpful to others.

1. Introduction

During the last half-century, cognitive scientists have catalogued dozens of common errors in human judgment and decision-making (Griffin et al. 2012; Gilovich et al. 2002). Stanovich (1999) provides a sobering introduction:

For example, people assess probabilities incorrectly, they display confirmation bias, they test hypotheses inefficiently, they violate the axioms of utility theory, they do not properly calibrate degrees of belief, they overproject their own opinions onto others, they allow prior knowledge to become implicated in deductive reasoning, they systematically underweight information about nonoccurrence when evaluating covariation, and they display numerous other information-processes biases...

The good news is that researchers have also begun to understand the cognitive mechanisms which produce these errors (Kahneman 2011; Stanovich 2010), they have found several "debiasing" techniques that groups or individuals may use to partially avoid or correct these errors (Larrick 2004), and they have discovered that environmental factors can be used to help people to exhibit fewer errors (Thaler and Sunstein 2009; Trout 2009).

This "heuristics and biases" research program teaches us many lessons that, if put into practice, could improve human welfare. Debiasing techniques that improve human rationality may be able to decrease rates of violence caused by ideological extremism (Lilienfeld et al. 2009). Knowledge of human bias can help executives make more profitable decisions (Kahneman et al. 2011). Scientists with improved judgment and decision-making skills ("rationality skills") may be more apt to avoid experimenter bias (Sackett 1979). Understanding the nature of human reasoning can also improve the practice of philosophy (Knobe et al. 2012; Talbot 2009; Bishop and Trout 2004; Muehlhauser 2012), which has too often made false assumptions about how the mind reasons (Weinberg et al. 2001; Lakoff and Johnson 1999; De Paul and Ramsey 1999). Finally, improved rationality could help decision makers to choose better policies, especially in domains likely by their very nature to trigger biased thinking, such as investing (Burnham 2008), military command (Lang 2011; Williams 2010; Janser 2007), intelligence analysis (Heuer 1999), or the study of global catastrophic risks (Yudkowsky 2008a).

But is it possible to improve human rationality? The answer, it seems, is "Yes." Lovallo and Sibony (2010) showed that when organizations worked to reduce the effect of bias on their investment decisions, they achieved returns of 7% or higher. Multiple studies suggest that a simple instruction to "think about alternative hypotheses" can counteract overconfidence, confirmation bias, and anchoring effects, leading to more accurate judgments (Mussweiler et al. 2000; Koehler 1994; Koriat et al. 1980). Merely warning people about biases can decrease their prevalence, at least with regard to framing effects (Cheng and Wu 2010), hindsight bias (Hasher et al. 1981; Reimers and Butler 1992), and the outcome effect (Clarkson et al. 2002). Several other methods have been shown to meliorate the effects of common human biases (Larrick 2004). Judgment and decision-making appear to be skills that can be learned and improved with practice (Dhami et al. 2012).

In this article, I first explain what I mean by "rationality" as a normative concept. I then review the state of our knowledge concerning the causes of human errors in judgment and decision-making (JDM). The largest section of our article summarizes what we currently know about how to improve human rationality through cognitive change (e.g. "rationality training"). We conclude by assessing the prospects for improving human rationality through cognitive change, and by recommending particular avenues for future research.

I sometimes go around saying that the fundamental question of rationality is Why do you believe what you believe?

-- Eliezer in Quantum Non-Realism

I was much impressed when they finally came out with a PC version of DragonBox, and I got around to testing it on some children I knew. Two kids, one of them four and the other eight years old, ended up blazing through several levels of solving first-degree equations while having a lot of fun doing so, even though they didn't know what it was that they were doing. That made me think that there has to be some way of making a computer game that would similarly teach rationality skills at the 5-second level. Some game where you would actually be forced to learn useful skills if you wanted to make progress.

After playing around with some ideas, I hit upon the notion of making a game centered around the Fundamental Question. I'm not sure whether this can be made to work, but it seems to have promise. The basic idea: you are required to figure out the solution to various mysteries by collecting various kinds of evidence. Some of the sources of evidence will be more reliable than others. In order to hit upon the correct solution, you need to consider where each piece of evidence came from, and whether you can rely on it.

Gameplay example

Now, let's go into a little more detail. Let's suppose that the game has a character called Bob. Bob tells you that tomorrow, eight o'clock, there will be an assassination attempt on Market Square. The fact that Bob has told you this is evidence for the claim being true, so the game automatically records the fact that you have such a piece of evidence, and that it came from Bob.

(Click on the pictures in case you don't see them properly.)

But how does Bob know that? You ask, and it turns out that Alice told him. So next, you go and ask Alice. Alice is confused and says that she never said anything about any assassination attempt: she just said that something big is going to be happen at the Market Square at that time, she heard it from the Mayor. The game records two new pieces of evidence: Alice's claim of something big happening at the Market Square tomorrow (which she heard from the Mayor), and her story of what she actually told Bob. Guess that Bob isn't a very reliable source of evidence: he has a tendency to come up with fancy invented details.

Or is he? After all, your sole knowledge about Bob being unreliable is that Alice claims she never said what Bob says she said. But maybe Alice has a grudge against Bob, and is intentionally out to make everyone disbelieve him. Maybe it's Alice who's unreliable. The evidence that you have is compatible with both hypotheses. At this point, you don't have enough information to decide between them, but the game lets you experiment with setting either of them as "true" and seeing the implications of this on your belief network. Or maybe they're both true - Bob is generally unreliable, and Alice is out to discredit him. That's another possibility that you might want to consider. In any case, the claim that there will be an assassination tomorrow isn't looking very likely at the moment.

Actually, having the possibility for somebody lying should probably be a pretty late-game thing, as it makes your belief network a lot more complicated, and I'm not sure whether this thing should display numerical probabilities at all. Instead of having to juggle the hypotheses of "Alice lied" and "Bob exaggerates things", the game should probably just record the fact that "Bob exaggerates things". But I spent a bunch of time making these pictures, and they do illustrate some of the general principles involved, so I'll just use them for now.

Game basics

So, to repeat the basic premise of the game, in slightly more words this time around: your task is to figure out something, and in order to do so, you need to collect different pieces of evidence. As you do so, the game generates a belief network showing the origin and history of the various pieces of evidence that you've gathered. That much is done automatically. But often, the evidence that you've gathered is compatible with many different hypotheses. In those situations, you can experiment with different ways of various hypotheses being true or false, and the game will automatically propagate the consequences of that hypothetical through your belief network, helping you decide what angle you should explore next.

Of course, people don't always remember the source of their knowledge, or they might just appeal to personal experiences. Or they might lie about the sources, though that will only happen at the more advanced levels.

As you proceed in the game, you will also be given access to more advanced tools that you can use for making hypothetical manipulations to the belief network. For example, it may happen that many different characters say that armies of vampire bats tend to move about at full moon. Since you hear that information from many different sources, it seems reliable. But then you find out that they all heard it from a nature documentary on TV that aired a few weeks back. This is reflected in your belief graph, as the game modifies it to show that all of those supposedly independent sources can actually be tracked back to a single one. That considerably reduces the reliability of the information.

But maybe you were already suspecting that the sources might not be independent? In that case, it would have been nice if the belief graph interface would let you postulate this beforehand, and see how big of an effect it would make on the plausibility of the different hypotheses if they were in fact reliant on each other. Once your character learns the right skills, it becomes possible to also add new hypothetical connections to the belief graph, and see how this would influence your beliefs. That will further help you decide what possibilities to explore and verify.

Because you can't explore every possible eventuality. There's a time limit: after a certain amount of moves, a bomb will go off, the aliens will invade, or whatever.

The various characters are also more nuanced than just "reliable" or "not reliable". As you collect information about the various characters, you'll figure out their mindware, motivations, and biases. Somebody might be really reliable most of the time, but have strong biases when it comes to politics, for example. Others are out to defame others, or invent fancy details to all the stories. If you talk to somebody you don't have any knowledge about yet, you can set a prior on the extent that you rely on their information, based on your experiences with other people.

You also have another source of evidence: your own intuitions and experience. As you get into various situations, a source of evidence that's labeled simply "your brain" will provide various gut feelings and impressions about things. The claim that Alice presented doesn't seem to make sense. Bob feels reliable. You could persuade Carol to help you if you just said this one thing. But in what situations, and for what things, can you rely on your own brain? What are your own biases and problems? If you have a strong sense of having heard something at some point, but can't remember where it was, are you any more reliable than anyone else who can't remember the source of their information? You'll need to figure all of that out.

As the game progresses to higher levels, your own efforts will prove insufficient for analyzing all the necessary information. You'll have to recruit a group of reliable allies, who you can trust to analyze some of the information on their own and report the results to you accurately. Of course, in order to make better decisions, they'll need you to tell them your conclusions as well. Be sure not to report as true things that you aren't really sure about, or they will end up drawing the wrong conclusions and focusing on the wrong possibilities. But you do need to condense your report somewhat: you can't just communicate your entire belief network to them.

Hopefully, all of this should lead to player learning on a gut level things like:

• Consider the origin of your knowledge: Obvious.
• Visualizing degrees of uncertainty: In addition to giving you a numerical estimate about the probability of something, the game also color-codes the various probabilities and shows the amount of probability mass associated with your various beliefs.
• Considering whether different sources really are independent: Some sources which seem independent won't actually be that, and some which seem dependent on each other won't be.
• Value of information: Given all the evidence you have so far, if you found out X, exactly how much would it change your currently existing beliefs? You can test this and find out, and then decide whether it's worth finding out.
• Seek disconfirmation: A lot of things that seem true really aren't, and acting on flawed information can cost you.
• Prefer simpler theories: Complex, detailed hypotheses are more likely to be wrong in this game as well.
• Common biases: Ideally, the list of biases that various characters have is derived from existing psychological research on the topic. Some biases are really common, others are more rare.
• Epistemic hygiene: Pass off wrong information to your allies, and it'll cost you.
• Seek to update your beliefs: The game will automatically update your belief network... to some extent. But it's still possible for you to assign mutually exclusive events probabilities that sum to more than 1, or otherwise have conflicting or incoherent beliefs. The game will mark these with a warning sign, and it's up to you to decide whether this particular inconsistency needs to be resolved or not.
• Etc etc.
  

Design considerations

It's not enough for the game to be educational: if somebody downloads the game because it teaches rationality skills, that's great, but we want people to also play it because it's fun. Some principles that help ensure that, as well as its general utility as an educational aid, include:

• Provide both short- and medium-term feedback: Ideally, there should be plenty of hints for how to find out the truth about something by investigating just one more thing: then the player can find out whether your guess was correct. It's no fun if the player has to work through fifty decisions before finding out whether they made the right move: they should get constant immediate feedback. At the same time, the player's decisions should be building up to a larger goal, with uncertainty about the overall goal keeping them interested.
• Don't overwhelm the player: In a game like this, it would be easy to throw a million contradictory pieces of evidence at the player, forcing them to go through countless of sources of evidence and possible interactions and have no clue of what they should be doing. But the game should be manageable. Even if it looks like there is a huge messy network of countless pieces of contradictory evidence, it should be possible to find the connections which reveal the network to be relatively simple after all. (This is not strictly realistic, but necessary for making the game playable.)
• Introduce new gameplay concepts gradually: Closely related to the previous item. Don't start out with making the player deal with every single gameplay concept at once. Instead, start them out in a trusted and safe environment where everyone is basically reliable, and then begin gradually introducing new things that they need to take into account.
• No tedium: A game is a series of interesting decisions. The game should never force the player to do anything uninteresting or tedious. Did Alice tell Bob something? No need to write that down, the game keeps automatic track of it. From the evidence that has been gathered so far, is it completely obvious what hypothesis is going to be right? Let the player mark that as something that will be taken for granted and move on.
• No glued-on tasks: A sign of a bad educational game is that the educational component is glued on to the game (or vice versa). Answer this exam question correctly, and you'll get to play a fun action level! There should be none of that - the educational component should be an indistinguishable part of the game play.
• Achievement, not fake achievement: Related to the previous point. It would be easy to make a game that wore the attire of rationality, and which used concepts like "probability theory", and then when your character leveled up he would get better probability attacks or whatever. And you'd feel great about your character learning cool stuff, while you yourself learned nothing. The game must genuinely require the player to actually learn new skills in order to get further.
• Emotionally compelling: The game should not be just an abstract intellectual exercise, but have an emotionally compelling story as well. Your choices should feel like they matter, and characters should be in risk of dying if you make the wrong decisions.
• Teach true things: Hopefully, the players should take the things that they've learned from the game and apply them to their daily lives. That means that we have a responsibility not to teach them things which aren't actually true.
• Replayable: Practice makes perfect. At least part of the game world needs to be randomly generated, so that the game can be replayed without a risk of it becoming boring because the player has memorized the whole belief network.

What next?

What you've just read is a very high-level design, and a quite incomplete one at that: I've spoken on the need to have "an emotionally compelling story", but said nothing about the story or the setting. This should probably be something like a spy or detective story, because that's thematically appropriate for a game which is about managing information; and it might be best to have it in a fantasy setting, so that you can question the widely-accepted truths of that setting without needing to get on anyone's toes by questioning widely-accepted truths of our society.

But there's still a lot of work that remains to be done with regard to things like what exactly does the belief network look like, what kinds of evidence can there be, how does one make all of this actually be fun, and so on. I mentioned the need to have both short- and medium-term feedback, but I'm not sure of how that could be achieved, or whether this design lets you achieve it at all. And I don't even know whether the game should show explicit probabilities.

And having a design isn't enough: the whole thing needs to be implemented as well, preferably while it's still being designed in order to take advantage of agile development techniques. Make a prototype, find some unsuspecting testers, spring it on them, revise. And then there are the graphics and music, things for which I have no competence for working on.

I'll probably be working on this in my spare time - I've been playing with the idea of going to the field of educational games at some point, and want the design and programming experience. If anyone feels like they could and would want to contribute to the project, let me know.

EDIT: Great to see that there's interest! I've created a mailing list for discussing the game. It's probably easiest to have the initial discussion here, and then shift the discussion to the list.

Related: 23 Cognitive Mistakes that make People Play Bad Poker

Followed by: Situational Awareness And You

If epistemic rationality is the art of updating one's beliefs based on new evidence to better correspond with reality, the zeroth skillset of epistemic rationality-- the one that enables all other skills to function-- is that of situational awareness. Situational awareness-- sometimes referred to as "situation awareness" or simply "SA"-- is the skillset and related state of mind that allows one to effectively perceive the world around them.

One might ask how this relates to rationality at all. The answer is simple. Just as the skill of lucid dreaming is near-useless without dream recall,^[1] the skills of updating based on evidence and actually changing your mind are near-useless without good awareness skills-- after all, you can't update based on evidence that you haven't collected! A high degree of situational awareness is thus an important part of one's rationalist toolkit, as it allows you to notice evidence about the world around you that you would otherwise miss. At times, this evidence can be of critical importance. I can attest that I have personally saved the lives of friends on two occasions thanks to good situational awareness, and have saved myself from serious injury or death many times more.

Situational awareness is further lauded by elite military units, police trainers, criminals, intelligence analysts, and human factors researchers. In other words, people who have to make very important-- often life-or-death-- decisions based on limited information consider situational awareness a critical skill. This should tell us something-- if those individuals for whom correct decisions are most immediately relevant all stress the importance of situational awareness, it may be a more critical skill than we realize.

Unfortunately, the only discussion of situational awareness that I've seen on LessWrong or related sites has been a somewhat oblique reference in Louie Helm's "roadmap of errors" from 23 Cognitive Mistakes that make People Play Bad Poker.^[2] I believe that situational awareness is important enough that it merits an explicit sequence of posts on its advantages and how to cultivate it, and this post will serve as the introduction to that sequence.

The first post in the sequence, unimaginatively titled "Situational Awareness and You," will be posted within the week. Other planned posts include "Cultivating Awareness," "How to Win a Duel," "Social Awareness," "Be Aware of Your Reference Class," "Signaling and Predation," and "Constant Vigilance!"

If you have any requests for things to add, general questions about the sequence, meta-thoughts about SA, and so on, this post is an appropriate place for that discussion; as this is primarily a meta post, it has been posted to Discussion. Core posts in the sequence will be posted to Main.

[1] What good are lucid dreams if you can't remember them?

[2] This is a very useful summary and you should read it even if you don't play poker.

A friend has been asking my views on the likelihood that there's anything to a correlation between changing levels of lead in paint (and automotive exhaust) and the levels of crime. He quoted from a Reason Blog:

So Nevin dove in further, digging up detailed data on lead emissions and crime rates to see if the similarity of the curves was as good as it seemed. It turned out to be even better: In a 2000 paper (PDF) he concluded that if you add a lag time of 23 years, lead emissions from automobiles explain 90 percent of the variation in violent crime in America. Toddlers who ingested high levels of lead in the '40s and '50s really were more likely to become violent criminals in the '60s, '70s, and '80s.

I responded with the following:

Sounds like a stretch to me. I'd want to hear that they didn't test more than 5 other hypothesis before coming to that conclusion, or the p value was far better than .05. I kind of doubt that either is the case. 

He's apparently continued to pursue the question, and just forwarded these remarks from Steven Pinker that I thought were very illuminating, and probably deserve a place in this community's toolkit for skeptics. Pinker's main point is that the association between Lead and crime is a long tenuous chain of suppositions, and several of the intermediate points should be far easier to measure. Finding correlations at this distance is not very informative.

http://stevenpinker.com/files/pinker/files/pinker_comments_on_lead_removal_and_declining_crime.pdf

Does the phrase "long-chain correlation" stick in your head and make it easier to dismiss this kind of argument?

Related: Reference Class of the Unclassreferenceable, Generalizing From One Example

Many people try to predict the future. Few succeed.

One common mistake made in predicting the future is to simply take a current trend and extrapolate it forward, as if it was the only thing that mattered-- think, for instance, of the future described by cyberpunk fiction, with sinister (and often Japanese) multinational corporations ruling the world. Where does this vision of the future stem from?

Bad or lazy predictions from the 1980s, when sinister multinational corporations (and often Japanese ones) looked to be taking over the world.^[1]

Similar errors have been committed by writers throughout history. George Orwell thought 1984 was an accurate prediction of the future, seeing World War II as inevitably bringing socialist revolution to the United Kingdom and predicting that the revolutionary ideals would then be betrayed in England as they were in Russia. Aldous Huxley agreed with Orwell but thought that the advent of hypnosis and psychoconditioning would cause the dystopia portrayed in 1984 to evolve into that he described in Brave New World. In today's high school English classes, these books are taught as literature, as well-written stories-- the fact that the authors took their ideas seriously would come as a surprise to many high school students, and their predictions would look laughably wrong.

Were such mistakes confined solely to the realm of fiction, they would perhaps be considered amusing errors at best, reflective of the sorts of mishaps that befall unstudied predictions. Unfortunately, they are not. Purported "experts" make just the same sort of error regularly, and failed predictions of this sort often have negative consequences in reality.

For instance, in 1999 two economists published the book Dow 36,000, predicting that stocks were about to reach record levels; the authors of the book were so wrapped up in recent gains to the stock market that they assumed that such gains were in fact the new normal state of affairs, that the market hadn't corrected for this yet, and that once stocks were correctly perceived as safe investments the market would skyrocket. This not only did not happen, but the dot-com bubble burst shortly after the book was published.^[2] Anyone following the market advice from this book lost big.

In 1968, the biologist Paul R. Ehrlich, seeing disturbing trends in world population growth, wrote a book called The Population Bomb, in which he forecast (among other things) that "The battle to feed all of humanity is over. In the 1970s hundreds of millions of people will starve to death in spite of any crash programs embarked upon now." Later, Ehrlich doubled down on this prediction with claims such as  "By the year 2000 the United Kingdom will be simply a small group of impoverished islands, inhabited by some 70 million hungry people ... If I were a gambler, I would take even money that England will not exist in the year 2000."

Based on these predictions, Ehrlich advocated cutting off food aid to India and Egypt in favor of preserving food supplies for nations that were not "lost causes;" luckily, his policies were not adopted, as they would have resulted in mass starvation in the countries suddenly deprived of aid. Instead, food aid continued, and as population grew, food production did as well. Contrary to the increase in starvation and global death rates predicted by Ehrlich, global death rates decreased, the population increased by more than Ehrlich had predicted would lead to disaster, and the average amount of calories consumed per person increased as well.^[3]

So what, then, is the weakness that causes these analysts to make such errors?

Well, just as you can generalize from one example when evaluating others and hence fail to understand those around you, you can generalize from one trend or set of trends when making predictions and hence fail to understand the broader world. This is a special case of the classic problem where "to a man with a hammer, everything looks like a nail;" if you are very familiar with one trend, and that's all you take into account with your future forecasts, you're bound to be wrong if that trend ends up not eating the world.

On the other hand, the trend sometimes does eat the world. It's very easy to find long lists of buffoonish predictions where someone woefully understimated the impact of a new technology.^[4] Further, determining exactly when and where a trend is going to stop is quite difficult, and most people are incompetent at it, even at a professional level-- if this were easy, the stock market would look extraordinarily different!

So my advice to those who would predict the future is simple. Don't generalize from one trend or even one group of trends. Especially beware of viewing evidence that seems to support your predictions as evidence that other people's predictions must be wrong-- the notebook of rationality cares not for what "side" things are on, but rather for what is true. Even if the trend you're relying on does end up being the "next big thing," the rest of the world will have a voice as well.^[5]

[1] I predict that the work of Cory Doctorow and those like him will seem similarly dated a decade down the line, as the trends they're riding die down. If you're reading this during or after December 2022, please let me know what you think of this prediction.

[2] The authors are, of course, still employed in cushy think-tank positions.

[3]  Ehrlich has doubled down on his statements, now claiming that he was "way too optimistic" in The Population Bomb and that the world is obviously doomed.

[4] I personally enjoy the Bad Opinion Generator (warning: potentially addictive)

[5] Technically, this isn't always true. But you should assume it is unless you have extremely good reasons to believe otherwise, and even still I would be very careful before assuming that your thing is the thing.

This is the public group instrumental rationality diary for the week of January 7th. It's a place to record and chat about it if you have done, or are actively doing, things like:

EDIT: Based on criticism below, I am reconsidering how to proceed with this idea (or something in the neighbourhood).

A topic that has been on my mind recently is where, in our complicated lives, there might be low-hanging fruit ready to be picked by a motivated rationalist. Actual, practical, dollars-and-cents fruit.

In possibly-related news, here is how the writer of About.com's beginner's guide to investing describes the stock market:

Imagine you are partners in a private business with a man named Mr. Market. Each day, he comes to your office or home and offers to buy your interest in the company or sell you his [the choice is yours]. The catch is, Mr. Market is an emotional wreck. At times, he suffers from excessive highs and at others, suicidal lows. When he is on one of his manic highs, his offering price for the business is high as well, because everything in his world at the time is cheery. His outlook for the company is wonderful, so he is only willing to sell you his stake in the company at a premium. At other times, his mood goes south and all he sees is a dismal future for the company. In fact, he is so concerned, he is willing to sell you his part of the company for far less than it is worth. All the while, the underlying value of the company may not have changed - just Mr. Market's mood.

I have heard this narrative many times before, and I'd like to test whether it is accurate - and in particular, whether LWers can consistently beat the market.

The skeptic may well ask: why should LWers have an advantage? Why not go to the professionals - investment advisors? Also, isn't there a whole chapter in Kahneman about how even smart people suck at picking stocks? And what do you, simplicio, know about this anyway?

LWers may have an advantage by virtue of being educated about such topics as cognitive biases, sunk cost fallacy, probabilistic prediction, and expected utility - topics with which investment advisors et al. may or may not be familiar on a gut level. I am not sure if we're any better, but I'd like to test it. Also, if LW turns out to be any good at offering such advice, that advice would presumably be free, unlike that of yon advisor (fees tend to kill returns on investment - just ask anybody who uses Intrade). As for what I personally know - not very much yet. But I find competition very stimulating.

Accordingly, my proposal is for a contest: over the course of 2013, I will set up & maintain a Google Drive spreadsheet. This spreadsheet will be shared with contest participants. Each participant will have say $5,000 of play money to use "buying" (or "selling") stocks on the exchange of their choice. Contestants will record the date of purchase or sale, quantity, and preferably provide comments regarding why they are buying or selling.

At the end of this contest (Dec 31, 2013?), I will commit to Paypal the winner (defined as the person with the highest market valuation of play assets as of midnight on that date) the equivalent of $50 CAD in their local currency. In the unlikely event that I win, I will donate that $50 to the Against Malaria Foundation. (Above commitment does not take effect until I actually gauge interest in this contest, figure out an end date & rules etc., and decide to proceed. If anyone else wants to throw money in the pot, please do.)

The purposes of this post are therefore:

• to find out who is interested - please leave a comment below, and e-mail me at ispollock [at] gmail.com if you want in;
• to solicit constructive and destructive criticism of the project, especially from any local experienced investors (in particular, perhaps a one-year timeframe is too short for a meaningful contest? Also, real-world experience of transaction costs in buying and selling would be extremely helpful);
• to ask if anyone knows of a better software platform for the contest than Google Drive, or knows of any extremely helpful resources I should be reading/linking to.

This is the public group instrumental rationality diary for Christmas week. It's a place to record and chat about it if you have done, or are actively doing, things like:

I am emotionally excited and/or deeply hurt by what st_rev wrote recently. You better take me seriously because you've spent a lot of time reading my posts already and feel invested in our common tribe. Anecdote about how people are tribal thinkers.

That thing that happened shows that everything I was already advocating for is correct and necessary. Indeed it is time for everyone to put their differences aside and come together to carry out my recommended course of action. If you continue to deny what both you and I know in our hearts to be correct, you want everyone to die and I am defriending you.

I don't even know where to begin. This is what blueist ideology has been workign towards for decades if not millennia, but to see it written here is hard to stomach even for one as used to the depravity caused by such delusions as I am. The lack of socially admired virtues among its adherents is frightening. Here I introduce an elaborate explanation of how blueist domination is not just completely obvious and a constant thorn in the side of all who wish more goodness but is achieved by the most questionable means often citing a particular blogger or public intellectual who I read in order to show how smart I am and because people I admire read him too. Followed by an appeal to the plot of a movie. Anecdote from my personal life. If you are familiar with the obscure work of an academic taken out of context and this does not convince you then you are clearly an intolerant sexual deviant engaging in motivated cognition.

Consider well: do you want to be on the wrong side of history? If you persist, millions or billions of people you will never meet will be simultaneously mystified and appalled that an issue so obvious caused such needless contention. They will argue whether you were motivated more by stupidity, malice, raw interest, or if you were a helpless victim of the times in which you lived. Characters in fiction set in your era will inevitably be on (or at worst, join) the right side unless they are unredeemable villains. (Including historical figures who were on the other side, lest they lose all audience sympathy.).

Remember: it's much more important what hypothetical future people will consider right than what you or current people you respect do. And you and I both know they'll agree with me.

While sympathetic to this criticism I must signal my world-weariness and sophistication by writing several long paragraphs about how this is much too optimistic and we are in grave danger of a imminent and eternal takeover by our opponents. The only solution is to begin work on an organization dedicated to preventing this which happens to give me access to material resources and attractive females.

Ciphergoth proves to be the lone voice of reason by encouraging us to recall what we all learned on 9/11:

However, we must also consider if this is not also a lesson to us all; a lesson that my political views are correct.

http://www.adequacy.org/stories/2001.9.12.102423.271.html

Consider the following statement: two heads are better than one.

It seems obvious to me that several rationalists working together can, effectively, bring more precious brainpower to bear on a problem, than one rationalist working alone (otherwise, what would be the point of having a Less Wrong forum community? You might as well just leave it as a curated community blog and excise the discussion forums.). Further, due to various efforts (HPMOR especially) it appears that LW is inevitably growing. This makes it not only desirable to find ways to effectively get groups of rationalists to think together, but also increasingly necessary.

It is also desirable that methods of getting groups to think should be feasibly doable over the Internet. (I am aware that real-life meetups and stuff exist, but please be reminded that some people in the world do have to live in shitty little third-world countries and might not at all find it economically feasible to go to first-world countries with atrociously high costs-of-living)

So first, let us start with the current "best methods" of getting groups of Traditional Rationalists to coordinate and think, while avoiding groupthink effects that diminish our aggregate rationality. Hopefully, we can then use it as the basis of part of the art of rationalist group thinking. So I'll discuss:

• Disputation Arenas - procedures with centrifugal and centripetal phases
  1. Delphi Method - members secretly answer questions, non-member summarizer anonymizes answers, members read anonymized summary, repeat
  2. Prediction Market - members place stock bets on market, market settles on price, members react to price signal, repeat
  3. Nominal Group Technique - members write down ideas privately, non-member facilitator guides members in sharing ideas, non-member facilitator guides members in paring down and cleaning up ideas, members vote
• Conclusion - strengths and weaknesses of the various disputation arenas shown here

Disputation Arenas

One of my favorite SF authors, David Brin, talks a bit about what he calls "disputation arenas". I won't discuss his ideas here, since his concept of "disputation arena" is actually a relatively "new", raw, and relatively untested procedure - what I intend to discuss for now are things that have at least been studied more rigorously than just a bunch of blog posts (or personal website pages, whatever).

However, I do want to co-opt the term "Disputation Arena" for any process that tries to achieve the following:

• Avoid groupthink - actively search for information without settling too early on an option considered desirable by certain influential members
• Achieve consensus - designate some choice as the best, given current information known by the group members

We want our group rationality process to avoid groupthink (possibly at some expense of efficiency) because actual, real-world rationalists are not perfect Bayesian reasoners - two words: Robert Aumann. Because rationalists are not perfect, we do not expect a clear consensus to form after the end of the process (i.e. Aumann's does not necessarily apply), so the process we use must force some consensus to become visible.

One thing that David Brin discusses is the general division of the procedures into two "phases":

• Centrifugal phase - members of the group generate ideas separately.
• Centripetal phase - ideas are judged by the group together.

This seems to me to be a good way of labeling parts of any group-coordination process that attempts to avoid groupthink and achieve consensus.

In my personal research, I've found three things that attempt to achieve those two goals (avoid groupthink, achieve consensus) and which might (perhaps with a stretch) be considered as approximately having two phases (centrifugal and centripetal).

Delphi Method

The Delphi Method was originally developed by RAND Corporation (Project RAND at that time, and no relation to Ayn Rand) in order to get better predictions on future military hardware. It is currently used to get better utilization of current human wetware. (^.^)v

Delphi Method: How To Do It!

Pen and paper version:

1. A panel of experts is chosen, and a questionnaire is prepared.
2. Experts answer the questionnaire, giving the answers and also justifications and reasons for those answers.
3. Summarizer provides anonymous summaries of the expert's answers and justifications.
4. Experts read the summary, and may revise their answers/justifications. The process is repeated (with the same experts and questionnaire) for a set number of rounds, or until everyone gets bored, or the military bunker everyone is in gets nuked.

Internet version (Wikipedia:Real-time Delphi):

1. Username/passwords are chosen and emailed, and a questionnaire is prepared. The questionnaire in the online version is somewhat restricted, however. Here are some ideas:
   • Use a "poll" question. Experts select one of the choices given.
   • Use "multiple-choice" questions. Experts then select from a range of (e.g.) 1 for (strongly disagree) to 10 (strongly agree) for each possible choice.
   • Use "multiple-choice" questions, and also give different aspects such as "feasibility", "desirability", "good side-effects", etc. Experts answer 1 to 10 for each combination of choice and aspect.
2. Experts answer the online questionnaire. Aside from the numerical or selection (quantitative) answer, experts should also supply a short sentence or two justifying each answer (qualitative).
3. After the expert submits his or her answers, he or she is shown the current averages (for scoring-type questionnaires) or current poll results - this is the quantitative group answer. Expert is also shown (or provided links to) randomly-sorted (and randomly-chosen, if groups are very large and the number of answers may overwhelm a typical human) qualitative answers for each poll item / choice / choice+aspect - the qualitative group answers - for each score or aspect.
   • IMPORTANT: individual qualitative answers should not show the username of the expert who gave them!
   • In effect, the average (or poll results) plus the randomized sample of qualitative answers are a simple, anonymous, machine-generated summary.
4. Experts may change their own quantitative and/or qualitative answers at any time, and see the current quantitative group answers and qualitative group answers at any time after they have submitted their own answers.
5. The questionnaire is kept open until some specified time, or somebody hacks the server to put LOLcats instead.

Delphi Method: Analysis

Delphi methods avoid groupthink largely by anonymity: this avoids the bandwagon effect, the halo effect, and the mind-killer. Anonymity and constant feedback also encourage people to revise their positions in light of new information from their peers (by reducing consistency pressure): in non-anonymous face-to-face meetings, people tend to stick to their previously stated opinions, and to conform to the meeting leader(s) or their own bosses in the meeting. A lot of those effects is reduced by anonymity. Pen-and-paper form makes anonymity much easier, since the summary gets the tone and language patterns of the summarizer; some amount of anonymity is lost in the online version (since language patterns might theoretically be analyzed) but hopefully the small sample size (just a short sentence or two) can make language pattern analysis difficult. Note that randomizing the order of the comments in the online version is important, as this reduces the effects of anchoring; sorting by time or karma may increase groupthink due to anchoring on earlier comments, but if each expert sees different "first comments", then this bias gets randomized (hopefully into irrelevancy).

Delphi methods achieve consensus by the summary (which often serves as the "final output" when the process is finished). Arguably, the pen-and-paper version is better at achieving consensus due to the "turn-based" arrival of the summary, which makes the expert pay more attention to the summary, compared to the real-time online system.

The Delphi method's centrifugal phase is the expert's private answering of the questionnaire: each expert makes this decision, and provides the justification, without other's knowledge or help.

The Delphi method's centripetal phase is the act of summarizing, and having the experts read the summary.

Delphi Method: Other Thoughts

I think that forum polls, in general, can be easily adapted into online real-time Delphis by adding the following:

• Members will be required to give a short sentence or two (probably limited to say 200 chars or so) justifying their poll choice.
• Members should be allowed to change their poll choice and their justification at any time until the poll closes or the forum is hacked by LOLcat vandals.
• Members should be able to click on a poll choice on the poll results page to get a random anonymous sampling of the justifications that the other members have made in choosing that poll choice.

The procedure says "experts" but I think that in something more democratic than the military you're supposed to read that as "anyone who bothers to participate".

Prediction Market

Prediction markets are speculation markets built around bets on what things will happen in the future. They are also the core of Robin Hanson's Futarchy, and which you can see somewhere in the background of LW's favorite tentacle alien porn novella (O.o);;.

Prediction Market: How To Do It!

Pen and paper version:

1. Convince a trusted monetary institution to sell you "X is gonna happen" stock and "X is not gonna happen" stock for $1 a pair (i.e. $1 for a pair of contracts, one that says "If X happens, Monetary Institution pays you $1" and another that says "If X doesn't happen, Monetary Institution pays you $1", so the pair costs $1 total since X can't both happen and not happen). You may need to pay some sort of additional fee or something if the monetary institution is for-profit.
2. Sell the stock (i.e. the contract) you think is false for as high as you can get on the open market. Buy more stock of what you think is true from others who are willing to sell to you.
3. Just buy and sell stocks depending on what you think is the best prediction, based on what you hear on the news, gossip you hear from neighbors, and predictions from the tea leaves. Keep doing this until X definitely happens or X definitely does not happen (in which case you cash in your stock contracts, if you bet correctly on what happened), or a market crash results because someone discovers that the weak nuclear force actually allows you to make nuclear bombs out of orange juice, and Einstein and the gang were lying about it and distracting you by talking about dice-playing gods.

(what I described above is the simplest and most basic form I found; refer to the Wikipedia article for better elaborations)

Internet version:

1. Hack Intrade so that the topic you want to bet on is in their list of markets. Or better yet just hack Intrade and put one million dollars into your account.

Prediction Market: Analysis

Prediction markets avoid groupthink by utilizing the invisible hand. Someone selling you a stock might be an idiot who can't read the tea leaves properly. Or the seller might have knowledge you do not possess, so maybe buying the stock wasn't such a good idea after all? Remember: if you can't find who the sucker on the table is, that sucker is you!! You can't simply assume that what your neighbor says is true and you should sell as many stock of X as possible: maybe he or she is trying to take advantage of you to get your hard-earned cash. Groupthink in such a mistrusting environment gets hard to sustain. Prediction markets work better with very large groups of people, so that you get practical anonymity (although not perfect, in theory you or anyone else can keep track of who's selling to who; online versions are also likely to hide user identities). Anonymity in the prediction market also has the advantages previously described under Delphi Method above.

Prediction markets achieve consensus by utilizing the invisible hand. The price point of any sale serves as an approximate judgment of the epistemic probability of X occurring (or not occurring, depending on the contract that got sold). This gives a real-time signal on what the group of traders as a whole think the probability of X occurring is.

The prediction market's centrifugal phase is each individual trader's thought process as he or she considers whether to buy or sell stock, and at what price.

The prediction market's centripetal phase is any actual sale at any actual price point.

Prediction Market: Other Thoughts

Prediction markets are well-represented online; Intrade is just one of the more famous online prediction markets. Prediction markets appear to be the most popular and widely-known of the disputation arenas I've researched. These all tend to suggest that prediction markets are one of the better disputation arenas - but then remember that the Internet itself has no protection against groupthink.

Nominal Group Technique

Nominal group technique is a group decision-making process, appropriate for groups of many different sizes. This procedure's pen-and-paper form is faster than the pen-and-paper forms of the other disputation arenas discussed here.

Nominal Group Technique: How To Do It!

Pen and paper version:

1. The facilitator informs the group of the issue to be discussed.
2. Silent idea generation: group members are provided pen and paper, and are told to write down all ideas they can think of about the issue on the paper. They are given a fixed amount of time to do this (usually 10 to 15 minutes).
   • IMPORTANT: members are not allowed to discuss, show, or otherwise share their ideas with others during this stage. There's a reason it's called "silent".
3. Idea sharing: the facilitator asks group members, one at a time, to discuss their own ideas, until all members have shared their ideas. The facilitator writes the shared ideas into a whiteboard, or a similar location visible to all members.
   • IMPORTANT: debate is not allowed at this stage; only one member at a time can speak at this stage.
   • Group members may also add additional ideas and notes to their written-down ideas while waiting for their turn.
4. Group discussion: members may ask for clarification or further details about particular ideas shared by other group members. The group may agree to split ideas, or merge ideas, or group ideas into categories.
   • IMPORTANT: the facilitator must ensure that (1) all members participate, and (2) no single idea gets too much attention (i.e. all ideas must be discussed).
   • The discussion should be as neutral as possible, avoiding judgment or criticism.
   • The final result of this stage should be a set of options to be chosen among.
5. Ranking: members secretly rank the options from 1 (best) to N (worst), where N is the number of options generated in the previous stage. The facilitator then tallies the (anonymous) rankings (by adding the rankings for each option) and declares the option with the lowest total as the group consensus.

Unlike the previous procedures, which have been extrapolated into Internet versions, there is currently no Internet version of nominal group technique.

Nominal Group Technique: Analysis

Nominal group techniques avoid groupthink by the two "secret" steps: silent idea generation, and the secret ranking. Having members write down their ideas in the silent idea generation step helps them precommit to those ideas in the idea sharing step, even though more influential group members may present opposite or incompatible ideas. Although the group discussion step disallows explicit criticism of ideas, those criticisms are implicitly expressed during the secret ranking step (i.e. if you have a criticism of an idea, then you should rank it lower).

Nominal group techniques achieve consensus by the idea sharing, group discussion, and ranking steps. In particular, tallying of option rankings is the final consensus-achieving step.

The nominal group technique's centrifugal phase is largely the silent idea generation step, and is the most explicit centrifugal phase among the disputation arenas discussed here.

The nominal group technique's centripetal phase is largely the rest of the procedure.

Nominal Group Technique: Other Thoughts

A modified form of nominal group technique eats up a quarter of my recently-finished novel, Judge on a Boat, which I talked about on LessWrong here, and whose latest raw text source you can read online. Yes, this entire article is just a self-serving advertisement to garner interest in my novel o(^.^o)(o^.^)o.

Compared to the other procedures here, nominal group technique is more complicated and much more dependent on the centralized facilitator; the extreme dependency on the facilitator makes it difficult to create an automated online version. On the other hand, a small group of say 5 to 10 people can finish the nominal group technique in 1 to 2 hours; the other procedures tend to work better when done over several days, and are largely impossible (in pen-and-paper form) to do in a similar time frame. Even the real-time online versions of the other procedures are difficult to do within 2 hours. Prediction markets in particular tend to fail badly if too thin (i.e. not enough participants); for small groups with tight schedules, nominal group technique tends to be the best.

Conclusion

For small groups that need to make a decision within one or two hours, use nominal group technique. It's relatively unwieldy compared to the other disputation arenas, and is less ideal (it has fewer protections against groupthink, in particular), but is fast compared to the others. Also, one might consider parallelizing nominal group technique: split a large group into smaller, randomly-selected sub-groups, have each perform the procedure independently, and then have them send a representative that performs the idea sharing, group discussion, and ranking steps with other representatives (i.e. each sub-group's nominal group technique serves as the silent idea generation of the super-group of representatives). This tends to bias the super-group towards the agendas of the chosen representatives, but if speed is absolutely necessary for a large group, this may be the best you can do.

As mentioned above, prediction markets tend to fail badly if there are too few participants in the speculation market; use it only for extremely large groups that are impossible to coordinate otherwise. In addition, using prediction markets for policy decisions is effectively futarchy; you may want to see the (defunct?) futarchy_discuss Yahoo! group's message archives. In particular the earlier messages in the archive tend to discuss the general principles of prediction markets. Prediction markets are the most famous of the disputation arenas here, but remember that the Internet is not a decent disputation arena.

The Delphi methods seem to be a "dark horse" of sorts. I don't see much discussion online about Delphi methods; I'm not sure whether it's because it's been tried and rejected, or if it simply isn't well known enough to actually be tried by most people. I tend to suspect the latter, since if the universe were in the former case I would at least see some "Delphi Methods suck!!" blog posts.

Both prediction markets and Delphi methods are continuously repeated methods. At any time, the procedure may be stopped or repeated in order to make decisions. However, unlike the nominal group techniques, both are targeted more towards generating advice for decision-makers, rather than making actual decisions themselves.

It may be possible to organize a large, hierarchical group (say a company) with a prediction market for the rank-and-file, some key experts (who should be aware of the prediction market's results) running a Delphi method, and the key decision-making individuals (who read the Delphi method's report) at the top who form a decision using nominal group technique. For more democratic processes, a "poll-style" real-time online Delphi method by itself may work.

This is the public group instrumental rationality diary for the week of December 10th. It's a place to record and chat about it if you have done, or are actively doing, things like:

Related to: Knowing About Biases Can Hurt People

HT: Marginal Revolution

Paper.

Social psychologists have identified various plausible sources of ideological polarization over climate change, gun violence, national security, and like societal risks. This paper reports a study of three of them: the predominance of heuristic-driven information processing by members of the public; ideologically motivated cognition; and personality-trait correlates of political conservativism. The results of the study suggest reason to doubt two common surmises about how these dynamics interact. First, the study presents both observational and experimental data inconsistent with the hypothesis that political conservatism is distinctively associated with closed-mindedness: conservatives did no better or worse than liberals on an objective measure of cognitive reflection; and more importantly, both demonstrated the same unconscious tendency to fit assessments of empirical evidence to their ideological predispositions. Second, the study suggests that this form of bias is not a consequence of overreliance on heuristic or intuitive forms of reasoning; on the contrary, subjects who scored highest in cognitive reflection were the most likely to display ideologically motivated cognition. These findings corroborated the hypotheses of a third theory, which identifies motivated cognition as a form of information processing that rationally promotes individuals’ interests in forming and maintaining beliefs that signify their loyalty to important affinity groups. The paper discusses the normative significance of these findings, including the need to develop science communication strategies that shield policy-relevant facts from the influences that turn them into divisive symbols of identity.

The Worst-Run Big City in the U.S.

A six page article that reads as a very interesting autopsy of what institutional dysfunction in the intersection of government and non-profits looks like. I recommend reading the whole thing.

Minus the alleged harassment, city government is filled with Yomi Agunbiades — and they're hardly ever disciplined, let alone fired. When asked, former Board of Supervisors President Aaron Peskin couldn't remember the last time a higher-up in city government was removed for incompetence. "There must have been somebody," he said at last, vainly searching for a name.

Accordingly, millions of taxpayer dollars are wasted on good ideas that fail for stupid reasons, and stupid ideas that fail for good reasons, and hardly anyone is taken to task.

The intrusion of politics into government pushes the city to enter long-term labor contracts it obviously can't afford, and no one is held accountable. A belief that good intentions matter more than results leads to inordinate amounts of government responsibility being shunted to nonprofits whose only documented achievement is to lobby the city for money. Meanwhile, piles of reports on how to remedy these problems go unread. There's no outrage, and nobody is disciplined, so things don't get fixed.

You don't say?

In 2007, the Department of Children, Youth, and Families (DCYF) held a seminar for the nonprofits vying for a piece of $78 million in funding. Grant seekers were told that in the next funding cycle, they would be required — for the first time — to provide quantifiable proof their programs were accomplishing something.

The room exploded with outrage. This wasn't fair. "What if we can bring in a family we've helped?" one nonprofit asked. Another offered: "We can tell you stories about the good work we do!" Not every organization is capable of demonstrating results, a nonprofit CEO complained. He suggested the city's funding process should actually penalize nonprofits able to measure results, so as to put everyone on an even footing. Heads nodded: This was a popular idea.

Reading this I had to bite my hand in frustration.

There are two lessons here. First, many San Francisco nonprofits believe they're entitled to money without having to prove that their programs work. Second, until 2007, the city agreed. Actually, most of the city still agrees. DCYF is the only city department that even attempts to track results. It's the model other departments are told to aspire to.

But Maria Su, DCYF's director, admitted that accountability is something her department still struggles with. It can track "output" — what a nonprofit does, how often, and with how many people — but it can't track "outcomes." It can't demonstrate that these outputs — the very things it pays nonprofits to do — are actually helping anyone.

"Believe me, there is still hostility to the idea that outcomes should be tracked," Su says. "I think we absolutely need to be able to provide that level of information. But it's still a work in progress." In the meantime, the city is spending about $500 million a year on programs that might or might not work.

What the efficient charity movement has done so far looks much more impressive in light of this. Reading the rest of the article I think you can on your own identify the problems caused by lost purposes, applause lights and a dozen or so other faults we've explored here for years.

Discussions here are in many respects a comforting illusion, this is what humanity is like out there in the real world, almost at its best, well educated, wealthy and interested in the public good.

Yes it really is that bad.

This is the public group instrumental rationality diary for the week of November 27th. It's a place to record and chat about it if you have done, or are actively doing, things like:

Hi LessWrong! I'm a LWer from Melbourne, Australia, and I'm taking a 3 month road trip (with a friend) through parts of the United States. I figure I'd enjoy hanging out with some fellow rationalists while I'm over here! 

I attended the May Rationality minicamp in San Francisco (and made some friends who I'm hoping to meet up with again), but I've also heard good things about the LessWrong groups all over the United States. I'd like to meet some of the awesome people involved in these communities!

We've been planning this trip for a while now and have accommodation pretty much everywhere except for the second half of San Francisco. 

Itinerary

• 17th-21st Nov - Los Angeles, CA
• 21st-28th Nov - San Francisco, CA
• 28th Nov-1st Dec - Las Vegas, NV
• 2nd-3rd Dec - Flagstaff, AZ
• 3rd-7th Dec - Phoenix, AZ
• 7th-9th Dec - Santa Fe, NM
• 9th-10th Dec - El Paso, TX
• 10th-13th Dec - San Antonio, TX
• 13th-21st Dec - Austin, TX
• 21st-26th Dec - Dallas, TX
• 26th-29th Dec - San Antonio, TX
• 29th Dec-2nd Jan - New York City, NY
• 2nd-3rd Jan - San Antonio, TX
• 3rd-6th Jan - Houston, TX
• 6th-9th Jan - New Orleans, LA
• 9th-12th Jan - Memphis, TN
• 12th-15th Jan - Nashville, TN
• 15th-18th Jan - Atlanta, GA
• 18th-22nd Jan - Miami, FL
• 22nd-26th Jan - Orlando, FL
• 26th Jan-1st Feb - Washington DC
• 1st-4th Feb - Philadelphia, PA
• 4th-6th Feb - New York City, NY
• 6th-9th Feb - Mount Snow, VT
• 9th-13th Feb - Boston, MA
• 13th-15th Feb - New York City, NY
• 15th-26th Feb - Columbus, OH

If you're in one of these locations when I am, contact me! Either ahead of time or at short notice is fine. I'll be checking meetup posts and mailing lists for events that I can make it to as well, but if you happen to know of an event or meetup happening that fits the schedule, feel free to let me know in the comments.

Message or call me on 4242 394 657, email me at shokwave.sf@gmail.com - or you can leave a toplevel comment on this post, or message my LW account directly. Looking forward to meeting any and all of you!

This is the public group instrumental rationality diary for the week of October 29th. It's a place to record and chat about it if you have done, or are actively doing, things like:

Some time ago I learned of the metaphor of 'digging the bull's horn'. This might sound a little strange, since horns are mostly hollow, but imagine a bull's horn used to store black powder. In the beginning the work is easy and you can scoop out a lot powder with very little effort. As you dig down, though, each scoop yields less powder as you dig into the narrow part of the horn until the only way you can get out more powder is to turn the horn over a dump it out.

It's often the same way with learning. When you start out in a subject there is a lot to be learned (both in quantity of material you have not yet seen and in quantity of benefits you have to gain from the information), but as you dig deeper into a subject the useful insights come less often or are more limited in scope. Eventually you dig down so far that the only way to learn more is to discover new things that no one has yet learned (to stretch the metaphor, you have to add your own powder back to dig out).

It's useful to know that you're digging the bull's horn when learning because, unless you really enjoy a subject or have some reason to believe that contributing to it is worthwhile, you can know in advance that most of the really valuable insights you'll gain will come early on. If you want to benefit from knowing about as much stuff as possible, you'll often want to stop actively pursuing a subject unless you want to make a career out of it.

But, for a few subjects, this isn't true. Sometimes, as you continue to learn the last few hard things that don't seem to provide big, broadly-useful insights, you manage to accumulate a critical level of knowledge about the subject that opens up a whole new world of insights to you that were previously hidden. To push the metaphor, you eventually dig so deep that you come out the other side to find a huge pile of powder.

The Way seems to be one of those subjects you can dig past the end of: there are some people who have mastered The Way to such an extent that they have access to a huge range of benefits not available to those still digging the horn. But when it comes to other subjects, how do you know? Great insights could be hiding beyond currently obscure fields of study because no one has bothered to dig deep enough. Aside from having clear examples of people who came out the other side to give us reason to believe it's worth while to deep really deep on some subjects, is there any way we can make a good prediction about what subjects may be worth digging to the end of the bull's horn?

Is there a useful heuristic for detecting rationally-challenged texts (as in Web pages, forum posts, facebook comments) which takes relatively superficial attributes such as formatting choices, spelling errors, etc. as input? Something a casual Internet reader may use to detect possibly unworthy content so they can suspend their belief and research the matter further. Let's call them "text smells" (analogue to code smells), like:

1. too much emphasis in text (ALL CAPS, bold, color, exclamations, etc.);
2. walls of text;
3. little concrete data/links/references;
4. too much irrelevant data and references;
5. poor spelling and grammar;
6. obvious half-truths and misinformation.

Since many crackpots, pseudoscientific con artists, and conspiracy theorists seem to have cleaned up their Web sites in recent years, I wonder do these low-cost baloney detection tools might be of real value. Does anyone know of any studies or analyses of correlation between these basic metrics and the actual quality of the content? Can you think of some other smells typical of Web baloney?

Related to: Voting is like donating thousands of dollars to charity, Does My Vote Matter?

And voting adds legitimacy to it.

Thank you.

#annoyedbymotivatedcognition

A rather good 10 minute YouTube video presenting the results of several papers relevant to how conformity affects our thinking:

http://www.youtube.com/watch?v=TrNIuFrso8I

The papers mentioned are:

Sherif, M. (1935). A study of some social factors in perception. Archives of Psychology, 27(187), pp.17-22.

Asch, S.E. (1951). Effects of group pressure upon the modification and distortion of judgment. In H. Guetzkow (ed.) Groups, leadership and men. Pittsburgh, PA: Carnegie Press.
Asch, S.E. (1955). Opinions and social pressure. Scientific American, 193(5), pp.31-35.

Berns, G.S., Chappelow, J., Zink, C.F., Pagnoni, G., Martin-Skurski, M.E., and Richards, J. (2005) 'Neurobiological Correlates of Social Conformity and Independence During Mental Rotation' Biological Psychiatry, 58(3), pp.245-253.

Weaver, K., Garcia, S.M., Schwarz, N., & Miller, D.T. (2007) Inferring the popularity of an opinion from its familiarity: A repetitive voice can sound like a chorus. Journal of Personality and Social Psychology, 92(5), 821-833.

What techniques do other posters, here on LessWrong, use to monitor and counter these effects in their lives?

The video also lists some of the advantages to a society of having a certain amount of this effect in place.   Does anyone here conform too little?