I've read about a quarter of the sequences, but I'm not sure if this topic has been addressed on LessWrong before. If it has, let me know.

The Upward Scaling Importance of Rationality goes like this:

The more influence your thought process and decisions have, the more important it is that you're rationalist. In the grand scheme of things, it is relatively unimportant that a barback at a restaurant is a rationalist, and I say this having done that. It is extremely important that a leader of a highly influential company, or a president of a university or country is a rationalist. Their decisions affect thousands if not millions of people.

The more influential you are, the more your decisions have potential to screw over other people. Influence doesn't necessarily have to be in a management position: elementary school teachers and police officers are highly influential, even though they aren't in control of an organization. Influence can even be by virtue of the people you reach out to. A famous person with a large fanbase or a parent of a child prodigy, both have the capacity to influence the world with their decisions.

Though arguably, this can  be extended to anyone who votes.

So rationality scales upward: the more influential someone is, the more important is it they're rationalists. Neglecting this can have bad consequences.

In the book "How to Measure Anything" D. Hubbard presents a step-by-step method for calibrating your confidence intervals, which he has tested on hundreds of people, showing that it can make 90% of people almost perfect estimators within half a day of training.

I've been told that the Less Wrong and CFAR community is mostly not aware of this work, so given the importance of making good estimates to rationality, I thought it would be of interest.

(although note CFAR has developed its own games for training confidence interval calibration)

The main techniques to employ are:

Equivalent bet:

For each estimate imagine that you are betting $1000 on the answer being within your 90% CI. Now compare this to betting $1000 on a spinner where 90% of the time you win and 10% of the time you lose. Would you prefer to take a spin? If so, your range is too small and you need to increase it. If you decide to answer the question your range is too large and you need to reduce it. If you don’t mind whether you answer the question or take a spin then it really is your 90% CI.

Absurdity Test:

Start with an absurdly large range, maybe from minus infinity to plus infinity, and then begin reducing it based upon things you know to be highly unlikely or even impossible.

Avoid Anchoring:

Anchoring occurs when you think of a single answer to the question and then add an error around this answer; this often leads to ranges which are too narrow. Using the absurdity test is a good way to counter problems brought on by anchoring; another is to change how you look at your 90% CI. For a 90% CI there is a 10% chance that the answer lies outside your estimate, and if you split this there is a 5% chance that the answer is above your upper bound and a 5% chance that the answer is below your lower bound. By treating each bound separately, rephrase the question to read ‘is there a 95% chance that the answer is above my lower bound?’. If the answer is no, then you need to increase or decrease the bound as required. You can then repeat this process for the other bound.

Pros and cons:

Identify two pros and two cons for the range that you have given to help clarify your reasons for making this estimate.

Once you have used these techniques you can make another equivalent bet to check whether your new estimate is your 90% CI.

 

To train yourself, practice making estimates repeatedly while using these techniques, until you reach 100% accuracy.

To read more and try sample questions, read the article we prepared on 80,000 Hours here.

Like Eliezer, I "do my best thinking into a keyboard." It starts with a burning itch to figure something out. I collect ideas and arguments and evidence and sources. I arrange them, tweak them, criticize them. I explain it all in my own words so I can understand it better. By then it is nearly something that others would want to read, so I clean it up and publish, say, How to Beat Procrastination. I write essays in the original sense of the word: "attempts."

This time, I'm trying to figure out something we might call "tacit rationality" (c.f. tacit knowledge).

I tried and failed to write a good post about tacit rationality, so I wrote a bad post instead — one that is basically a patchwork of somewhat-related musings on explicit and tacit rationality. Therefore I'm posting this article to LW Discussion. I hope the ensuing discussion ends up leading somewhere with more clarity and usefulness.

Three methods for training rationality

Which of these three options do you think will train rationality (i.e. systematized winning, or "winning-rationality") most effectively?

1. Spend one year reading and re-reading The Sequences, studying the math and cognitive science of rationality, and discussing rationality online and at Less Wrong meetups.
2. Attend a CFAR workshop, then spend the next year practicing those skills and other rationality habits every week.
3. Run a startup or small business for one year.

Option 1 seems to be pretty effective at training people to talk intelligently about rationality (let's call that "talking-rationality"), and it seems to inoculate people against some common philosophical mistakes.

We don't yet have any examples of someone doing Option 2 (the first CFAR workshop was May 2012), but I'd expect Option 2 — if actually executed — to result in more winning-rationality than Option 1, and also a modicum of talking-rationality.

What about Option 3? Unlike Option 2 or especially Option 1, I'd expect it to train almost no ability to talk intelligently about rationality. But I would expect it to result in relatively good winning-rationality, due to its tight feedback loops.

Talking-rationality and winning-rationality can come apart

I've come to believe... that the best way to succeed is to discover what you love and then find a way to offer it to others in the form of service, working hard, and also allowing the energy of the universe to lead you.

Oprah Winfrey

Oprah isn't known for being a rational thinker. She is a known peddler of pseudoscience, and she attributes her success (in part) to allowing "the energy of the universe" to lead her.

Yet she must be doing something right. Oprah is a true rags-to-riches story. Born in Mississippi to an unwed teenage housemaid, she was so poor she wore dresses made of potato sacks. She was molested by a cousin, an uncle, and a family friend. She became pregnant at age 14.

But in high school she became an honors student, won oratory contests and a beauty pageant, and was hired by a local radio station to report the news. She became the youngest-ever news anchor at Nashville's WLAC-TV, then hosted several shows in Baltimore, then moved to Chicago and within months her own talk show shot from last place to first place in the ratings there. Shortly afterward her show went national. She also produced and starred in several TV shows, was nominated for an Oscar for her role in a Steven Spielberg movie, launched her own TV cable network and her own magazine (the "most successful startup ever in the [magazine] industry" according to Fortune), and became the world's first female black billionaire.

I'd like to suggest that Oprah's climb probably didn't come merely through inborn talent, hard work, and luck. To get from potato sack dresses to the Forbes billionaire list, Oprah had to make thousands of pretty good decisions. She had to make pretty accurate guesses about the likely consequences of various actions she could take. When she was wrong, she had to correct course fairly quickly. In short, she had to be fairly rational, at least in some domains of her life.

Similarly, I know plenty of business managers and entrepreneurs who have a steady track record of good decisions and wise judgments, and yet they are religious, or they commit basic errors in logic and probability when they talk about non-business subjects.

What's going on here? My guess is that successful entrepreneurs and business managers and other people must have pretty good tacit rationality, even if they aren't very proficient with the "rationality" concepts that Less Wrongers tend to discuss on a daily basis. Stated another way, successful businesspeople make fairly rational decisions and judgments, even though they may confabulate rather silly explanations for their success, and even though they don't understand the math or science of rationality well.

LWers can probably outperform Mark Zuckerberg on the CRT and the Berlin Numeracy Test, but Zuckerberg is laughing at them from atop a huge pile of utility.

Explicit and tacit rationality

Patri Friedman, in Self-Improvement or Shiny Distraction: Why Less Wrong is anti-Instrumental Rationality, reminded us that skill acquisition comes from deliberate practice, and reading LW is a "shiny distraction," not deliberate practice. He said a real rationality practice would look more like... well, what Patri describes is basically CFAR, though CFAR didn't exist at the time.

In response, and again long before CFAR existed, Anna Salamon wrote Goals for which Less Wrong does (and doesn't) help. Summary: Some domains provide rich, cheap feedback, so you don't need much LW-style rationality to become successful in those domains. But many of us have goals in domains that don't offer rapid feedback: e.g. whether to buy cryonics, which 40-year investments are safe, which metaethics to endorse. For this kind of thing you need LW-style rationality. (We could also state this as "Domains with rapid feedback train tacit rationality with respect to those domains, but for domains without rapid feedback you've got to do the best you can with LW-style "explicit rationality".)

The good news is that you should be able to combine explicit and tacit rationality. Explicit rationality can help you realize that you should force tight feedback loops into whichever domains you want to succeed in, so that you can have develop good intuitions about how to succeed in those domains. (See also: Lean Startup or Lean Nonprofit methods.)

Explicit rationality could also help you realize that the cognitive biases most-discussed in the literature aren't necessarily the ones you should focus on ameliorating, as Aaron Swartz wrote:

Cognitive biases cause people to make choices that are most obviously irrational, but not most importantly irrational... Since cognitive biases are the primary focus of research into rationality, rationality tests mostly measure how good you are at avoiding them... LW readers tend to be fairly good at avoiding cognitive biases... But there a whole series of much more important irrationalities that LWers suffer from. (Let's call them "practical biases" as opposed to "cognitive biases," even though both are ultimately practical and cognitive.)

...Rationality, properly understood, is in fact a predictor of success. Perhaps if LWers used success as their metric (as opposed to getting better at avoiding obvious mistakes), they might focus on their most important irrationalities (instead of their most obvious ones), which would lead them to be more rational and more successful.

Final scattered thoughts

• If someone is consistently winning, and not just because they have tons of wealth or fame, then maybe you should conclude they have pretty good tacit rationality even if their explicit rationality is terrible.
• The positive effects of tight feedback loops might trump the effects of explicit rationality training.
• Still, I suspect explicit rationality plus tight feedback loops could lead to the best results of all.
• I really hope we can develop a real rationality dojo.
• If you're reading this post, you're probably spending too much time reading Less Wrong, and too little time hacking your motivation system, learning social skills, and learning how to inject tight feedback loops into everything you can.

I'm shopping for a car, and I've spent many hours this past month reading user reviews of cars. There are seven things American car buyers have cared and complained about consistently for at least the past ten years.  In roughly decreasing importance:

• Performance
• Gas mileage
• Frequency and expense of repairs
• Smoothness of ride
• Exterior and interior styling
• Cup-holders
• Cargo space

Six of these things are complicated design trade-offs. For a good design, increasing any one of them makes most of the other five take a hit.

Cup-holders are not a complicated design trade-off. This should be a solved problem: Put two large, sturdy cup-holders somewhere accessible from the driver's seat. There is nothing to be gained from saving a few centimeters on cup-holder space that could be worth the millions of buyers who will walk away from a $50,000 car because they don't like its cup-holders.

Seriously, build the cup-holders first and design the rest of the interior around them. They're that important.

In the 1970s, no one had cup-holders or knew that they needed them. Things began changing in the 1980s, perhaps due to the expansion of Starbucks, perhaps due to the sudden increase in commute lengths. Today I like to have at least two and preferably three drinks with me for my 1-hour morning commute: A hot coffee to wake up, cold water for when I burn myself with the coffee, and a soda or tea for variety.

But car manufacturers were glacially slow to respond. I've been looking at used Jaguar XJs. These cars originally cost about $100,000 in today's money. Their owners complained continually about the cheap tiny plastic folding cup-holders that couldn't hold cups. They posted do-it-yourself fixes in online forums. Jaguar didn't even begin to address this until 2004, at least fifteen years into the cup-holder crisis, when they made the cup-holders slightly (but not much) less-crappy, and large enough to hold a small coffee (but not a medium).

Most new cars today finally have two cup-holders up front, and the collapsible cup-holders that enraged drivers for years by (predictably) collapsing are finally gone, but many cup-holders still aren't large enough to hold a Starbucks venti.

What the cup-holder paradox implies is that there are many multi-billion dollar care companies that spend hundreds of millions of dollars on product development every year without ever assigning a single summer intern to take one day to read some of the many thousands of user reviews available for free on cars.com, autotrader.com, and other websites. If they had, they'd have realized the depth of America's anger at shoddy cup-holders.

Or perhaps they read the reviews and dismiss them, because their customers are obviously morons who don't appreciate good auto design. Even today, auto manufacturers post photos of the interiors of all their new cars on their websites, but never in a dozen photos give you a clear view of the cup-holders, which makes me lean toward this view.

Or perhaps the cup-holders aren't even considered during design, but are added on at the last minute, because cars didn't used to have cup-holders at all and so that's not part of the design process. Perhaps automakers have internalized their process of producing and selling cars, and they can't conceive of adding a new element to that process, at least not until all the old automakers die out.

My priors say that it's more likely that I'm imagining the whole thing, that I selectively remember reviews complaining about cup-holders because of my own preferences, than that there has been a massive, systematic cognitive failure on the part of all the world's auto-makers, spanning 20 years, during which many of them somehow failed to observe, comprehend, or address this trivially-simple complaint of their customers, despite the billions of dollars at stake.

Am I?

The chapter on judgment under uncertainty in the (excellent) new Oxford Handbook of Cognitive Psychology has a handy little section on recent critiques of the "heuristics and biases" tradition. It also discusses problems with the somewhat-competing "fast and frugal heuristics" school of thought, but for now let me just quote the section on heuristics and biases (pp. 608-609):

The heuristics and biases program has been highly influential; however, some have argued that in recent years the influence, at least in psychology, has waned (McKenzie, 2005). This waning has been due in part to pointed critiques of the approach (e.g., Gigerenzer, 1996). This critique comprises two main arguments: (1) that by focusing mainly on coherence standards [e.g. their rationality given the subject's other beliefs, as contrasted with correspondence standards having to do with the real-world accuracy of a subject's beliefs] the approach ignores the role played by the environment or the context in which a judgment is made; and (2) that the explanations of phenomena via one-word labels such as availability, anchoring, and representativeness are vague, insufficient, and say nothing about the processes underlying judgment (see Kahneman, 2003; Kahneman & Tversky, 1996 for responses to this critique).

The accuracy of some of the heuristics proposed by Tversky and Kahneman can be compared to correspondence criteria (availability and anchoring). Thus, arguing that the tradition only uses the “narrow norms” (Gigerenzer, 1996) of coherence criteria is not strictly accurate (cf. Dunwoody, 2009). Nonetheless, responses in famous examples like the Linda problem can be reinterpreted as sensible rather than erroneous if one uses conversational or pragmatic norms rather than those derived from probability theory (Hilton, 1995). For example, Hertwig, Benz and Krauss (2008) asked participants which of the following two statements is more probable:

[X] The percentage of adolescent smokers in Germany decreases at least 15% from current levels by September 1, 2003.

[X&Y] The tobacco tax in Germany is increased by 5 cents per cigarette and the percentage of adolescent smokers in Germany decreases at least 15% from current levels by September 1, 2003.

According to the conjunction rule, [X&Y cannot be more probable than X] and yet the majority of participants ranked the statements in that order. However, when subsequently asked to rank order four statements in order of how well each one described their understanding of X&Y, there was an overwhelming tendency to rank statements like “X and therefore Y” or “X and X is the cause for Y” higher than the simple conjunction “X and Y.” Moreover, the minority of participants who did not commit the conjunction fallacy in the first judgment showed internal coherence by ranking “X and Y” as best describing their understanding in the second judgment.These results suggest that people adopt a causal understanding of the statements, in essence ranking the probability of X, given Y as more probable than X occurring alone. If so, then arguably the conjunction “error” is no longer incorrect. (See Moro, 2009 for extensive discussion of the reasons underlying the conjunction fallacy, including why “misunderstanding” cannot explain all instances of the fallacy.)

The “vagueness” argument can be illustrated by considering two related phenomena: the gambler’s fallacy and the hot-hand (Gigerenzer & Brighton, 2009). The gambler’s fallacy is the tendency for people to predict the opposite outcome after a run of the same outcome (e.g., predicting heads after a run of tails when flipping a fair coin); the hot-hand, in contrast, is the tendency to predict a run will continue (e.g., a player making a shot in basketball after a succession of baskets; Gilovich, Vallone, & Tversky, 1985). Ayton and Fischer (2004) pointed out that although these two behaviors are opposite - ending or continuing runs - they have both been explained via the label “representativeness.” In both cases a faulty concept of randomness leads people to expect short sections of a sequence to be “representative” of their generating process. In the case of the coin, people believe (erroneously) that long runs should not occur, so the opposite outcome is predicted; for the player, the presence of long runs rules out a random process so a continuation is predicted (Gilovich et al., 1985). The “representativeness” explanation is therefore incomplete without specifying a priori which of the opposing prior expectations will result. More important, representativeness alone does not explain why people have the misconception that random sequences should exhibit local representativeness when in reality they do not (Ayton & Fischer, 2004).

My thanks to MIRI intern Stephen Barnes for transcribing this text.

My paper "General purpose intelligence: arguing the Orthogonality thesis" has been accepted for publication in the December edition of Analysis and Metaphysics. Since that's some time away, I thought I'd put the final paper up here; the arguments are similar to those here, but this is the final version, for critique and citation purposes.

General purpose intelligence: arguing the Orthogonality thesis

STUART ARMSTRONG
stuart.armstrong@philosophy.ox.ac.uk
Future of Humanity Institute, Oxford Martin School
Philosophy Department, University of Oxford

In his paper “The Superintelligent Will”, Nick Bostrom formalised the Orthogonality thesis: the idea that the final goals and intelligence levels of artificial agents are independent of each other. This paper presents arguments for a (narrower) version of the thesis. It proceeds through three steps. First it shows that superintelligent agents with essentially arbitrary goals can exist in our universe – both as theoretical impractical agents such as AIXI and as physically possible real-world agents. Then it argues that if humans are capable of building human-level artificial intelligences, we can build them with an extremely broad spectrum of goals. Finally it shows that the same result holds for any superintelligent agent we could directly or indirectly build. This result is relevant for arguments about the potential motivations of future agents: knowing an artificial agent is of high intelligence does not allow us to presume that it will be moral, we will need to figure out its goals directly.

Keywords: AI; Artificial Intelligence; efficiency; intelligence; goals; orthogonality

1                       The Orthogonality thesis

Scientists and mathematicians are the stereotypical examples of high intelligence humans. But their morality and ethics have been all over the map. On modern political scales, they can be left- (Oppenheimer) or right-wing (von Neumann) and historically they have slotted into most of the political groupings of their period (Galois, Lavoisier). Ethically, they have ranged from very humanitarian (Darwin, Einstein outside of his private life), through amoral (von Braun) to commercially belligerent (Edison) and vindictive (Newton). Few scientists have been put in a position where they could demonstrate genuinely evil behaviour, but there have been a few of those (Teichmüller, Philipp Lenard, Ted Kaczynski, Shirō Ishii).

This is the introduction to a paper I started writing long ago, but have since given up on. The paper was going to be an overview of methods for improving human rationality through cognitive change. Since it contains lots of handy references on rationality, I figured I'd publish it, in case it's helpful to others.

1. Introduction

During the last half-century, cognitive scientists have catalogued dozens of common errors in human judgment and decision-making (Griffin et al. 2012; Gilovich et al. 2002). Stanovich (1999) provides a sobering introduction:

For example, people assess probabilities incorrectly, they display confirmation bias, they test hypotheses inefficiently, they violate the axioms of utility theory, they do not properly calibrate degrees of belief, they overproject their own opinions onto others, they allow prior knowledge to become implicated in deductive reasoning, they systematically underweight information about nonoccurrence when evaluating covariation, and they display numerous other information-processes biases...

The good news is that researchers have also begun to understand the cognitive mechanisms which produce these errors (Kahneman 2011; Stanovich 2010), they have found several "debiasing" techniques that groups or individuals may use to partially avoid or correct these errors (Larrick 2004), and they have discovered that environmental factors can be used to help people to exhibit fewer errors (Thaler and Sunstein 2009; Trout 2009).

This "heuristics and biases" research program teaches us many lessons that, if put into practice, could improve human welfare. Debiasing techniques that improve human rationality may be able to decrease rates of violence caused by ideological extremism (Lilienfeld et al. 2009). Knowledge of human bias can help executives make more profitable decisions (Kahneman et al. 2011). Scientists with improved judgment and decision-making skills ("rationality skills") may be more apt to avoid experimenter bias (Sackett 1979). Understanding the nature of human reasoning can also improve the practice of philosophy (Knobe et al. 2012; Talbot 2009; Bishop and Trout 2004; Muehlhauser 2012), which has too often made false assumptions about how the mind reasons (Weinberg et al. 2001; Lakoff and Johnson 1999; De Paul and Ramsey 1999). Finally, improved rationality could help decision makers to choose better policies, especially in domains likely by their very nature to trigger biased thinking, such as investing (Burnham 2008), military command (Lang 2011; Williams 2010; Janser 2007), intelligence analysis (Heuer 1999), or the study of global catastrophic risks (Yudkowsky 2008a).

But is it possible to improve human rationality? The answer, it seems, is "Yes." Lovallo and Sibony (2010) showed that when organizations worked to reduce the effect of bias on their investment decisions, they achieved returns of 7% or higher. Multiple studies suggest that a simple instruction to "think about alternative hypotheses" can counteract overconfidence, confirmation bias, and anchoring effects, leading to more accurate judgments (Mussweiler et al. 2000; Koehler 1994; Koriat et al. 1980). Merely warning people about biases can decrease their prevalence, at least with regard to framing effects (Cheng and Wu 2010), hindsight bias (Hasher et al. 1981; Reimers and Butler 1992), and the outcome effect (Clarkson et al. 2002). Several other methods have been shown to meliorate the effects of common human biases (Larrick 2004). Judgment and decision-making appear to be skills that can be learned and improved with practice (Dhami et al. 2012).

In this article, I first explain what I mean by "rationality" as a normative concept. I then review the state of our knowledge concerning the causes of human errors in judgment and decision-making (JDM). The largest section of our article summarizes what we currently know about how to improve human rationality through cognitive change (e.g. "rationality training"). We conclude by assessing the prospects for improving human rationality through cognitive change, and by recommending particular avenues for future research.

I sometimes go around saying that the fundamental question of rationality is Why do you believe what you believe?

-- Eliezer in Quantum Non-Realism

I was much impressed when they finally came out with a PC version of DragonBox, and I got around to testing it on some children I knew. Two kids, one of them four and the other eight years old, ended up blazing through several levels of solving first-degree equations while having a lot of fun doing so, even though they didn't know what it was that they were doing. That made me think that there has to be some way of making a computer game that would similarly teach rationality skills at the 5-second level. Some game where you would actually be forced to learn useful skills if you wanted to make progress.

After playing around with some ideas, I hit upon the notion of making a game centered around the Fundamental Question. I'm not sure whether this can be made to work, but it seems to have promise. The basic idea: you are required to figure out the solution to various mysteries by collecting various kinds of evidence. Some of the sources of evidence will be more reliable than others. In order to hit upon the correct solution, you need to consider where each piece of evidence came from, and whether you can rely on it.

Gameplay example

Now, let's go into a little more detail. Let's suppose that the game has a character called Bob. Bob tells you that tomorrow, eight o'clock, there will be an assassination attempt on Market Square. The fact that Bob has told you this is evidence for the claim being true, so the game automatically records the fact that you have such a piece of evidence, and that it came from Bob.

(Click on the pictures in case you don't see them properly.)

But how does Bob know that? You ask, and it turns out that Alice told him. So next, you go and ask Alice. Alice is confused and says that she never said anything about any assassination attempt: she just said that something big is going to be happen at the Market Square at that time, she heard it from the Mayor. The game records two new pieces of evidence: Alice's claim of something big happening at the Market Square tomorrow (which she heard from the Mayor), and her story of what she actually told Bob. Guess that Bob isn't a very reliable source of evidence: he has a tendency to come up with fancy invented details.

Or is he? After all, your sole knowledge about Bob being unreliable is that Alice claims she never said what Bob says she said. But maybe Alice has a grudge against Bob, and is intentionally out to make everyone disbelieve him. Maybe it's Alice who's unreliable. The evidence that you have is compatible with both hypotheses. At this point, you don't have enough information to decide between them, but the game lets you experiment with setting either of them as "true" and seeing the implications of this on your belief network. Or maybe they're both true - Bob is generally unreliable, and Alice is out to discredit him. That's another possibility that you might want to consider. In any case, the claim that there will be an assassination tomorrow isn't looking very likely at the moment.

Actually, having the possibility for somebody lying should probably be a pretty late-game thing, as it makes your belief network a lot more complicated, and I'm not sure whether this thing should display numerical probabilities at all. Instead of having to juggle the hypotheses of "Alice lied" and "Bob exaggerates things", the game should probably just record the fact that "Bob exaggerates things". But I spent a bunch of time making these pictures, and they do illustrate some of the general principles involved, so I'll just use them for now.

Game basics

So, to repeat the basic premise of the game, in slightly more words this time around: your task is to figure out something, and in order to do so, you need to collect different pieces of evidence. As you do so, the game generates a belief network showing the origin and history of the various pieces of evidence that you've gathered. That much is done automatically. But often, the evidence that you've gathered is compatible with many different hypotheses. In those situations, you can experiment with different ways of various hypotheses being true or false, and the game will automatically propagate the consequences of that hypothetical through your belief network, helping you decide what angle you should explore next.

Of course, people don't always remember the source of their knowledge, or they might just appeal to personal experiences. Or they might lie about the sources, though that will only happen at the more advanced levels.

As you proceed in the game, you will also be given access to more advanced tools that you can use for making hypothetical manipulations to the belief network. For example, it may happen that many different characters say that armies of vampire bats tend to move about at full moon. Since you hear that information from many different sources, it seems reliable. But then you find out that they all heard it from a nature documentary on TV that aired a few weeks back. This is reflected in your belief graph, as the game modifies it to show that all of those supposedly independent sources can actually be tracked back to a single one. That considerably reduces the reliability of the information.

But maybe you were already suspecting that the sources might not be independent? In that case, it would have been nice if the belief graph interface would let you postulate this beforehand, and see how big of an effect it would make on the plausibility of the different hypotheses if they were in fact reliant on each other. Once your character learns the right skills, it becomes possible to also add new hypothetical connections to the belief graph, and see how this would influence your beliefs. That will further help you decide what possibilities to explore and verify.

Because you can't explore every possible eventuality. There's a time limit: after a certain amount of moves, a bomb will go off, the aliens will invade, or whatever.

The various characters are also more nuanced than just "reliable" or "not reliable". As you collect information about the various characters, you'll figure out their mindware, motivations, and biases. Somebody might be really reliable most of the time, but have strong biases when it comes to politics, for example. Others are out to defame others, or invent fancy details to all the stories. If you talk to somebody you don't have any knowledge about yet, you can set a prior on the extent that you rely on their information, based on your experiences with other people.

You also have another source of evidence: your own intuitions and experience. As you get into various situations, a source of evidence that's labeled simply "your brain" will provide various gut feelings and impressions about things. The claim that Alice presented doesn't seem to make sense. Bob feels reliable. You could persuade Carol to help you if you just said this one thing. But in what situations, and for what things, can you rely on your own brain? What are your own biases and problems? If you have a strong sense of having heard something at some point, but can't remember where it was, are you any more reliable than anyone else who can't remember the source of their information? You'll need to figure all of that out.

As the game progresses to higher levels, your own efforts will prove insufficient for analyzing all the necessary information. You'll have to recruit a group of reliable allies, who you can trust to analyze some of the information on their own and report the results to you accurately. Of course, in order to make better decisions, they'll need you to tell them your conclusions as well. Be sure not to report as true things that you aren't really sure about, or they will end up drawing the wrong conclusions and focusing on the wrong possibilities. But you do need to condense your report somewhat: you can't just communicate your entire belief network to them.

Hopefully, all of this should lead to player learning on a gut level things like:

• Consider the origin of your knowledge: Obvious.
• Visualizing degrees of uncertainty: In addition to giving you a numerical estimate about the probability of something, the game also color-codes the various probabilities and shows the amount of probability mass associated with your various beliefs.
• Considering whether different sources really are independent: Some sources which seem independent won't actually be that, and some which seem dependent on each other won't be.
• Value of information: Given all the evidence you have so far, if you found out X, exactly how much would it change your currently existing beliefs? You can test this and find out, and then decide whether it's worth finding out.
• Seek disconfirmation: A lot of things that seem true really aren't, and acting on flawed information can cost you.
• Prefer simpler theories: Complex, detailed hypotheses are more likely to be wrong in this game as well.
• Common biases: Ideally, the list of biases that various characters have is derived from existing psychological research on the topic. Some biases are really common, others are more rare.
• Epistemic hygiene: Pass off wrong information to your allies, and it'll cost you.
• Seek to update your beliefs: The game will automatically update your belief network... to some extent. But it's still possible for you to assign mutually exclusive events probabilities that sum to more than 1, or otherwise have conflicting or incoherent beliefs. The game will mark these with a warning sign, and it's up to you to decide whether this particular inconsistency needs to be resolved or not.
• Etc etc.
  

Design considerations

It's not enough for the game to be educational: if somebody downloads the game because it teaches rationality skills, that's great, but we want people to also play it because it's fun. Some principles that help ensure that, as well as its general utility as an educational aid, include:

• Provide both short- and medium-term feedback: Ideally, there should be plenty of hints for how to find out the truth about something by investigating just one more thing: then the player can find out whether your guess was correct. It's no fun if the player has to work through fifty decisions before finding out whether they made the right move: they should get constant immediate feedback. At the same time, the player's decisions should be building up to a larger goal, with uncertainty about the overall goal keeping them interested.
• Don't overwhelm the player: In a game like this, it would be easy to throw a million contradictory pieces of evidence at the player, forcing them to go through countless of sources of evidence and possible interactions and have no clue of what they should be doing. But the game should be manageable. Even if it looks like there is a huge messy network of countless pieces of contradictory evidence, it should be possible to find the connections which reveal the network to be relatively simple after all. (This is not strictly realistic, but necessary for making the game playable.)
• Introduce new gameplay concepts gradually: Closely related to the previous item. Don't start out with making the player deal with every single gameplay concept at once. Instead, start them out in a trusted and safe environment where everyone is basically reliable, and then begin gradually introducing new things that they need to take into account.
• No tedium: A game is a series of interesting decisions. The game should never force the player to do anything uninteresting or tedious. Did Alice tell Bob something? No need to write that down, the game keeps automatic track of it. From the evidence that has been gathered so far, is it completely obvious what hypothesis is going to be right? Let the player mark that as something that will be taken for granted and move on.
• No glued-on tasks: A sign of a bad educational game is that the educational component is glued on to the game (or vice versa). Answer this exam question correctly, and you'll get to play a fun action level! There should be none of that - the educational component should be an indistinguishable part of the game play.
• Achievement, not fake achievement: Related to the previous point. It would be easy to make a game that wore the attire of rationality, and which used concepts like "probability theory", and then when your character leveled up he would get better probability attacks or whatever. And you'd feel great about your character learning cool stuff, while you yourself learned nothing. The game must genuinely require the player to actually learn new skills in order to get further.
• Emotionally compelling: The game should not be just an abstract intellectual exercise, but have an emotionally compelling story as well. Your choices should feel like they matter, and characters should be in risk of dying if you make the wrong decisions.
• Teach true things: Hopefully, the players should take the things that they've learned from the game and apply them to their daily lives. That means that we have a responsibility not to teach them things which aren't actually true.
• Replayable: Practice makes perfect. At least part of the game world needs to be randomly generated, so that the game can be replayed without a risk of it becoming boring because the player has memorized the whole belief network.

What next?

What you've just read is a very high-level design, and a quite incomplete one at that: I've spoken on the need to have "an emotionally compelling story", but said nothing about the story or the setting. This should probably be something like a spy or detective story, because that's thematically appropriate for a game which is about managing information; and it might be best to have it in a fantasy setting, so that you can question the widely-accepted truths of that setting without needing to get on anyone's toes by questioning widely-accepted truths of our society.

But there's still a lot of work that remains to be done with regard to things like what exactly does the belief network look like, what kinds of evidence can there be, how does one make all of this actually be fun, and so on. I mentioned the need to have both short- and medium-term feedback, but I'm not sure of how that could be achieved, or whether this design lets you achieve it at all. And I don't even know whether the game should show explicit probabilities.

And having a design isn't enough: the whole thing needs to be implemented as well, preferably while it's still being designed in order to take advantage of agile development techniques. Make a prototype, find some unsuspecting testers, spring it on them, revise. And then there are the graphics and music, things for which I have no competence for working on.

I'll probably be working on this in my spare time - I've been playing with the idea of going to the field of educational games at some point, and want the design and programming experience. If anyone feels like they could and would want to contribute to the project, let me know.

EDIT: Great to see that there's interest! I've created a mailing list for discussing the game. It's probably easiest to have the initial discussion here, and then shift the discussion to the list.

Related: 23 Cognitive Mistakes that make People Play Bad Poker

Followed by: Situational Awareness And You

If epistemic rationality is the art of updating one's beliefs based on new evidence to better correspond with reality, the zeroth skillset of epistemic rationality-- the one that enables all other skills to function-- is that of situational awareness. Situational awareness-- sometimes referred to as "situation awareness" or simply "SA"-- is the skillset and related state of mind that allows one to effectively perceive the world around them.

One might ask how this relates to rationality at all. The answer is simple. Just as the skill of lucid dreaming is near-useless without dream recall,^[1] the skills of updating based on evidence and actually changing your mind are near-useless without good awareness skills-- after all, you can't update based on evidence that you haven't collected! A high degree of situational awareness is thus an important part of one's rationalist toolkit, as it allows you to notice evidence about the world around you that you would otherwise miss. At times, this evidence can be of critical importance. I can attest that I have personally saved the lives of friends on two occasions thanks to good situational awareness, and have saved myself from serious injury or death many times more.

Situational awareness is further lauded by elite military units, police trainers, criminals, intelligence analysts, and human factors researchers. In other words, people who have to make very important-- often life-or-death-- decisions based on limited information consider situational awareness a critical skill. This should tell us something-- if those individuals for whom correct decisions are most immediately relevant all stress the importance of situational awareness, it may be a more critical skill than we realize.

Unfortunately, the only discussion of situational awareness that I've seen on LessWrong or related sites has been a somewhat oblique reference in Louie Helm's "roadmap of errors" from 23 Cognitive Mistakes that make People Play Bad Poker.^[2] I believe that situational awareness is important enough that it merits an explicit sequence of posts on its advantages and how to cultivate it, and this post will serve as the introduction to that sequence.

The first post in the sequence, unimaginatively titled "Situational Awareness and You," will be posted within the week. Other planned posts include "Cultivating Awareness," "How to Win a Duel," "Social Awareness," "Be Aware of Your Reference Class," "Signaling and Predation," and "Constant Vigilance!"

If you have any requests for things to add, general questions about the sequence, meta-thoughts about SA, and so on, this post is an appropriate place for that discussion; as this is primarily a meta post, it has been posted to Discussion. Core posts in the sequence will be posted to Main.

[1] What good are lucid dreams if you can't remember them?

[2] This is a very useful summary and you should read it even if you don't play poker.

A friend has been asking my views on the likelihood that there's anything to a correlation between changing levels of lead in paint (and automotive exhaust) and the levels of crime. He quoted from a Reason Blog:

So Nevin dove in further, digging up detailed data on lead emissions and crime rates to see if the similarity of the curves was as good as it seemed. It turned out to be even better: In a 2000 paper (PDF) he concluded that if you add a lag time of 23 years, lead emissions from automobiles explain 90 percent of the variation in violent crime in America. Toddlers who ingested high levels of lead in the '40s and '50s really were more likely to become violent criminals in the '60s, '70s, and '80s.

I responded with the following:

Sounds like a stretch to me. I'd want to hear that they didn't test more than 5 other hypothesis before coming to that conclusion, or the p value was far better than .05. I kind of doubt that either is the case. 

He's apparently continued to pursue the question, and just forwarded these remarks from Steven Pinker that I thought were very illuminating, and probably deserve a place in this community's toolkit for skeptics. Pinker's main point is that the association between Lead and crime is a long tenuous chain of suppositions, and several of the intermediate points should be far easier to measure. Finding correlations at this distance is not very informative.

http://stevenpinker.com/files/pinker/files/pinker_comments_on_lead_removal_and_declining_crime.pdf

Does the phrase "long-chain correlation" stick in your head and make it easier to dismiss this kind of argument?

Related: Reference Class of the Unclassreferenceable, Generalizing From One Example

Many people try to predict the future. Few succeed.

One common mistake made in predicting the future is to simply take a current trend and extrapolate it forward, as if it was the only thing that mattered-- think, for instance, of the future described by cyberpunk fiction, with sinister (and often Japanese) multinational corporations ruling the world. Where does this vision of the future stem from?

Bad or lazy predictions from the 1980s, when sinister multinational corporations (and often Japanese ones) looked to be taking over the world.^[1]

Similar errors have been committed by writers throughout history. George Orwell thought 1984 was an accurate prediction of the future, seeing World War II as inevitably bringing socialist revolution to the United Kingdom and predicting that the revolutionary ideals would then be betrayed in England as they were in Russia. Aldous Huxley agreed with Orwell but thought that the advent of hypnosis and psychoconditioning would cause the dystopia portrayed in 1984 to evolve into that he described in Brave New World. In today's high school English classes, these books are taught as literature, as well-written stories-- the fact that the authors took their ideas seriously would come as a surprise to many high school students, and their predictions would look laughably wrong.

Were such mistakes confined solely to the realm of fiction, they would perhaps be considered amusing errors at best, reflective of the sorts of mishaps that befall unstudied predictions. Unfortunately, they are not. Purported "experts" make just the same sort of error regularly, and failed predictions of this sort often have negative consequences in reality.

For instance, in 1999 two economists published the book Dow 36,000, predicting that stocks were about to reach record levels; the authors of the book were so wrapped up in recent gains to the stock market that they assumed that such gains were in fact the new normal state of affairs, that the market hadn't corrected for this yet, and that once stocks were correctly perceived as safe investments the market would skyrocket. This not only did not happen, but the dot-com bubble burst shortly after the book was published.^[2] Anyone following the market advice from this book lost big.

In 1968, the biologist Paul R. Ehrlich, seeing disturbing trends in world population growth, wrote a book called The Population Bomb, in which he forecast (among other things) that "The battle to feed all of humanity is over. In the 1970s hundreds of millions of people will starve to death in spite of any crash programs embarked upon now." Later, Ehrlich doubled down on this prediction with claims such as  "By the year 2000 the United Kingdom will be simply a small group of impoverished islands, inhabited by some 70 million hungry people ... If I were a gambler, I would take even money that England will not exist in the year 2000."

Based on these predictions, Ehrlich advocated cutting off food aid to India and Egypt in favor of preserving food supplies for nations that were not "lost causes;" luckily, his policies were not adopted, as they would have resulted in mass starvation in the countries suddenly deprived of aid. Instead, food aid continued, and as population grew, food production did as well. Contrary to the increase in starvation and global death rates predicted by Ehrlich, global death rates decreased, the population increased by more than Ehrlich had predicted would lead to disaster, and the average amount of calories consumed per person increased as well.^[3]

So what, then, is the weakness that causes these analysts to make such errors?

Well, just as you can generalize from one example when evaluating others and hence fail to understand those around you, you can generalize from one trend or set of trends when making predictions and hence fail to understand the broader world. This is a special case of the classic problem where "to a man with a hammer, everything looks like a nail;" if you are very familiar with one trend, and that's all you take into account with your future forecasts, you're bound to be wrong if that trend ends up not eating the world.

On the other hand, the trend sometimes does eat the world. It's very easy to find long lists of buffoonish predictions where someone woefully understimated the impact of a new technology.^[4] Further, determining exactly when and where a trend is going to stop is quite difficult, and most people are incompetent at it, even at a professional level-- if this were easy, the stock market would look extraordinarily different!

So my advice to those who would predict the future is simple. Don't generalize from one trend or even one group of trends. Especially beware of viewing evidence that seems to support your predictions as evidence that other people's predictions must be wrong-- the notebook of rationality cares not for what "side" things are on, but rather for what is true. Even if the trend you're relying on does end up being the "next big thing," the rest of the world will have a voice as well.^[5]

[1] I predict that the work of Cory Doctorow and those like him will seem similarly dated a decade down the line, as the trends they're riding die down. If you're reading this during or after December 2022, please let me know what you think of this prediction.

[2] The authors are, of course, still employed in cushy think-tank positions.

[3]  Ehrlich has doubled down on his statements, now claiming that he was "way too optimistic" in The Population Bomb and that the world is obviously doomed.

[4] I personally enjoy the Bad Opinion Generator (warning: potentially addictive)

[5] Technically, this isn't always true. But you should assume it is unless you have extremely good reasons to believe otherwise, and even still I would be very careful before assuming that your thing is the thing.

This is the public group instrumental rationality diary for the week of January 7th. It's a place to record and chat about it if you have done, or are actively doing, things like:

EDIT: Based on criticism below, I am reconsidering how to proceed with this idea (or something in the neighbourhood).

A topic that has been on my mind recently is where, in our complicated lives, there might be low-hanging fruit ready to be picked by a motivated rationalist. Actual, practical, dollars-and-cents fruit.

In possibly-related news, here is how the writer of About.com's beginner's guide to investing describes the stock market:

Imagine you are partners in a private business with a man named Mr. Market. Each day, he comes to your office or home and offers to buy your interest in the company or sell you his [the choice is yours]. The catch is, Mr. Market is an emotional wreck. At times, he suffers from excessive highs and at others, suicidal lows. When he is on one of his manic highs, his offering price for the business is high as well, because everything in his world at the time is cheery. His outlook for the company is wonderful, so he is only willing to sell you his stake in the company at a premium. At other times, his mood goes south and all he sees is a dismal future for the company. In fact, he is so concerned, he is willing to sell you his part of the company for far less than it is worth. All the while, the underlying value of the company may not have changed - just Mr. Market's mood.

I have heard this narrative many times before, and I'd like to test whether it is accurate - and in particular, whether LWers can consistently beat the market.

The skeptic may well ask: why should LWers have an advantage? Why not go to the professionals - investment advisors? Also, isn't there a whole chapter in Kahneman about how even smart people suck at picking stocks? And what do you, simplicio, know about this anyway?

LWers may have an advantage by virtue of being educated about such topics as cognitive biases, sunk cost fallacy, probabilistic prediction, and expected utility - topics with which investment advisors et al. may or may not be familiar on a gut level. I am not sure if we're any better, but I'd like to test it. Also, if LW turns out to be any good at offering such advice, that advice would presumably be free, unlike that of yon advisor (fees tend to kill returns on investment - just ask anybody who uses Intrade). As for what I personally know - not very much yet. But I find competition very stimulating.

Accordingly, my proposal is for a contest: over the course of 2013, I will set up & maintain a Google Drive spreadsheet. This spreadsheet will be shared with contest participants. Each participant will have say $5,000 of play money to use "buying" (or "selling") stocks on the exchange of their choice. Contestants will record the date of purchase or sale, quantity, and preferably provide comments regarding why they are buying or selling.

At the end of this contest (Dec 31, 2013?), I will commit to Paypal the winner (defined as the person with the highest market valuation of play assets as of midnight on that date) the equivalent of $50 CAD in their local currency. In the unlikely event that I win, I will donate that $50 to the Against Malaria Foundation. (Above commitment does not take effect until I actually gauge interest in this contest, figure out an end date & rules etc., and decide to proceed. If anyone else wants to throw money in the pot, please do.)

The purposes of this post are therefore:

• to find out who is interested - please leave a comment below, and e-mail me at ispollock [at] gmail.com if you want in;
• to solicit constructive and destructive criticism of the project, especially from any local experienced investors (in particular, perhaps a one-year timeframe is too short for a meaningful contest? Also, real-world experience of transaction costs in buying and selling would be extremely helpful);
• to ask if anyone knows of a better software platform for the contest than Google Drive, or knows of any extremely helpful resources I should be reading/linking to.

This is the public group instrumental rationality diary for Christmas week. It's a place to record and chat about it if you have done, or are actively doing, things like:

I am emotionally excited and/or deeply hurt by what st_rev wrote recently. You better take me seriously because you've spent a lot of time reading my posts already and feel invested in our common tribe. Anecdote about how people are tribal thinkers.

That thing that happened shows that everything I was already advocating for is correct and necessary. Indeed it is time for everyone to put their differences aside and come together to carry out my recommended course of action. If you continue to deny what both you and I know in our hearts to be correct, you want everyone to die and I am defriending you.

I don't even know where to begin. This is what blueist ideology has been workign towards for decades if not millennia, but to see it written here is hard to stomach even for one as used to the depravity caused by such delusions as I am. The lack of socially admired virtues among its adherents is frightening. Here I introduce an elaborate explanation of how blueist domination is not just completely obvious and a constant thorn in the side of all who wish more goodness but is achieved by the most questionable means often citing a particular blogger or public intellectual who I read in order to show how smart I am and because people I admire read him too. Followed by an appeal to the plot of a movie. Anecdote from my personal life. If you are familiar with the obscure work of an academic taken out of context and this does not convince you then you are clearly an intolerant sexual deviant engaging in motivated cognition.

Consider well: do you want to be on the wrong side of history? If you persist, millions or billions of people you will never meet will be simultaneously mystified and appalled that an issue so obvious caused such needless contention. They will argue whether you were motivated more by stupidity, malice, raw interest, or if you were a helpless victim of the times in which you lived. Characters in fiction set in your era will inevitably be on (or at worst, join) the right side unless they are unredeemable villains. (Including historical figures who were on the other side, lest they lose all audience sympathy.).

Remember: it's much more important what hypothetical future people will consider right than what you or current people you respect do. And you and I both know they'll agree with me.

While sympathetic to this criticsm I must signal my world-weariness and sophistication by writing several long paragraphs about how this is much too optimistic and we are in grave danger of a imminent and eternal takeover by our opponents. The only solution is to begin work on an organization dedicated to preventing this which happens to give me access to material resources and attractive females.

Ciphergoth proves to be the lone voice of reason by encouraging us to recall what we all learned on 9/11:

However, we must also consider if this is not also a lesson to us all; a lesson that my political views are correct.

http://www.adequacy.org/stories/2001.9.12.102423.271.html

Consider the following statement: two heads are better than one.

It seems obvious to me that several rationalists working together can, effectively, bring more precious brainpower to bear on a problem, than one rationalist working alone (otherwise, what would be the point of having a Less Wrong forum community? You might as well just leave it as a curated community blog and excise the discussion forums.). Further, due to various efforts (HPMOR especially) it appears that LW is inevitably growing. This makes it not only desirable to find ways to effectively get groups of rationalists to think together, but also increasingly necessary.

It is also desirable that methods of getting groups to think should be feasibly doable over the Internet. (I am aware that real-life meetups and stuff exist, but please be reminded that some people in the world do have to live in shitty little third-world countries and might not at all find it economically feasible to go to first-world countries with atrociously high costs-of-living)

So first, let us start with the current "best methods" of getting groups of Traditional Rationalists to coordinate and think, while avoiding groupthink effects that diminish our aggregate rationality. Hopefully, we can then use it as the basis of part of the art of rationalist group thinking. So I'll discuss:

• Disputation Arenas - procedures with centrifugal and centripetal phases
  1. Delphi Method - members secretly answer questions, non-member summarizer anonymizes answers, members read anonymized summary, repeat
  2. Prediction Market - members place stock bets on market, market settles on price, members react to price signal, repeat
  3. Nominal Group Technique - members write down ideas privately, non-member facilitator guides members in sharing ideas, non-member facilitator guides members in paring down and cleaning up ideas, members vote
• Conclusion - strengths and weaknesses of the various disputation arenas shown here

Disputation Arenas

One of my favorite SF authors, David Brin, talks a bit about what he calls "disputation arenas". I won't discuss his ideas here, since his concept of "disputation arena" is actually a relatively "new", raw, and relatively untested procedure - what I intend to discuss for now are things that have at least been studied more rigorously than just a bunch of blog posts (or personal website pages, whatever).

However, I do want to co-opt the term "Disputation Arena" for any process that tries to achieve the following:

• Avoid groupthink - actively search for information without settling too early on an option considered desirable by certain influential members
• Achieve consensus - designate some choice as the best, given current information known by the group members

We want our group rationality process to avoid groupthink (possibly at some expense of efficiency) because actual, real-world rationalists are not perfect Bayesian reasoners - two words: Robert Aumann. Because rationalists are not perfect, we do not expect a clear consensus to form after the end of the process (i.e. Aumann's does not necessarily apply), so the process we use must force some consensus to become visible.

One thing that David Brin discusses is the general division of the procedures into two "phases":

• Centrifugal phase - members of the group generate ideas separately.
• Centripetal phase - ideas are judged by the group together.

This seems to me to be a good way of labeling parts of any group-coordination process that attempts to avoid groupthink and achieve consensus.

In my personal research, I've found three things that attempt to achieve those two goals (avoid groupthink, achieve consensus) and which might (perhaps with a stretch) be considered as approximately having two phases (centrifugal and centripetal).

Delphi Method

The Delphi Method was originally developed by RAND Corporation (Project RAND at that time, and no relation to Ayn Rand) in order to get better predictions on future military hardware. It is currently used to get better utilization of current human wetware. (^.^)v

Delphi Method: How To Do It!

Pen and paper version:

1. A panel of experts is chosen, and a questionnaire is prepared.
2. Experts answer the questionnaire, giving the answers and also justifications and reasons for those answers.
3. Summarizer provides anonymous summaries of the expert's answers and justifications.
4. Experts read the summary, and may revise their answers/justifications. The process is repeated (with the same experts and questionnaire) for a set number of rounds, or until everyone gets bored, or the military bunker everyone is in gets nuked.

Internet version (Wikipedia:Real-time Delphi):

1. Username/passwords are chosen and emailed, and a questionnaire is prepared. The questionnaire in the online version is somewhat restricted, however. Here are some ideas:
   • Use a "poll" question. Experts select one of the choices given.
   • Use "multiple-choice" questions. Experts then select from a range of (e.g.) 1 for (strongly disagree) to 10 (strongly agree) for each possible choice.
   • Use "multiple-choice" questions, and also give different aspects such as "feasibility", "desirability", "good side-effects", etc. Experts answer 1 to 10 for each combination of choice and aspect.
2. Experts answer the online questionnaire. Aside from the numerical or selection (quantitative) answer, experts should also supply a short sentence or two justifying each answer (qualitative).
3. After the expert submits his or her answers, he or she is shown the current averages (for scoring-type questionnaires) or current poll results - this is the quantitative group answer. Expert is also shown (or provided links to) randomly-sorted (and randomly-chosen, if groups are very large and the number of answers may overwhelm a typical human) qualitative answers for each poll item / choice / choice+aspect - the qualitative group answers - for each score or aspect.
   • IMPORTANT: individual qualitative answers should not show the username of the expert who gave them!
   • In effect, the average (or poll results) plus the randomized sample of qualitative answers are a simple, anonymous, machine-generated summary.
4. Experts may change their own quantitative and/or qualitative answers at any time, and see the current quantitative group answers and qualitative group answers at any time after they have submitted their own answers.
5. The questionnaire is kept open until some specified time, or somebody hacks the server to put LOLcats instead.

Delphi Method: Analysis

Delphi methods avoid groupthink largely by anonymity: this avoids the bandwagon effect, the halo effect, and the mind-killer. Anonymity and constant feedback also encourage people to revise their positions in light of new information from their peers (by reducing consistency pressure): in non-anonymous face-to-face meetings, people tend to stick to their previously stated opinions, and to conform to the meeting leader(s) or their own bosses in the meeting. A lot of those effects is reduced by anonymity. Pen-and-paper form makes anonymity much easier, since the summary gets the tone and language patterns of the summarizer; some amount of anonymity is lost in the online version (since language patterns might theoretically be analyzed) but hopefully the small sample size (just a short sentence or two) can make language pattern analysis difficult. Note that randomizing the order of the comments in the online version is important, as this reduces the effects of anchoring; sorting by time or karma may increase groupthink due to anchoring on earlier comments, but if each expert sees different "first comments", then this bias gets randomized (hopefully into irrelevancy).

Delphi methods achieve consensus by the summary (which often serves as the "final output" when the process is finished). Arguably, the pen-and-paper version is better at achieving consensus due to the "turn-based" arrival of the summary, which makes the expert pay more attention to the summary, compared to the real-time online system.

The Delphi method's centrifugal phase is the expert's private answering of the questionnaire: each expert makes this decision, and provides the justification, without other's knowledge or help.

The Delphi method's centripetal phase is the act of summarizing, and having the experts read the summary.

Delphi Method: Other Thoughts

I think that forum polls, in general, can be easily adapted into online real-time Delphis by adding the following:

• Members will be required to give a short sentence or two (probably limited to say 200 chars or so) justifying their poll choice.
• Members should be allowed to change their poll choice and their justification at any time until the poll closes or the forum is hacked by LOLcat vandals.
• Members should be able to click on a poll choice on the poll results page to get a random anonymous sampling of the justifications that the other members have made in choosing that poll choice.

The procedure says "experts" but I think that in something more democratic than the military you're supposed to read that as "anyone who bothers to participate".

Prediction Market

Prediction markets are speculation markets built around bets on what things will happen in the future. They are also the core of Robin Hanson's Futarchy, and which you can see somewhere in the background of LW's favorite tentacle alien porn novella (O.o);;.

Prediction Market: How To Do It!

Pen and paper version:

1. Convince a trusted monetary institution to sell you "X is gonna happen" stock and "X is not gonna happen" stock for $1 a pair (i.e. $1 for a pair of contracts, one that says "If X happens, Monetary Institution pays you $1" and another that says "If X doesn't happen, Monetary Institution pays you $1", so the pair costs $1 total since X can't both happen and not happen). You may need to pay some sort of additional fee or something if the monetary institution is for-profit.
2. Sell the stock (i.e. the contract) you think is false for as high as you can get on the open market. Buy more stock of what you think is true from others who are willing to sell to you.
3. Just buy and sell stocks depending on what you think is the best prediction, based on what you hear on the news, gossip you hear from neighbors, and predictions from the tea leaves. Keep doing this until X definitely happens or X definitely does not happen (in which case you cash in your stock contracts, if you bet correctly on what happened), or a market crash results because someone discovers that the weak nuclear force actually allows you to make nuclear bombs out of orange juice, and Einstein and the gang were lying about it and distracting you by talking about dice-playing gods.

(what I described above is the simplest and most basic form I found; refer to the Wikipedia article for better elaborations)

Internet version:

1. Hack Intrade so that the topic you want to bet on is in their list of markets. Or better yet just hack Intrade and put one million dollars into your account.

Prediction Market: Analysis

Prediction markets avoid groupthink by utilizing the invisible hand. Someone selling you a stock might be an idiot who can't read the tea leaves properly. Or the seller might have knowledge you do not possess, so maybe buying the stock wasn't such a good idea after all? Remember: if you can't find who the sucker on the table is, that sucker is you!! You can't simply assume that what your neighbor says is true and you should sell as many stock of X as possible: maybe he or she is trying to take advantage of you to get your hard-earned cash. Groupthink in such a mistrusting environment gets hard to sustain. Prediction markets work better with very large groups of people, so that you get practical anonymity (although not perfect, in theory you or anyone else can keep track of who's selling to who; online versions are also likely to hide user identities). Anonymity in the prediction market also has the advantages previously described under Delphi Method above.

Prediction markets achieve consensus by utilizing the invisible hand. The price point of any sale serves as an approximate judgment of the epistemic probability of X occurring (or not occurring, depending on the contract that got sold). This gives a real-time signal on what the group of traders as a whole think the probability of X occurring is.

The prediction market's centrifugal phase is each individual trader's thought process as he or she considers whether to buy or sell stock, and at what price.

The prediction market's centripetal phase is any actual sale at any actual price point.

Prediction Market: Other Thoughts

Prediction markets are well-represented online; Intrade is just one of the more famous online prediction markets. Prediction markets appear to be the most popular and widely-known of the disputation arenas I've researched. These all tend to suggest that prediction markets are one of the better disputation arenas - but then remember that the Internet itself has no protection against groupthink.

Nominal Group Technique

Nominal group technique is a group decision-making process, appropriate for groups of many different sizes. This procedure's pen-and-paper form is faster than the pen-and-paper forms of the other disputation arenas discussed here.

Nominal Group Technique: How To Do It!

Pen and paper version:

1. The facilitator informs the group of the issue to be discussed.
2. Silent idea generation: group members are provided pen and paper, and are told to write down all ideas they can think of about the issue on the paper. They are given a fixed amount of time to do this (usually 10 to 15 minutes).
   • IMPORTANT: members are not allowed to discuss, show, or otherwise share their ideas with others during this stage. There's a reason it's called "silent".
3. Idea sharing: the facilitator asks group members, one at a time, to discuss their own ideas, until all members have shared their ideas. The facilitator writes the shared ideas into a whiteboard, or a similar location visible to all members.
   • IMPORTANT: debate is not allowed at this stage; only one member at a time can speak at this stage.
   • Group members may also add additional ideas and notes to their written-down ideas while waiting for their turn.
4. Group discussion: members may ask for clarification or further details about particular ideas shared by other group members. The group may agree to split ideas, or merge ideas, or group ideas into categories.
   • IMPORTANT: the facilitator must ensure that (1) all members participate, and (2) no single idea gets too much attention (i.e. all ideas must be discussed).
   • The discussion should be as neutral as possible, avoiding judgment or criticism.
   • The final result of this stage should be a set of options to be chosen among.
5. Ranking: members secretly rank the options from 1 (best) to N (worst), where N is the number of options generated in the previous stage. The facilitator then tallies the (anonymous) rankings (by adding the rankings for each option) and declares the option with the lowest total as the group consensus.

Unlike the previous procedures, which have been extrapolated into Internet versions, there is currently no Internet version of nominal group technique.

Nominal Group Technique: Analysis

Nominal group techniques avoid groupthink by the two "secret" steps: silent idea generation, and the secret ranking. Having members write down their ideas in the silent idea generation step helps them precommit to those ideas in the idea sharing step, even though more influential group members may present opposite or incompatible ideas. Although the group discussion step disallows explicit criticism of ideas, those criticisms are implicitly expressed during the secret ranking step (i.e. if you have a criticism of an idea, then you should rank it lower).

Nominal group techniques achieve consensus by the idea sharing, group discussion, and ranking steps. In particular, tallying of option rankings is the final consensus-achieving step.

The nominal group technique's centrifugal phase is largely the silent idea generation step, and is the most explicit centrifugal phase among the disputation arenas discussed here.

The nominal group technique's centripetal phase is largely the rest of the procedure.

Nominal Group Technique: Other Thoughts

A modified form of nominal group technique eats up a quarter of my recently-finished novel, Judge on a Boat, which I talked about on LessWrong here, and whose latest raw text source you can read online. Yes, this entire article is just a self-serving advertisement to garner interest in my novel o(^.^o)(o^.^)o.

Compared to the other procedures here, nominal group technique is more complicated and much more dependent on the centralized facilitator; the extreme dependency on the facilitator makes it difficult to create an automated online version. On the other hand, a small group of say 5 to 10 people can finish the nominal group technique in 1 to 2 hours; the other procedures tend to work better when done over several days, and are largely impossible (in pen-and-paper form) to do in a similar time frame. Even the real-time online versions of the other procedures are difficult to do within 2 hours. Prediction markets in particular tend to fail badly if too thin (i.e. not enough participants); for small groups with tight schedules, nominal group technique tends to be the best.

Conclusion

For small groups that need to make a decision within one or two hours, use nominal group technique. It's relatively unwieldy compared to the other disputation arenas, and is less ideal (it has fewer protections against groupthink, in particular), but is fast compared to the others. Also, one might consider parallelizing nominal group technique: split a large group into smaller, randomly-selected sub-groups, have each perform the procedure independently, and then have them send a representative that performs the idea sharing, group discussion, and ranking steps with other representatives (i.e. each sub-group's nominal group technique serves as the silent idea generation of the super-group of representatives). This tends to bias the super-group towards the agendas of the chosen representatives, but if speed is absolutely necessary for a large group, this may be the best you can do.

As mentioned above, prediction markets tend to fail badly if there are too few participants in the speculation market; use it only for extremely large groups that are impossible to coordinate otherwise. In addition, using prediction markets for policy decisions is effectively futarchy; you may want to see the (defunct?) futarchy_discuss Yahoo! group's message archives. In particular the earlier messages in the archive tend to discuss the general principles of prediction markets. Prediction markets are the most famous of the disputation arenas here, but remember that the Internet is not a decent disputation arena.

The Delphi methods seem to be a "dark horse" of sorts. I don't see much discussion online about Delphi methods; I'm not sure whether it's because it's been tried and rejected, or if it simply isn't well known enough to actually be tried by most people. I tend to suspect the latter, since if the universe were in the former case I would at least see some "Delphi Methods suck!!" blog posts.

Both prediction markets and Delphi methods are continuously repeated methods. At any time, the procedure may be stopped or repeated in order to make decisions. However, unlike the nominal group techniques, both are targeted more towards generating advice for decision-makers, rather than making actual decisions themselves.

It may be possible to organize a large, hierarchical group (say a company) with a prediction market for the rank-and-file, some key experts (who should be aware of the prediction market's results) running a Delphi method, and the key decision-making individuals (who read the Delphi method's report) at the top who form a decision using nominal group technique. For more democratic processes, a "poll-style" real-time online Delphi method by itself may work.

This is the public group instrumental rationality diary for the week of December 10th. It's a place to record and chat about it if you have done, or are actively doing, things like:

Related to: Knowing About Biases Can Hurt People

HT: Marginal Revolution

Paper.

Social psychologists have identified various plausible sources of ideological polarization over climate change, gun violence, national security, and like societal risks. This paper reports a study of three of them: the predominance of heuristic-driven information processing by members of the public; ideologically motivated cognition; and personality-trait correlates of political conservativism. The results of the study suggest reason to doubt two common surmises about how these dynamics interact. First, the study presents both observational and experimental data inconsistent with the hypothesis that political conservatism is distinctively associated with closed-mindedness: conservatives did no better or worse than liberals on an objective measure of cognitive reflection; and more importantly, both demonstrated the same unconscious tendency to fit assessments of empirical evidence to their ideological predispositions. Second, the study suggests that this form of bias is not a consequence of overreliance on heuristic or intuitive forms of reasoning; on the contrary, subjects who scored highest in cognitive reflection were the most likely to display ideologically motivated cognition. These findings corroborated the hypotheses of a third theory, which identifies motivated cognition as a form of information processing that rationally promotes individuals’ interests in forming and maintaining beliefs that signify their loyalty to important affinity groups. The paper discusses the normative significance of these findings, including the need to develop science communication strategies that shield policy-relevant facts from the influences that turn them into divisive symbols of identity.

The Worst-Run Big City in the U.S.

A six page article that reads as a very interesting autopsy of what institutional dysfunction in the intersection of government and non-profits looks like. I recommend reading the whole thing.

Minus the alleged harassment, city government is filled with Yomi Agunbiades — and they're hardly ever disciplined, let alone fired. When asked, former Board of Supervisors President Aaron Peskin couldn't remember the last time a higher-up in city government was removed for incompetence. "There must have been somebody," he said at last, vainly searching for a name.

Accordingly, millions of taxpayer dollars are wasted on good ideas that fail for stupid reasons, and stupid ideas that fail for good reasons, and hardly anyone is taken to task.

The intrusion of politics into government pushes the city to enter long-term labor contracts it obviously can't afford, and no one is held accountable. A belief that good intentions matter more than results leads to inordinate amounts of government responsibility being shunted to nonprofits whose only documented achievement is to lobby the city for money. Meanwhile, piles of reports on how to remedy these problems go unread. There's no outrage, and nobody is disciplined, so things don't get fixed.

You don't say?

In 2007, the Department of Children, Youth, and Families (DCYF) held a seminar for the nonprofits vying for a piece of $78 million in funding. Grant seekers were told that in the next funding cycle, they would be required — for the first time — to provide quantifiable proof their programs were accomplishing something.

The room exploded with outrage. This wasn't fair. "What if we can bring in a family we've helped?" one nonprofit asked. Another offered: "We can tell you stories about the good work we do!" Not every organization is capable of demonstrating results, a nonprofit CEO complained. He suggested the city's funding process should actually penalize nonprofits able to measure results, so as to put everyone on an even footing. Heads nodded: This was a popular idea.

Reading this I had to bite my hand in frustration.

There are two lessons here. First, many San Francisco nonprofits believe they're entitled to money without having to prove that their programs work. Second, until 2007, the city agreed. Actually, most of the city still agrees. DCYF is the only city department that even attempts to track results. It's the model other departments are told to aspire to.

But Maria Su, DCYF's director, admitted that accountability is something her department still struggles with. It can track "output" — what a nonprofit does, how often, and with how many people — but it can't track "outcomes." It can't demonstrate that these outputs — the very things it pays nonprofits to do — are actually helping anyone.

"Believe me, there is still hostility to the idea that outcomes should be tracked," Su says. "I think we absolutely need to be able to provide that level of information. But it's still a work in progress." In the meantime, the city is spending about $500 million a year on programs that might or might not work.

What the efficient charity movement has done so far looks much more impressive in light of this. Reading the rest of the article I think you can on your own identify the problems caused by lost purposes, applause lights and a dozen or so other faults we've explored here for years.

Discussions here are in many respects a comforting illusion, this is what humanity is like out there in the real world, almost at its best, well educated, wealthy and interested in the public good.

Yes it really is that bad.

This is the public group instrumental rationality diary for the week of November 27th. It's a place to record and chat about it if you have done, or are actively doing, things like:

Hi LessWrong! I'm a LWer from Melbourne, Australia, and I'm taking a 3 month road trip (with a friend) through parts of the United States. I figure I'd enjoy hanging out with some fellow rationalists while I'm over here! 

I attended the May Rationality minicamp in San Francisco (and made some friends who I'm hoping to meet up with again), but I've also heard good things about the LessWrong groups all over the United States. I'd like to meet some of the awesome people involved in these communities!

We've been planning this trip for a while now and have accommodation pretty much everywhere except for the second half of San Francisco. 

Itinerary

• 17th-21st Nov - Los Angeles, CA
• 21st-28th Nov - San Francisco, CA
• 28th Nov-1st Dec - Las Vegas, NV
• 2nd-3rd Dec - Flagstaff, AZ
• 3rd-7th Dec - Phoenix, AZ
• 7th-9th Dec - Santa Fe, NM
• 9th-10th Dec - El Paso, TX
• 10th-13th Dec - San Antonio, TX
• 13th-21st Dec - Austin, TX
• 21st-26th Dec - Dallas, TX
• 26th-29th Dec - San Antonio, TX
• 29th Dec-2nd Jan - New York City, NY
• 2nd-3rd Jan - San Antonio, TX
• 3rd-6th Jan - Houston, TX
• 6th-9th Jan - New Orleans, LA
• 9th-12th Jan - Memphis, TN
• 12th-15th Jan - Nashville, TN
• 15th-18th Jan - Atlanta, GA
• 18th-22nd Jan - Miami, FL
• 22nd-26th Jan - Orlando, FL
• 26th Jan-1st Feb - Washington DC
• 1st-4th Feb - Philadelphia, PA
• 4th-6th Feb - New York City, NY
• 6th-9th Feb - Mount Snow, VT
• 9th-13th Feb - Boston, MA
• 13th-15th Feb - New York City, NY
• 15th-26th Feb - Columbus, OH

If you're in one of these locations when I am, contact me! Either ahead of time or at short notice is fine. I'll be checking meetup posts and mailing lists for events that I can make it to as well, but if you happen to know of an event or meetup happening that fits the schedule, feel free to let me know in the comments.

Message or call me on 4242 394 657, email me at shokwave.sf@gmail.com - or you can leave a toplevel comment on this post, or message my LW account directly. Looking forward to meeting any and all of you!

This is the public group instrumental rationality diary for the week of October 29th. It's a place to record and chat about it if you have done, or are actively doing, things like:

Some time ago I learned of the metaphor of 'digging the bull's horn'. This might sound a little strange, since horns are mostly hollow, but imagine a bull's horn used to store black powder. In the beginning the work is easy and you can scoop out a lot powder with very little effort. As you dig down, though, each scoop yields less powder as you dig into the narrow part of the horn until the only way you can get out more powder is to turn the horn over a dump it out.

It's often the same way with learning. When you start out in a subject there is a lot to be learned (both in quantity of material you have not yet seen and in quantity of benefits you have to gain from the information), but as you dig deeper into a subject the useful insights come less often or are more limited in scope. Eventually you dig down so far that the only way to learn more is to discover new things that no one has yet learned (to stretch the metaphor, you have to add your own powder back to dig out).

It's useful to know that you're digging the bull's horn when learning because, unless you really enjoy a subject or have some reason to believe that contributing to it is worthwhile, you can know in advance that most of the really valuable insights you'll gain will come early on. If you want to benefit from knowing about as much stuff as possible, you'll often want to stop actively pursuing a subject unless you want to make a career out of it.

But, for a few subjects, this isn't true. Sometimes, as you continue to learn the last few hard things that don't seem to provide big, broadly-useful insights, you manage to accumulate a critical level of knowledge about the subject that opens up a whole new world of insights to you that were previously hidden. To push the metaphor, you eventually dig so deep that you come out the other side to find a huge pile of powder.

The Way seems to be one of those subjects you can dig past the end of: there are some people who have mastered The Way to such an extent that they have access to a huge range of benefits not available to those still digging the horn. But when it comes to other subjects, how do you know? Great insights could be hiding beyond currently obscure fields of study because no one has bothered to dig deep enough. Aside from having clear examples of people who came out the other side to give us reason to believe it's worth while to deep really deep on some subjects, is there any way we can make a good prediction about what subjects may be worth digging to the end of the bull's horn?

Is there a useful heuristic for detecting rationally-challenged texts (as in Web pages, forum posts, facebook comments) which takes relatively superficial attributes such as formatting choices, spelling errors, etc. as input? Something a casual Internet reader may use to detect possibly unworthy content so they can suspend their belief and research the matter further. Let's call them "text smells" (analogue to code smells), like:

1. too much emphasis in text (ALL CAPS, bold, color, exclamations, etc.);
2. walls of text;
3. little concrete data/links/references;
4. too much irrelevant data and references;
5. poor spelling and grammar;
6. obvious half-truths and misinformation.

Since many crackpots, pseudoscientific con artists, and conspiracy theorists seem to have cleaned up their Web sites in recent years, I wonder do these low-cost baloney detection tools might be of real value. Does anyone know of any studies or analyses of correlation between these basic metrics and the actual quality of the content? Can you think of some other smells typical of Web baloney?

What about an online group for high schoolers devoted to refining the art of human rationality?

Hello rationalists-in-training of the internet. My name is Joseph Gnehm, I am 15 and I live in Montreal. Discovering LessWrong had a profound effect on me, shedding light on the way I study thought processes and helping me with a more rational approach. As a teenager in high school, I wish I could share LessWrong's teachings and philosophies with others at my level. 

It would be awesome if we could create a list for the interests of LessWrong readers who are in their teens/in high school. I think this would allow a rational online community such as LessWrong to help develop more rationalists whether by outlining some plans to start rationality clubs in high school or discussing ways teenagers an approach rationality. I also think it would help more timid readers to express themselves and talk with other teenagers about common interests (adults could be allowed in to, if they are deemed appropriate for the community). Correct me if I'm wrong, but rationality training should start as soon as possible in the development process and what better age group to target than teenagers? Adolescence is a crucial transitional phase psychologically, biologically and culturally. I would love to see more collected articles on the evolution of rationality in the amazing, flexible mind of an adolescent. If the goal of this blog is to train humans to be rational-minded, more importance should be allocated to training teenagers. I do not think it hasn't happened yet for want of need among teenagers and if we concentrate some resources, gather a list of interested individuals and garner some interest we can make this work. This article is a good example of something that could be distributed in the proposed group:

For LW readers under 20: Note that the Thiel Fellowships (20 under 20) are now open for their next round of applications, and as they put it, "you have a huge readership of folk who would make great applicants". More info here. (from http://lesswrong.com/lw/f9r/weekly_lw_meetups_austin_berlin_cambridge_uk/)

There is also this LessWrong Highschoolers Facebook group created by Curtis SerVaas:

https://www.facebook.com/groups/201577993258819/

I recently Skyped (not officially a verb yet?) Anna Salamon who is the Executive Director of CFAR (Center for Applied Rationality). We had begun to develop this proposal. She is on the e-mail list and will be involved as a quasi-supervisor person. You can reach her at anna@appliedrationality.org. Drop me a one-line e-mail with your name, age, and situation at josephgnehm@gmail.com if you'd like to join the list. Speak up! Teenagers should be the subject of concentrated effort on LessWrong. We are the future, help us to reach the fruits of human rationality.

- Joseph Gnehm

Related to: Voting is like donating thousands of dollars to charity, Does My Vote Matter?

And voting adds legitimacy to it.

Thank you.

#annoyedbymotivatedcognition

A rather good 10 minute YouTube video presenting the results of several papers relevant to how conformity affects our thinking:

http://www.youtube.com/watch?v=TrNIuFrso8I

The papers mentioned are:

Sherif, M. (1935). A study of some social factors in perception. Archives of Psychology, 27(187), pp.17-22.

Asch, S.E. (1951). Effects of group pressure upon the modification and distortion of judgment. In H. Guetzkow (ed.) Groups, leadership and men. Pittsburgh, PA: Carnegie Press.
Asch, S.E. (1955). Opinions and social pressure. Scientific American, 193(5), pp.31-35.

Berns, G.S., Chappelow, J., Zink, C.F., Pagnoni, G., Martin-Skurski, M.E., and Richards, J. (2005) 'Neurobiological Correlates of Social Conformity and Independence During Mental Rotation' Biological Psychiatry, 58(3), pp.245-253.

Weaver, K., Garcia, S.M., Schwarz, N., & Miller, D.T. (2007) Inferring the popularity of an opinion from its familiarity: A repetitive voice can sound like a chorus. Journal of Personality and Social Psychology, 92(5), 821-833.

What techniques do other posters, here on LessWrong, use to monitor and counter these effects in their lives?

The video also lists some of the advantages to a society of having a certain amount of this effect in place.   Does anyone here conform too little?

Link.

A paper on the psychology of religious belief, Paranormal and Religious Believers Are More Prone to Illusory Face Perception than Skeptics and Non-believers, came onto my radar recently. I used to talk a lot about the theory of religious cognitive psychology years ago, but the interest kind of faded when it seemed that empirical results were relatively thin in relation to the system building (Ara Norenzayan’s work being an exception to this generality). The theory is rather straightforward: religious belief is a naturally evoked consequence of the general architecture of our minds. For example, gods are simply extensions of persons, and make natural sense in light of our tendency to anthromorphize the world around us (this may have had evolutionary benefit, in that false positives for detection of other agents was far less costly than false negatives; think an ambush by a rival clan).*

 

But enough theory. Are religious people cognitively different from those who are atheists? I suspect so. I speak as someone who never ever really believed in God, despite being inculcated in religious ideas from childhood. By the time I was seven years of age I realized that I was an atheist, and that my prior “beliefs” about God were basically analogous to Spinozan Deism. I had simply never believed in a personal God, but for many of earliest years it was less a matter of disbelief, than that did not even comprehend or cogently in my mind elaborate the idea of this entity, which others took for granted as self-evidently obvious. From talking to many other atheists I have come to the conclusion that Atheism is a mental deviance. This does not mean that mental peculiarities are necessary or sufficient for atheism, but they increase the odds.

And yet after reading the above paper my confidence in that theory is reduced. The authors used ~50 individuals, and attempted to correct demographic confounds. Additionally, the results were statistically significant. But to me the above theory should make powerful predictions in terms of effect size. The differences between non-believers, the religious, and those who accepted the paranormal, were just not striking enough for me.

Because of theoretical commitments my prejudiced impulse was to accept these findings. But looking deeply within they just aren’t persuasive in light of my prior expectations. This a fundamental problem in much of social science. Statistical significance is powerful when you have a preference for the hypothesis forwarded. In contrast, the knives of skepticism come out when research is published which goes against your preconceptions.

So a question for psychologists: which results are robust and real, to the point where you would be willing to make a serious monetary bet on it being the orthodoxy in 10 years? My primary interest is cognitive psychology, but I am curious about other fields too.

* In Gods We Trust and Religion Explained are good introductions to this area of research.

Considering the communities heavy reliance on such results I think we should answer the question as well.

This is the public group instrumental rationality diary for the week of October 29th. It's a place to record and chat about it if you have done, or are actively doing, things like:

Related to: RationalWiki's take on LW, David Gerard's Comments, Vladimir_M's comments, Public Drafts

I wanted to bring more attention to this argument because I've ran into related discussion several times in the comment section and because it demonstrates a failure mode that LessWrong may find itself vulnerable to.

Since it has been cited as a source especially on the reputation LessWrong may or may not have elsewhere I think readers should be aware Rational Wiki has a certain reputation here as well. I'm not talking about the object level disagreements such as cryonics, existential risk, many-worlds interpretation and artificial intelligence because we have some reasonable disagreement on those here as well. Even its cheeky tone while not helping its stated goals can be amusing. I'm somewhat less forgiving about their casual approach to epistemology and their vulnerability to cargo cult science, as long as it is peer reviewed cargo cult science.

While factually it is as about as accurate as Wikipedia, it is very selective about the facts that it is interested in. For example what would you expect from a site calling itself "Rational Wiki" to have on its page about charity. Do you expect information on how much good charity actually does? What kinds of charities do not do what they say on the label? How to avoid getting misled? The ethics of charity? The psychology, sociology or economics of charity?

I'm sorry to disappoint you but the article consists of some haphazardly arranged facts and stats on how much members of some religions give or are supposed to give to charity, a dig against Christianity and a non-sequitur unfavourable comparison of the US to Sweden. Contrast this with what you can find on the topic on sites like LessWrong or 80, 000 Hours. Basically the material presented is what a slightly left of centre atheist needs to win an internet debate. As is much of the rest of the site.

Indeed some entries have a clear ideological bias that is quite startling to behold on a "rational wiki" and it has been noted by some.

Now to avoid any misunderstandings there are good articles, a few LWers are contributors to the rational wiki and there is certainly nothing wrong with being a left of centre atheist! Nearly everyone on this site is an atheist, and people who identify as left wing politically form a large majority here. The tribal markers and its political agenda aren't the biggest problem. Sites with all sorts of agendas, even political ones, promoting rationality are a good thing.

Its problem is that it is an ammunition depot to aid in winning debates. Very specific kinds of debates too. This may sound harsh, but consider: How many people reading the site that aren't already atheists will change their mind on religion? How many people who follow a "crankish" belief won't do so afterwards? While I'm sure it happens the site obviously isn't optimized for this. How many people will read the wiki and try to find errors and biases in their own thinking to debug it instead of breaking if further with confirmation bias or using it as a club? How many will apply this knowledge to help them with any real world problems? Truth seeeking? As a source or community that could aid in that quest it is less useful and reliable than Wikipedia, which while a rather good and extensive encyclopaedia (despite snickering to the contrary) has a subtly but importantly different stated goal.

What else remains? What other plausible function does it serve?

Related to: Eliezer's Sequences and Mainstream Academia, Intellectual insularity and productivity, I Stand by the Sequences, Why don't people like markets?

Looking at some of the more recent arguments against them showing up in discussions I've been quite disappointed, they seem betray a sort of lack of background knowledge or opinions built up from a bottom line of "markets are baaad therefore prediction markets are baaad". The casual arguments for them are lacking as well. I will say the same of other discussions on economic, since it is apparently suddenly too mind-killing or too political to talk about markets and similar things at all. We didn't use to have tribal alerts flying up in our brains discussing such matters.

The Overcoming Bias community started with an assumption of certain kinds of background knowledge, this included economics and things like game theory. In the early days of LessWrong/Overcoming Bias Eliezer did a whole sequnece on filling in people on Quantum mechanics which despite his claims to the contrary doesn't seem that vital (if still important).  

We now have a different demographic that we used to. Not only that, we  now have young people basically using the sequences as their primary source for education on matters of human rationality, quite different from the autodidacts exploring the literature on their own terms who where common in previous years. We've recognized this to a certain extent. We wrote a series of introductory sequences and articles to fill in such background knowledge explicitly such as Yvain's recent one on Game Theory. Also part of the reason we now have a norm of more citations that EY originally did is to give study and research aids to people. Indeed I think adding comments to old articles featuring more citations or editing those in would be wise so as to avoid misconceptions.

I think we need several sequences on economics, and a good one to start would be one systematically investigating prediction markets. To a certain extent just reading Robin Hanson's relevant posts on this topic would do much the same, but unfortunately we don't have an organized series of sequences by him (beyond the tags he uses on his articles). I still hope Karmakaiser or someone else will one day undertake a project of writing up summary articles that organize links to RH's posts into sequences so new members will read them as well.

I'd write these myself but I just don't have a good background in what works and studies influence the positions of early key LW authors on economics and its relevance to rationality. I'm also only beginning my studies in that area since my background is in the hard sciences with only some half-serious opinions formed from Moldbuggian insights and 20th century social science.   

From Julian Sanchez, a brilliant idea unlikely to be implemented:

American politics sometimes seems like a contest to see which group of partisans can take greater umbrage at the most recent outrageous remark from a member of the opposing tribe.  As a mild countermeasure, I offer a modest proposal for American universities. All freshmen should be required to take a course called “Offense 101,” where the readings will consist of arguments from across the political and philosophical spectrum that some substantial proportion of the student body is likely to find offensive. Selections from The Bell Curve. Essays from one of the New Atheists and one of their opponents, and from hardcore pro-lifers and pro-choicers. Ward Churchill’s “little Eichmanns” monograph. Defenses of eugenics, torture, violent revolution, authoritarianism, aggressive censorship, and absolute free speech. Positive reviews of the Star Wars prequels. Assemble your own curriculum—there’s no shortage of material.

For each reading, students will have to make a good faith, unironic effort to reconstruct the offensive argument in its most persuasive form, marshaling additional supporting evidence and amending weak arguments to better support the author’s conclusion. Points deducted if an observer can tell the student doesn’t really agree with the position they’re defending.

Only after this phase is complete will students be allowed to begin rebutting the arguments. Anyone who thinks it’s relevant to point out that the argument is offensive (or bigoted, sexist, unpatriotic, fascistic, communistic, whatever) will receive a patronizing look from the professor that says: “Yes,  obviously, did you not read the course title? Let’s move on.” Insofar as these labels are shorthand for an argument that certain categories of views are wrong and can be rejected as a class, the actual argument will have to be presented. 

Many of us are familiar with the marshmallow test.If you are not, here.

It is predictive of success, income, level of education, and several other correlated measures.

I'm here to argue for the marshmallow eaters, as a devil's advocate. Contra Ainslie, for instance. I do it out of genuine curiosity, real suspicion, and maybe so that smart people get me back to my original position, pro-long term.

There is also the e-marshmallow test (link is not very relevant), in which children have to face the tough choice between surfing an open connected computer with games, internet etc... and waiting patiently for the experimenter to get back. Upon the experimenter's arrival, they get a pile of marshmallows. I presume it also correlates with interesting things, though haven't found much on it.

I have noticed that rationalists, LessWrongers, Effective Altruists, Singularitarians, Immortalists, X-risk worried folk, transhumanists, are all in favor of taking the long view.  Nick Bostrom starts his TED by saying: "I've been asked to take the long view"

I haven't read most of Less Wrong, but did read the sequences, the 50 top scoring posts and random posts. The overwhelming majority view is that the long view is the most rational view. The long term perspective is the rational way for agents to act.

Lukeprog, for instance, commented:

"[B]ut imagine what one of them could do if such a thing existed: a real agent with the power to reliably do things it believed would fulfill its desires. It could change its diet, work out each morning, and maximize its health and physical attractiveness."

To which I responded:

I fear that in this phrases lies one of the big issues I have with the rationalist people I've met thus far. Why would there be a "one" agent, with "its" desires, that would be fulfilled. Agents are composed of different time-spans. Some time-spans do not desire to diet. Others do (all above some amount of time). Who is to say that the "agent" is the set that would be benefited by those acts, not the set that would be harmed by it.
My view is that picoeconomics is just half the story.
In this video, I talk about picoeconomics from 7:00 to 13:20 I'd suggest to take a look at what I say at 13:20-18:00 and 20:35-23:55, a pyramidal structure of selfs, or agents. 

So you don't have to see the video, let us design a structure of selfhood.

First there is intertemporal conflict, conflict between desires that can be fulfilled at different moments of time. Those reliably fall under a hyperbolic characterization, and the theory that described this is called Picoeconomics, mostly developed by George Ainslie in his Breakdown of Will and elsewhere.

But there is also time-length, or time-span conflict.  The conflict that arises from the fact that you are, at the same time, the entity that will last 200milliseconds, the entity that will last one second, and the entity that will last a year, or maybe, a thousand years.

What do we (humanity) know about personal identity at this point in history? If mainstream anglophone philosophical thought is to be trusted, we have to look for Derek Parfit's work Reasons and Persons, and posterior related work, to get that.

I'll sum it up very briefly: As far as we are concerned, there are facts about continuity of different mental classes. There is continuity of memory, continuity of conscious experience, continuity of psychological traits and tendencies, continuity of character, and continuity of inferential structure (the structure that we use to infer things from beliefs we acquire or access).   

For each of these traits, you can take an individual at two points in time and measure how related I_t1 and I_t2 are with respect to that psychological characteristic.  This is how much I at T₂ is like himself at T₁.

Assign weights for traits according to how much you care (or how important each is in the problem at hand) and you get a composed individual, for which you can do the same exercise, using all of them at once and getting a number between 0 and 1, or a percentage. I'll call this number Self-Relatedness, following the footsteps of David Lewis.

This is our current state of knowledge on Personal Identity: There is Trait-Relatedness, and there is Self-Relatedness. After you know all about those two, there is no extra fact about personal identity. Personal Identity is a confused concept, and when we decompose it into less confused, but more useful, sub-sets, there is nothing left to be the meta-thing "Personal Identity".

Back to the time-length issue, consider how much more me  the shorter term selves are (that is how much more Self-Relatedness there is between any two moments within them). 

Sure if you go all the way down to 10 milliseconds, this stops being true, because there are not even traits to be found. Yet, it seems straightforward that I'm more like me 10 seconds ago than like me 4 months ago, not always, but in the vast majority of cases.

So when we speak of maximizing my utility function, if we overlook what me is made of, we might end up stretching ourselves to as long-term as we possibly can, and letting go of the most instantaneous parts, which de facto are more ourselves than those ones.

One person I met from the LessWrong Singinst cluster claimed: "I see most of my expected utility after the singularity, thus I spend my willpower entirely in increasing the likelihood of a positive singularity, and care little about my current pre-singularity emotions"

Is this an amazing feat of self-control, a proof that we can hope to live according to ideal utility functions after all? Or is it a defunct conception of what a Self is?

I'm not here to suggest a canonical curve of time-lengths of which the Self is composed. Different people are different in this regard. Some time-lengths are stretchable, some can be shortened. Different people will also value the time-lengths differently. 

It would be unreasonable for me to expect that people would, from now on, put on a disclaimer on their writings "I'm assuming 'rational' to mean 'rational to time-lenghts above the X treshold' for this writing". It does, however, seem reasonable to keep an internal reminder when we reason about life choices, decisions, and writings, that not only there are the selves which are praised by the Rationalist cluster, the long term ones, but also, the short term ones.

A decision to eat the marshmallow can, after all, be described as a rational decision, it all depends on how you frame the agent, the child.

So when a superintelligence arises that, despite being Friendly and having the correct goals, does the AGI equivalent of scrolling 9gag, eating Pringles and drinking booze all day long, tell the programmers that the concept of Self, Personal Identity, Agent, or Me-ness was not sufficiently well described, and vit cares too much for vits short-term selves. If they tell you: "Too late, vit is a Singleton already" you just say "Don't worry, just make sure the change is ve-e-e-ery slow..."

A short argument from an interesting blog.

Anti-Groupism

Basic Aretaevian talking points:

1. Human brains are effectively populated by rabbits.  Your conscious mind is like a very small person attempting to ride a large herd of rabbits, which aren't all going the same direction.  Your job is to pretend to be in control, and make shit up to explain where the rabbits went, and what you did.
2. Humans bunny brains are optimized for social activity, not intellectual activity.  If your brain thinks principles first, instead of groups first, it's broken, and not just a little bit.
3. Of course, this means that anyone thinking group first is almost completely full of crap regarding their reasoning process.  They're (99.86% certainty) making shit up that makes the group look good, and the actual rational value of the statement is near zero.  The nominal process "A->B->C" is actually C, now let's backfill with B and A.  
4. Therefore I'm almost only interested in listening to folks who are group-free.  If your brain is broken in the kind of way that prohibits group-attachment...then you're far far more likely to be thinking independently, and shifting perspectives.
5. Aside:  FWIW, this is the core (unsolvable?) problem that inhabits rationalist groups.  There is a deep and abiding conflict between groupism and thinking.  The Randians have encountered this most loudly, but it's also there in the libertarians, the extropians, the David Deutch-led popperian rationalists, and the LessWrongers.

New discovery, shouldn't have been as surprising as it was.   When looking for folks who are group-avoidant, I seem to have phenomenally good luck finding great people when talking with Gays from non-leftist areas (rural Texas, Tennessee, downstate Illinois).  Because they don't/can't fit in with their local culture, and often can't conveniently exit, they become interesting people.   It's a surprisingly good metric.

This is the public group instrumental rationality diary for the week of October 15th. It's a place to record and chat about it if you have done, or are actively doing, things like:

I've seen there's discussion on LW about rationality, namely, about what it means. I don't think a satisfactory answer can be found without defining what rationality is not. And this seems to be a problem. As far as I know, rationality on LW does not include systematic methods for categorizing and analyzing irrational things. Instead, the discussion seems to draw a circle around rationality. Everyone on LW is excepted to be inside this circle - think of it as a set in a Venn diagram. On the border of the circle there is a sign saying: "Here be dragons". And beyond the circle there is irrationality.

How can we differentiate the irrational from the rational, if we do not know what the irrational is?

But how can we approach the irrational, if we want to be rational?

It seems to me there is no way to give a satisfactory account of rationality from within rationality itself. If we presuppose rationality is the only way to attain justification, and then try to find justification for rationalism (the doctrine according to which we should strive for rationality), we are simply making a circular argument. We already presupposed rationalism before trying to find justification for doing so.

Therefore it seems to me we ought to make a metatheory of rationality in order to find out what is rational and what is irrational. The metatheory itself has to be as rational as possible. That would include having an analytically defined structure, which permits us to at least examine whether the metatheory is logically consistent or inconsistent. This would also allow us to also examine whether the metatheory is mathematically elegant, or whether the same thing could be expressed in a simpler form. The metatheory should also correspond with our actual observations so that we could figure out whether it contradicts empirical findings or not.

How much interest is there for such a metatheory?

A post from Gene Expression by Razib Khan who some of you may also know from the old gnxp site or perhaps from his BHTV debate with Eliezer. Some thoughts on the problem of trying to optimize your interactions to help you be less wrong. Your time is quite limited. Expect trade-offs.

A few days ago I was having drinks with some friends, and it came up that some of them had only recently become conscious of the fact that I leaned more toward the Republican party than the Democratic (I had remarked that my wife preferred that I keep my sideburns, as otherwise I would look too much like a Republican…though I sort of was one!). More shockingly for them was that I did not consider myself a liberal. I was somewhat bemused by the whole situation because it isn’t as if I’m particularly shy about expressing my various politically-incorrect opinions on any specific topic at work or play (these are people who I have met within the past ~2 years).

I assume that the problem here is that I violated a cognitive schema: liberal people are smarter than conservative people. Since I was conservative, they were, logically, smarter than me. The reality is probably not so convenient for the theory in this case, generating some dissonance. In the course of conversation I expressed frankly what I actually do hold to be a rough & ready approximation of my attitude toward discussion: I have almost no interest in persuading anyone of the truth of my particular views on any issue. This was relevant in that context because on occasion people try and draw me out as to the details of my disagreement with the consensus on an array of topics, when I often have no interest in expending the mental energy to do any such thing. It isn’t that I’m worried about getting into any argument with everyone else in the room. My friends are mostly natural scientists so I am very confident that I can alone hold my ground on any topic having to do with history and quantitative social science. Rather, the problem is my worry as to the point of it all. Who exactly is being edified by such exchanges? I never learn anything, as I am well acquainted with the standard arsenal of conventional Left-liberal talking points, while my interlocutors are often too amazed as my incomprehensible existence (i.e., not stupid, but not right-thinking) to really take in anything I’m saying.

Yet on a one-on-one basis I am much more likely to be open to a deep and thorough exchange. Why? The dynamic of signalling and group conformity is strongly dampened by removing third party observers from the interaction. With that tension removed I myself often feel less irritated if I have to invest a great deal of background information to make my own position clearer. Similarly, I often feel that my interlocutors are much less likely to trot out hackneyed and unpersuasive, but group approved, arguments.* There is quite often idiocy in crowds.

Ultimately we have to take a step back and reflect on what the point of it all is. For me the answer is rather easy: the point of it all is to understand the shape of reality as best as I can. It is impossible to do such a thing sitting back in an armchair and reflecting as an individual. Learning is a social process. You need feedback from others, and you need to mine and cull appropriate data and analyses from those who are more well versed in a given topic than you are. This is not easy, and time is finite. Avoiding stupid people is easy. The more difficult trick, at least for me, is avoiding smart people who offer stupid opinions on topics with which they are absolutely unfamiliar.** Creationist engineers are classic cases of the power of ignorance in the hands of the intelligent.

This brings me to learning more generally. Obviously I have no problem with people being autodidacts. Today the ability for one to be an autodidact has greatly expanded, but with power comes responsibility, and the necessity of prudence. I’m speaking obviously about the internet. But now we have the rise of online education. Recently MRUniversity opened, and Khan Academy is already rather famous. Tyler Cowen and Alex Taborrak’s endeavor has already received some praise:

MRU is ultimately aiming for a better actual education, not a better means of signaling. Cowen and Tabarrok are betting that there is an extraordinary amount of dead weight in current university classes (for example, on MRU the professor need not repeat himself as he inevitably must during live lectures, because if a student requires repetition, she can just watch the video again). “You can think of this,” Cowen says, laughing for the only time during our phone conversation and only lightly, “as a marginal attempt—a marginal revolution, so to speak—to get education to be more about learning.”

I am moderately skeptical, but I also think such experiments are necessary. Over the long term it seems likely that new forms of educational delivery and assessment with replace the middle and lower tiers of American higher education, and modify even the elite levels. But I don’t think we know yet what the exact nature of the information ecology is going to be.

Here is what I’d really like in the future: an app which analyzes someone’s stream of assertions and immediately assesses whether they are full of crap or not.*** There are many domains where I can do this analysis myself, and know to tune someone out because I know they’re signalling to ignorant people. But, there are many, many, more domains where I am ignorant and lost, and may fall prey to the bluffs and assertions of high caliber signalers, who have fashioned the simulacrum of intelligence. More concretely, people who are trying to impress without deep knowledge often fumble on many facts, something which could be run through an application such as WolframAlpha.

Of course things have changed a great deal. Over the past few years smartphones have cast a pall over the skills of the professional bullshitter. I think that there has been a qualitative change for the better.  Bullshitters known that they need to be cautious, so there is a preemptive effect.

* I am never in social circumstances where the political context is conservative.

** You also need to avoid socializing only with your own ideological set. This is easy for me since I don’t socialize with anyone who shares my politics or metaphysical opinions.

*** Looking things up manually is time consuming.

Debating group consensus with the group is less productive than debating it with individuals making up that group. Avoiding smart people who offer stupid opinions on topics with which they are absolutely unfamiliar is expensive. The internet has made this somewhat harder. We should like make an app to fix this or something.

This is the public group instrumental rationality diary for the week of October 1st. It's a place to record and chat about it if you have done, or are actively doing, things like:

You have desires. You also have desires about your desires: perhaps you desire cake but you also desire that you didn't desire cake. You also have desires about the processes which produce your desires: perhaps you desire X and Y but only because of a weird evolutionary turn and you wish the processes which created your desires weren't so far beyond your own control.

But what should you do, when these different kinds of desires are in conflict with each other? If you could reflect upon and then rewrite your own desires, how should you choose to resolve those conflicts?

Nozick (1993) proposes 23 constraints on rational preferences, which one could also interpret as 23 constraints on the process of resolving conflicts among one's preferences. I reproduce this passage below, for those who are interested:

Let me emphasize that my purpose is not to endorse the particular conditions I shall put forward or to defend their particular details. Rather, I hope to show what promising room there is for conditions of the sort that I discuss...

Some of these conditions are justified by instrumental considerations, such as the “money pump” argument that preferences be transitive, while others are presented as normatively appealing on their face. (Unless these latter can be given an instrumental justification also, isn’t this already a step beyond instrumental rationality?) Contemporary decision theory takes this one step beyond Hume: group of them together can be. Let us suppose that there are normative principles specifying the structure of several preferences together and that these principles are conditions of rationality. (The literature contains putative counterexamples and objections to some of the Von Neumann–Morgenstern conditions; the point here is not to use those particular ones but some such appropriate set of conditions.)

I. The person satisfies the Von Neumann–Morgenstern or some other specified appropriate set of conditions upon preferences and their relations to probabilities.

This suggests at least one further condition that a person’s preferences must satisfy in order to be rational, namely, that she must prefer satisfying the normative conditions to not satisfying them. Indeed, for any valid structural condition C of rationality, whether rationality of preference, of action, or of belief:

II. The person prefers satisfying the rationality condition C to not satisfying C.

(This condition should be stated as a prima facie one or with a ceteris paribus clause, as should many of the ones below. The person who knows that he will be killed if he always satisfies the condition that indifference be transitive, or the condition that he not believe any statement whose credibility is less than that of an incompatible statement, may well prefer not to.) Since the person is, let us assume, instrumentally rational,

III. The person will, all other things being equal, desire the means and preconditions to satisfying rationality conditions C.

These rationality conditions C not only concern the structure of preferences but also include whatever the appropriate structural conditions are on the rationality of belief. Hence the person will desire the means and preconditions of rational belief, she will desire the means and preconditions for the effective assignment of credibility values (and for deciding about the utility of holding a particular belief).

A person lacks rational integration when he prefers some alternative x to another alternative y, yet prefers that he did not have this preference, that is, when he also prefers not preferring x to y to preferring x to y. When such a second-order preference conflicts with a first-order one, it is an open question which of these preferences should be changed. What is clear is that they do not hang together well, and a rational person would prefer that this not (continue to) be the case. We thus have a requirement that a person have a particular third-order preference, namely, preferring that the conflict of preferences not obtain. Let S stand for this conflict situation, where the person prefers x to y yet prefers not having this preference, that is, let S stand for: xPy & [not-(xPy) P (xPy)]. Then

IV. For every x and y, the person prefers not-S to S, all other things being equal. 

This does not mean the person must choose not-S over S no matter what. An addict who desires not to desire heroin may know that he cannot feasibly obliterate his first-order desire for heroin, and thus know that the only way to resolve the conflict of preferences is to drop his second-order desire not to have that first-order desire. Still, he may prefer to keep the conflict among desires because, with it, the addiction will be less completely pursued or his addictive desire less of a flaw.

Hume claims that all preferences are equally rational. But an under- standing of what a preference is, and what preferences are for, might make further conditions appropriate. In recent theories, a preference has been understood as a disposition to choose one thing over an- other.8 The function of preferences, the reason evolution instilled the capacity for them within us, is to eventuate in preferential choice. But one can make preferential choices only in some situations: being alive, having the capacity to know of alternatives, having the capacity to make a choice, being able to effectuate an action toward a chosen alter- native, facing no interference with these capacities that makes it im- possible to exercise them. These are preconditions (means) for prefer- ential choice. Now, one does not have to prefer that these conditions continue; some people might have reason to prefer being dead. But they need a reason, I think; the mere preference for being dead, for no reason at all, is irrational. There is a presumption that the person will prefer that the necessary conditions for preferential choice, for making any preferential choice at all, be satisfied; she need not actually have the preference, but she needs a reason for not having it.

V. The person prefers that each of the preconditions (means) for her making any preferential choices be satisfied, in the absence of any particular reason for not preferring this.

So a person prefers being alive and not dying, having a capacity to know of alternatives and not having this capacity removed, having the capacity to effectuate a choice and not having this capacity destroyed, and so on. Again, we might add

VI. The person prefers, all other things being equal, that the capacities that are the preconditions for preferential choice not be interfered with by a penalty (= a much unpreferred alternative) that makes him prefer never to exercise these capacities in other situations.

There is something more to be said about reasons, I think. (I propose this very tentatively; more work is needed to get this matter right.)

Suppose I simply prefer x to y for no reason at all. Then I will be willing, and should be willing, to reverse my preference to gain something else that I prefer having. I should be willing, were it in my power, to reverse my preference, to now start preferring y to x, in order to receive 25 cents. I then would move from a situation of preferring x to y to one of preferring y to x and having 25 additional cents. And won't I prefer the latter to the former? Perhaps not, perhaps I strongly prefer x to y, and do so for no reason at all. Having a strong preference for no reason at all is, I think, anomalous. Given that I have it, I will act upon it; but it is irrational to be wedded to it, paying the cost of pursuing it or keeping it when I have no reason to hold it. Or perhaps I prefer preferring x to y to not having this preference, and I prefer that strongly enough to outweigh 25 cents. So this second-order preference for preferring x to y might make me unwilling to give up that preference. But why do I have this second-order preference? I want to say that, unlike any arbitrary first-order preference, a second-order preference requires a reason behind it. A second-order preference for preferring x to y is irrational unless the person has some reason for preferring x to y. That is, he must have a reason for preferring to have that first-order preference-perhaps his mother told him to, or perhaps that preference now has become part of his identity and hence something he would not wish to change-or have a direct reason for preferring x to y, a reason concerning the attributes of x and y. But what is a direct reason? Must a reason in this context be anything more than another preference? It must at least be another preference that functions like a reason, that is, one that is general, though defeasible. To have a reason for preferring x to y is standardly thought to involve knowing some feature F of x such that, in general, all other things being equal, you prefer things with F to things without them, among things of the type that x is. (Preferring cold drinks to warm does not require preferring cold rooms to warm ones.)

VII. If the person prefers x to y, either: (a) the person is willing to switch to preferring y to x for a small gain, or (b) the person has some reason to prefer x to y, or (c) the person has some reason to prefer preferring x to y to not doing that.

I don't say that all of a person's preferences require reasons for them- it is unclear what to say about ones that are topmost; perhaps they are anchored by ones under them-but first-level ones do require reasons when the person is not willing to shift them. Once we are launched within the domain of reasons for preferences, we can consider how more general reasons relate to less general ones, we can impose consistency conditions among the reasons, and so forth. The way becomes open for further normative conditions upon preferences, at least for those preferences a person is not willing to switch at the drop of a hat. Especially in the case of preferences that go against the preconditions for preferential choice mentioned above, a person will need not just any reasons but reasons of a certain weight, where this means at least that the reasons must intertwine with many of the person's other preferences, perhaps at various levels.

We also might want to add that the desires and preferences are in equilibrium, in that knowing the causes of your having them does not lead you to (want to) stop having them. The desires and preferences withstand a knowledge of their causes.

VIII. The person's desires and preferences are in equilibrium (with his beliefs about their causes).

Since preferences and desires are to be realized or satisfied, a person whose preferences were so structured that he always wanted to be in the other situation-preferring y to x when he has x and preferring x to y when he has y-would be doomed to dissatisfaction, to more dissatisfaction than is inherent in the human condition. The grass shouldn't always be greener in the other place. So

IX. For no x and y does the person always prefer x to y when y is the case and y to x when x is the case. (His conditional preferences are not such that for some x and y he prefers x to y/given that y is the case, and prefers y to x/given that x is the case.)

Desires are not simply preferences. A level of filtering or processing takes place in the step from preferences to desires-as (we shall see) another does in the step from desires to goals. We might say that rational desires are those it is possible to fulfill, or at least those you believe it is possible to fulfill, or at least those you don't believe it is impossible to fulfill. Let us be most cautious and say

X. The person does not have desires that she knows are impossible to fulfill.

Perhaps it is all right to prefer to fly unaided, but it is not rational for a person to desire this. (It might be rational, though, to wish it were possible.) Desires, unlike mere preferences, will feed into some decision process. They must pass some feasibility tests, and not simply in isolation: your desires must be jointly copossible to satisfy. And when it is discovered they are not, the desires must get changed, although a desire that is altered or dropped may remain as a preference.

Goals, in turn, are different from preferences or desires. To have or accept goals is to use them to filter from consideration in choice situations those actions that don't serve these goals well enough or at all. For beings of limited capacity who cannot at each moment consider and evaluate every possible action available to them-try to list all of the actions available to you now-such a filtering device is crucial. Moreover, we can use goals to generate actions for serious consideration, actions that do serve these goals. And the goals provide salient dimensions of the outcomes, dimensions that will get weight in assessing the utility of these outcomes. Given these multiple and important functions of goals, one would expect that for an important goal that is stable over time we would devote one of our few channels of alertness to it, to noticing promising routes to its achievement, monitoring how we currently are doing, and so on.

How do our goals arise? How are they selected? It seems plausible to think that they arise out of a matrix of preferences, desires, and beliefs about probabilities, possibilities, and feasibilities. (And then goals reorganize our desires and preferences, giving more prominence to some and reversing others because that reversed preference fits or advances the goal.) One possibility is that goals arise in an application of expected utility theory. For each goal Gi, treat pursuing goal Gi as an action with its own probability distribution over outcomes, and compute the expected utility of this "action." Adopt that goal with the maximum expected utility, and then use it to generate options, exclude others, and so forth.

There is an objection to this easy way of fitting goals within an expected utility framework. The effect of making something Gi a goal is a large one. Now Gi functions as an exclusionary device and has a status very different from another possible goal Gj that came very close but just missed having maximum expected utility. A marginal difference now makes a very great difference. It seems that large differences, such as one thing setting the framework whereby other things are excluded, should be based upon pre-existing differences that are significant. Consider the descriptive finding a dominant structure, and she uses mechanisms such as combining and altering attributes and collapsing alternatives in order to get one action weakly dominating all others on all (considered) attributes. Thereby, conflict is avoided, for one action clearly is best; there is no reason for doing another. Will such dominance always set up a gulf between actions that is significant enough to make a qualitative difference with large effects and so be applicable to the formation of goals? Yet one action can weakly dominate another when there are six dimensions, the two actions tying on five of these while the first action is (only) slightly better on the sixth. Even in this framework, we seem to need more than simply weak dominance; perhaps we need strong winning on one dimension or winning on many of them.

Returning to the expected utility framework, we might say that goal Gi is to be chosen not simply when it has maximum expected utility but when it beats the other candidate goals decisively. For each j, EU(Gi) − EU(Gj) is greater than or equal to some fixed positive specified quantity q. (There remains a similar but smaller problem, though. Gi beats the other goals decisively, yet there is no decisive difference between beating decisively and not doing so; the difference EU(Gi) − EU(Gj) might barely reach, or just fail to reach, q.) To make something a goal is, in part, to adopt a desire to find a feasible route from where you are to the achievement of that goal. Therefore,

XI. A person will not have a goal for which he knows that there is no feasible route, however long, from his current situation to the achievement of that goal.

Moreover, we might say that a rational person will have some goals toward which she will search for feasible routes and not just have merely preferences and desires. She will filter out actions that cannot reach these goals, generate for consideration actions that might reach them, and so on. And some of these goals will have some stability, so that they can be pursued over time with some prospect of success.

XII. A person will have some stable goals.

A rational person will consider not only particular (external) outcomes but also what he himself is like, and he will have some preferences among the different ways he might be. Let Wp be the way the person believes he will be when p is the case; let Wq be the way he believes he will be when q is the case. (These include the ways that p, or q, will cause or shape or prompt him to be.) There is a presumption, which can be overriden by reasons, that preferences among ways of being will take precedence over lower-level preferences that are personal ones. (Personal preferences are ones derived solely from estimates of benefits to himself.)

XIII. If the person prefers Wp to Wq, then (all things being equal) he does not hold the (personal) preference of q to p.

Condition XIII holds that the way the person is, what kind of person he is, will have greater weight in his preferences than (what otherwise would be) his personal preferences. (Is this condition culture-bound and plausible only to people in certain kinds of cultures?)

The dutch book argument that someone's probability beliefs should satisfy the axioms of probability theory says that if they do not, and if she is willing always to bet upon such probability beliefs, then someone can so arrange things so that she is sure to lose money and hence reach a less preferred alternative. This argument says that if her (probabilistic) beliefs are irrational, she can be guaranteed to end up worse off on her utility scale. We might try the dual of this argument, imposing as a condition:

XIV. A person's desires are not such that acting upon them guarantees that she will end up with irrational beliefs or probabilities.

Various things might come under the ban of this condition: desiring to believe something no matter what the evidence; desiring to spend time with a known liar without any safeguards; desiring to place oneself in a state-through alcohol, drugs, or whatever-that will have continuing effects on the rationality of one's beliefs. But this requirement is too strong as stated; perhaps acting upon the desire will bring her something she (legitimately) values more than avoiding some particular irrational beliefs or probabilities. Similarly, the dutch book requirement is too strong as usually stated, for perhaps some situation holds in the world so that having incoherent probabilities will bring a far greater benefit-someone will bestow a large prize upon you for those incoherent probabilities-than the loss to be incurred in the bets. The dutch book argument points out that loss can be guaranteed, but still it might be counterbalanced; so too the irrational beliefs or probabilities you have through violating condition XIV might be counterbalanced. To avoid this, the moral of the dutch book argument must not be put too strongly, and similarly for condition XIV.

These fourteen conditions can take us some considerable distance past Hume toward substantive constraints upon preferences and desires. Empirical information about the actual preconditions of satisfying the conditions of rationality, and of making preferential choices- mandated by conditions III, V, and VI-might require quite specific substantive content to one's preferences and desires, the more so when combined with the constraints of the other conditions.

Can we proceed further to specific content? One intriguing route is to attempt to parallel with desire what we want to say about the rationality of belief. For example, people have held that a belief is rational if it is formed by a reliable process whose operation yields a high percentage of true beliefs. To be sure, the details are more complicated, but we might hope to parallel these complications also. A rational desire, then, would be one formed by a process that reliably yields a high percentage of dominant desires. But how are we to fill in that blank? What, for desires, corresponds to truth in the case of beliefs? For now, I have no independent substantive criterion to propose.

We can, however, use our previous conditions, and any additional similar ones, to specify the goal of that process: a desire or preference is rational only if it was formed by a process that reliably yields desires and preferences that satisfy the previous conditions on how preferences are to be structured, namely, conditions I–XIV. This says more than just that these fourteen conditions are to be satisfied, for any process (we can follow) that reliably yields the satisfaction of these conditions may also further constrain a person's desires and preferences.

XV. A particular preference or desire is rational only if there is a process P for arriving at desires and preferences, and (a) that preference or desire was arrived at through that process P, and (b) that process P reliably yields desires and preferences that satisfy the above normative structural conditions I–XIV, and (c) there is no narrower process P′ such that the desire or preference was arrived at through P′, and P′ tends to produce desires and preferences that fail to satisfy conditions I–XIV.

If we say that preferences and desires are rationally coherent when they satisfy conditions I–XIV (and similar conditions), then condition XV says that a preference or desire is rational only if (it is rationally coherent and) it is arrived at by a process that yields rationally coherent preferences and desires.

Not only can that process P reliably yield rationally coherent preferences and desires, it can aim at such preferences and desires, it can shape and guide preferences and desires into rational coherence. The process P can be a homeostatic mechanism, one of whose goal-states is that preferences and desires be rationally coherent. In that case, a function of preferences and desires is to be rationally coherent. (Similarly, if the belief-forming mechanism B aims at beliefs being approximately true, then one function of beliefs is to be approximately true.)

We therefore might add the following condition.

XVI. The process P that yields preferences and desires aims at their being rationally coherent; it is a homeostatic mechanism, one of whose goal-states is that preferences and desires be rationally coherent.

And similarly,

XVII. The cognitive mechanism B that yields beliefs aims at these beliefs satisfying particular cognitive goals, such as these beliefs being (approximately) true, having explanatory power, and so on. B is a homeostatic mechanism, one of whose goal-states is that the beliefs meet the cognitive goals.

A function of preferences and desires is to be rationally coherent; a function of beliefs is to meet the cognitive goals. That follows from our earlier account of function, if these mechanisms P and B are indeed such homeostatic mechanisms. Suppose these homeostatic mechanisms do produce beliefs and desires with these functions. Is it their function to do so? That depends upon what other mechanisms and processes produce and maintain those desireand belief-forming mechanisms. If those preference and cognitive mechanisms P and B were themselves designed, produced, or altered and maintained by homeostatic devices whose goals included aiming P and B at being devices that produced rationally coherent preferences and approximately true beliefs, then we have a double functionality. It is a function of the preferences and beliefs to be rationally coherent and approximately true, and it also is a function of the mechanisms that produce such beliefs and preferences to produce things like that, with those functions.

XVIII. There is a homeostatic mechanism M1 whose goal-state is that the preference mechanism P yield rationally coherent preferences, and P is produced or maintained by M1 (through M1's pursuit of this goalstate).

XIX. There is a homeostatic mechanism M2, whose goal-state is that the belief mechanism B yield beliefs that fulfill cognitive goals, and B is produced or maintained by M2 (through M2's pursuit of this goal-state).

It is plausible to think that our desireand belief-forming mechanisms have undergone evolutionary and social shaping that in some signifi-cant part aimed at their having these functions. There is more. Once people become self-conscious about their preferences and beliefs, they can guide them, monitor them for deviations from rational coherence and truth, and make appropriate corrections. Conscious awareness becomes a part of the processes P and B, and consciously aims them at the goals of rational coherence and truth.

XX. One component of the homeostatic preferenceand desire-forming process P is the person's consciously aiming at rationally coherent preferences and desires.

XXI. One component of the homeostatic belief-forming process B is the person's consciously aiming at beliefs that fulfill cognitive goals.

This self-awareness and monitoring gives us a fuller rationality. (Some might suggest that only when these conditions are satisfied do we have any rationality at all.)

Self-conscious awareness can monitor not just preferences and beliefs but also the processes by which these are formed, P and B themselves. It can alter and improve these processes; it can reshape them. Conscious awareness thus becomes a part of the mechanisms M1 and M2 and so comes to play a role in determining the functions of the preferenceand belief-forming mechanisms themselves.

XXII. One component of the homeostatic mechanism M1 that maintains P is the person's consciously aiming at P's yielding rationally coherent preferences.

XXIII. One component of the homeostatic mechanism M2 that maintains B is the person's consciously aiming at B's yielding beliefs that satisfy cognitive goals.

(Edit: Gwern points out in the comments that there is previous discussion on this study at New study on choice blindness in moral positions.)

Earlier this month, a group of Swedish scientists published a study that describes a new type of bias that I haven't seen listed in any of the sequences or on the wiki. Their methodology:

We created a self-transforming paper survey of moral opinions, covering both foundational principles, and current dilemmas hotly debated in the media. This survey used a magic trick to expose participants to a reversal of their previously stated attitudes, allowing us to record whether they were prepared to endorse and argue for the opposite view of what they had stated only moments ago.

In other words, people were surveyed on their beliefs and were immediately asked to defend them after finishing the survey. Despite having just written down how they felt, 69% did not even notice that at least one of their answers were surreptitiously changed. Amazingly, a majority of people actually "argued unequivocally for the opposite of their original attitude".

Perhaps this type of effect is already discussed here on LessWrong, but, if so, I have not yet run across any such discussion. (It is not on the LessWrong wiki nor the other wiki, for example.) This appears to be some kind of confabulation bias, where invented positions thrust upon people result in confabulated reasons for believing them.

Some people might object to my calling this a bias. (After all, the experimenters themselves did not use that word.) But I'm trying to refer less to the trick involved in the experiment and more toward the bias this experiment shows that we have toward our own views. This is a fine distinction to make, but I feel it is important for us to recognize.

When I say we prefer our own opinions, this is obvious on its face. Of course we think our own positions are correct; they're the result of our previously reasoned thought. We have reason to believe they are correct. But this study shows that our preference for our own views goes even further than this. We actually are biased toward our own positions to such a degree that we will actually verbally defend them even when we were tricked into thinking we held those positions. This is what I mean when I call it confabulation bias.

Of particular interest to the LessWrong community is the fact that this bias apparently is more susceptible to those of us that are more capable of good argumentation. This puts confabulation bias in the same category as the sophistication effect in that well informed people should take special care to not fall for it. (The idea that confabulation bias is more likely to occur with those of us that argue better is not shown in this study, but it seems like a reasonable hypothesis to make.)

As a final minor point, I just want to point out that the effect did not disappear when the changed opinion was extreme. The options available to participants involved agreeing or disagreeing on a 1-9 scale; a full 31% of respondents who chose an extreme position (like 1 or 9) did not even notice when they were shown to have said the opposite extreme.