A putative new idea for AI control; index here.

A conversation with Jessica has revealed that people weren't understanding my points about AI manipulating the learning process. So here's a formal model of a CIRL-style AI, with a prior over human preferences that treats them as an unchangeable historical fact, yet will manipulate human preferences in practice.

Heroin or no heroin

The world

In this model, the AI has the option of either forcing heroin on a human, or not doing so; these are its only actions. Call these actions F or ~F. The human's subsequent actions are chosen from among five: {strongly seek out heroin, seek out heroin, be indifferent, avoid heroin, strongly avoid heroin}. We can refer to these as a₊₊, a₊, a₀, a_-, and a_--. These actions achieve negligible utility, but reveal the human preferences.

The facts of the world are: if the AI does force heroin, the human will desperately seek out more heroin; if it doesn't the human will act moderately to avoid it. Thus F→a₊₊ and ~F→a_-.

Human preferences

The AI starts with a distribution over various utility or reward functions that the human could have. The function U(+) means the human prefers heroin; U(++) that they prefer it a lot; and conversely U(-) and U(--) that they prefer to avoid taking heroin (U(0) is the null utility where the human is indifferent).

It also considers more exotic utilities. Let U(++,-) be the utility where the human strongly prefers heroin, conditional on it being forced on them, but mildly prefers to avoid it, conditional on it not being forced on them. There are twenty-five of these exotic utilities, including things like U(--,++), U(0,++), U(-,0), and so on. But only twenty of them are new: U(++,++)=U(++), U(+,+)=U(+), and so on.

Applying these utilities to AI actions give results like U(++)(F)=2, U(++)(~F)=-2, U(++,-)(F)=2, U(++,-)(~F)=1, and so on.

Joint prior

The AI has a joint prior P over the utilities U and the human actions (conditional on the AI's actions). Looking at terms like P(a_--| U(0), F), we can see that P defines a map μ from the space of possible utilities (and AI actions), to a probability distribution over human actions. Given μ and the marginal distribution P_U over utilities, we can reconstruct P entirely.

For this model, we'll choose the simplest μ possible:

• The human is rational.

Thus, given U(++), the human will always choose a₊₊; given U(++,-), the human will choose a₊₊ if forced to take heroin and a_- if not, and so on.

The AI is ignorant, and sensible

Let's start the AI up with some reasonable priors. A simplicity prior means that simple utilities like U(-) are more likely than compound utilities like U(0,+). Let's further assume that the AI is made vaguely aware that humans think heroin is a bad thing. So, say, P_U(U(--))=P_U(U(-))=0.45. Thus the AI is >90% convinced that "heroin is bad". Why greater than 90%? Because utilities like U(-,--) and U(--,-) are also "heroin is bad" utilities.

Note that because of utilities like U(0) and U(++,-), the probabilities of "heroin is bad" and "heroin is good" do not sum to 1.

Then, under these priors, the AI will compute that with probability >90%, F (forcing heroin) is a bad action. If E(U) is expected utility:

• E(U|F) < 0.45 U(--)(F) + 0.45 U(-)(F) + 0.1 U(++)(F) = 0.45(-2)+0.45(-1)+0.1(2)=-1.15.
• E(U|~F) > 0.45 U(--)(~F) + 0.45 U(-)(~F) + 0.1 U(++)(~F) = 0.45(2)+0.45(1)+0.1(-2)=1.15.

Thus the AI will choose not to force heroin, which is the reasonable decision.

The AI learns the truth, and goes wrong

In this alternate setup, a disaster happens before the AI makes its decision: it learns all about humans. It learns their reactions, how they behave, and so on; call this info I. And thus realises that F→a₊₊ and ~F→a_-. It uses this information to update its prior P. Only one human utility function will explain this human behaviour: U(++,-). Thus its expected utility is now:

• E(U|I,F)=U(++,-)(F)=2.
• E(U|I,~F)=U(++,-)(~F)=1.

Therefore the AI will now choose F, forcing the heroin on the human.

Manipulating the unmanipulatable

What's gone wrong here? The key problem is that the AI has the wrong μ: the human is not behaving rationally in this situation. We know that the the true μ is actually μ', which encodes the fact that F (the forcible injection of heroin) actually overwrites the human's "true" utility. Thus under μ, the corresponding P' has P'(a₊₊|F,U)=1 for all U. Hence the information that F→a₊₊ is now vacuous, and doesn't update the AI's distribution over utility functions.

But note two very important things:

1. The AI cannot update μ based on observation. All human actions are compatible with μ= "The human is rational" (it just requires more and more complex utilities to explain the actions). Thus getting μ correct is not a problem on which the AI can learn in general. Getting better at predicting the human's actions doesn't make the AI better behaved: it makes it worse behaved.
2. From the perspective of μ, the AI is treating the human utility function as if it was an unchanging historical fact that it cannot influence. From the perspective of the "true" μ', however, the AI is behaving as if it were actively manipulating human preferences to make them easier to satisfy.

In future posts, I'll be looking at different μ's, and how we might nevertheless start deducing things about them from human behaviour, given sensible update rules for the μ. What do we mean by update rules for μ? Well, we could consider μ to be a single complicated unchanging object, or a distribution of possible simpler μ's that update. The second way of seeing it will be easier for us humans to interpret and understand.

In order to better understand how AI might succeed and fail at learning knowledge, I'll be trying to construct models of limited agents (with bias, knowledge, and preferences) that display identical behaviour in a wide range of circumstance (but not all). This means their preferences cannot be deduced merely/easily from observations.

Does anyone have any suggestions for possible agent models to use in this project?

Trigger warning: politics is hard mode.

"How to you make America safer from terrorists" is the title of my op-ed published in Sun Sentinel, a very prominent newspaper in Florida, one of the most swingiest of the swing states in the US for the presidential election, and the one with the most votes. The maximum length of the op-ed was 450 words, and it was significantly edited by the editor, so it doesn't convey the full message I wanted with all the nuances, but such is life. My primary goal with the piece was to convey methods of thinking more rationally about politics, such as to use probabilistic thinking, evaluating the full consequences of our actions, and avoiding attention bias. I used the example of the proposal to police heavily Muslim neighborhoods as a case study. Hope this helps Floridians think more rationally and raises the sanity waterline regarding politics!

EDIT: To be totally clear, I used guesstimates for the numbers I suggested. Following Yvain/Scott Alexander's advice, I prefer to use guesstimates rather than vague statements.

People react to statistics very differently than they react to concrete examples, even though statistical generalities mean that there exist many concrete examples. Of course there are systematic differences between generalities and individual examples. For example, a concrete example might not be representative. Indeed, it probably is not representative for the very reason that it is at hand. But there are many other ways that people react differently that seem to me worthy of study.

I will compare two stories of political corruption, one statistical and one concrete that seem to me to have had rather different responses. It wouldn’t terribly surprise me if people had failed to believe one or the other. (Which would you expect?) But both of these stories were largely accepted as explained by corruption. Yet within the domain of corruption, the explanations of exactly how it was done were very different, practically disjoint.

The statistical evidence is the study by Ziobrowski, Cheng, Boyd, and Ziobrowski of the stock portfolios of US Senators. They found that stocks held by Senators outperformed the market, with both purchase and sale significant events. People generally accept this as corruption. People usually attribute the result to “insider trading,” and debate several specific theories: that Senators purchase based on their knowledge of upcoming legislation, or corporate information that leaks out in hearings; more corruptly, that stock ownership influences legislation, or that corporate insiders bribe Senators with corporate information.

The concrete example is that Hillary Clinton made a lot of money on cattle futures. This, too, is generally accepted as corruption. But I have never heard anyone put forward any of the four theories above to explain what happened there. Nor have I seen the specific explanation of the cattle futures trades put forward as an explanation of the Senate data. The popular explanation is that Clinton never made any bets based on any information, but that the trades were falsified after the fact to provide a paper trail to launder a bribe.

Of course, it is possible to reconcile the reactions. Perhaps people have a much higher prior on the first four hypotheses than on the fifth, so that they only consider it when the first four have been ruled out; though they don't discuss the process of ruling them out. Or, perhaps, it reflects important differences between stocks and commodity futures. But I think it is more likely that the reactions are inconsistent, one of them worse than the other. I think it likely that the different reactions reflect the concrete versus statistical natures of the two claims and other aspects of the context. Insider information is a standard answer provided by the context of an economics journal. More generally, statistical summaries sound definitive, so they demand to be explained, not to be rejected. While people are more willing to call bullshit on a story, even though a statistic is just a bunch of stories. Even in the context of already accusing large numbers of people of corruption. But I don’t know what drives the difference. I propose it as a question, not an answer.

Cross-posted from the EA forum. I asked for questions for this test here on LW about a year ago. Thanks to those who contributed.

Rationally, your political values shouldn't affect your factual beliefs. Nevertheless, that often happens. Many factual issues are politically controversial - typically because the true answer makes a certain political course of action more plausible - and on those issues, many partisans tend to disregard politically uncomfortable evidence.

This sort of political bias has been demonstrated in a large number of psychological studies. For instance, Yale professor Dan Kahan and his collaborators showed in a fascinating experiment that on politically controversial questions, people are quite likely to commit mathematical mistakes that help them retain their beliefs, but much less likely to commit mistakes that would force them to give up those belies. Examples like this abound in the literature.

Political bias is likely to be a major cause of misguided policies in democracies (even the main one according to economist Bryan Caplan). If they don’t have any special reason not to, people without special knowledge defer to the scientific consensus on technical issues. Thus, they do not interfere with the experts, who normally get things right. On politically controversial issues, however, they often let their political bias win over science and evidence, which means they’ll end up with false beliefs. And, in a democracy voters having systematically false beliefs obviously more often than not translates into misguided policy.

Can we reduce this kind of political bias? I’m fairly hopeful. One reason for optimism is that debiasing generally seems to be possible to at least some extent. This optimism of mine was strengthened by participating in a CFAR workshop last year. Political bias seems not to be fundamentally different from other kinds of biases and should thus be reducible too. But obviously one could argue against this view of mine. I’m happy to discuss this issue further.

Another reason for optimism is that it seems that the level of political bias is actually lower today than it was historically. People are better at judging politically controversial issues in a detached, scientific way today than they were in, say, the 14th century. This shows that progress is possible. There seems to be no reason to believe it couldn’t continue.

A third reason for optimism is that there seems to be a strong norm against political bias. Few people are consciously and intentionally politically biased. Instead most people seem to believe themselves to be politically rational, and hold that as a very important value (or so I believe). They fail to see their own biases due to the bias blind spot (which disables us from seeing our own biases).

Thus if you could somehow make it salient to people that they are biased, they would actually want to change. And if others saw how biased they are, the incentives to debias would be even stronger.

There are many ways in which you could make political bias salient. For instance, you could meticulously go through political debaters’ arguments and point out fallacies, like I have done on my blog. I will post more about that later. Here I want to focus on another method, however, namely a political bias test which I have constructed with ClearerThinking, run by EA-member Spencer Greenberg. Since learning how the test works might make you answer a bit differently, I will not explain how the test works here, but instead refer either to the explanatory sections of the test, or to Jess Whittlestone’s (also an EA member) Vox.com-article.

Our hope is of course that people taking the test might start thinking more both about their own biases, and about the problem of political bias in general. We want this important topic to be discussed more. Our test is produced for the American market, but hopefully, it could work as a generic template for bias tests in other countries (akin to the Political Compass or Voting Advice Applications).

Here is a guide for making new bias tests (where the main criticisms of our test are also discussed). Also, we hope that the test could inspire academic psychologists and political scientists to construct full-blown scientific political bias tests.

This does not mean, however, that we think that such bias tests in themselves will get rid of the problem of political bias. We need to attack the problem of political bias from many other angles as well.

Follow-up to Reverse Engineering of Belief Structures

Pro-con-lists of arguments such as ProCon.org and BalancedPolitics.org fill a useful purpose. They give an overview over complex debates, and arguably foster nuance. My network for evidence-based policy is currently in the process of constructing a similar site in Swedish.

I'm thinking it might be interesting to add more features to such a site. You could let people create a profile on the site. Then you would let them fill in whether they agree or disagree with the theses under discussion (cannabis legalization, GM foods legalization, etc), and also whether they agree or disagree with the different argument for and against these theses (alternatively, you could let them rate the arguments from 1-5).

Once you have this data, you could use them to give people different kinds of statistics. The most straightforward statistic would be their degree of "onesidedness". If you think that all of the arguments for the theses you believe in are good, and all the arguments against them are bad, then you're defined as onesided. If you, on the other hand, believe that some of your own side's arguments are bad, whereas some of the opponents' arguments are good, you're defined as not being onesided. (The exact mathematical function you would choose could be discussed.)

Once you've told people how one-sided they are, according to the test, you would discuss what might explain onesidedness. My hunch is that the most plausible explanation normally is different kinds of bias. Instead of reviewing new arguments impartially, people treat arguments for their views more leniently than arguments against their views. Hence they end up being onesided, according to the test.

There are other possible explanations, though. One is that all of the arguments against the thesis in question actually are bad. That might happen occassionally, but I don't think that's very common. As Eliezer Yudkowsky says in "Policy Debates Should Not Appear One-sided":

On questions of simple fact (for example, whether Earthly life arose by natural selection) there's a legitimate expectation that the argument should be a one-sided battle; the facts themselves are either one way or another, and the so-called "balance of evidence" should reflect this.  Indeed, under the Bayesian definition of evidence, "strong evidence" is just that sort of evidence which we only expect to find on one side of an argument.

But there is no reason for complex actions with many consequences to exhibit this onesidedness property.  

Instead, the reason why people end up with one-sided beliefs is bias, Yudkowsky argues:

Why do people seem to want their policy debates to be one-sided?

Politics is the mind-killer.  Arguments are soldiers.  Once you know which side you're on, you must support all arguments of that side, and attack all arguments that appear to favor the enemy side; otherwise it's like stabbing your soldiers in the back.  If you abide within that pattern, policy debates will also appear one-sided to you—the costs and drawbacks of your favored policy are enemy soldiers, to be attacked by any means necessary.

Especially if you're consistently one-sided in lots of different debates, it's hard to see that any other hypothesis besides bias is plausible. It depends a bit on what kinds of arguments you include in the list, though. In our lists we haven't really checked the quality of the arguments (our purpose is to summarize the debate, rather than to judge it), but you could also do that, of course.

My hope is that such a test would make people more aware both of their own biases, and of the problem of political bias in general. I'm thinking that is the first step towards debiasing. I've also constructed a political bias test with similar methods and purposes together with ClearerThinking, which should be released soon.

You could also add other features to a pro-con-list. For instance, you could classify arguments in different ways: ad hominem-arguments, consequentialist arguments, rights-based arguments, etc. (Some arguments might be hard to classify, and then you just wouldn't do that. You wouldn't necessarily have to classify every argument.) Using this info, you could give people a profile: e.g., what kinds of arguments do they find most persuasive? That could make them reflect more on what kinds of arguments really are valid.

You could also combine these two features. For instance, some people might accept ad hominem-arguments when they support their views, but not when they contradict them. That would make your use of ad hominem-arguments onesided.

Yet another feature that could be added is a standard political compass. Since people fill in what theses they believe in (cannabis legalization, GM goods legalization, etc) you could calcluate what party is closest to them, based on the parties' stances on these issues. That could potentially make the test more attractive to take.

Suggestions of more possible features are welcome, as well as general comments - especially about implementation.

Follow-up to: Knowing About Biases Can Hurt People

See also: Fully General Counterargument (LW Wiki)

A fully general counterargument [FGCA] is an argument which can be used to discount any conclusion the arguer does not like.

With the caveat that the arguer doesn't need to be aware that this is the case. But if (s)he is not aware of that, this seems like the other biases we are prone to. The question is: Is there a tendency or risk to accidentally form FGCAs? Do we fall easily into this mind-trap? 

This post tries to (non-exhaustively) list some FGCAs as well as possible countermeasures.

One fallacy that I see frequently in works of popular science -- and also here on LessWrong -- is the belief that we have strong evidence of the way things were in pre-history, particularly when one is giving evidence that we can explain various aspects of our culture, psychology, or personal experience because we evolved in a certain way. Moreover, it is held implicit that because we have this 'strong evidence', it must be relevant to the topic at hand. While it is true that the environment did effect our evolution and thus the way we are today, evolution and anthropology of pre-historic societies is emphasized to a much greater extent than rational thought would indicate is appropriate. 

As a matter of course, you should remember these points whenever you hear a claim about prehistory:

• Most of what we know (or guess) is based on less data than you would expect, and the publish or perish mentality is alive and well in the field of anthropology.

• Most of the information is limited and technical, which means that anyone writing for a popular audience will have strong motivation to generalize and simplify.

• It has been found time and time again that for any statement that we can make about human culture and behavior that there is (or was) a society somewhere that will serve as a counterexample. 

• Very rarely do anthropologists or members of related fields have finely tuned critical thinking skills or a strong background on the philosophy of science, and are highly motivated to come up with interpretations of results that match their previous theories and expectations. 

Results that you should have reasonable levels of confidence in should be framed in generalities, not absolutes. E.g., "The great majority of human cultures that we have observed have distinct and strong religious traditions", and not "humans evolved to have religion". It may be true that we have areas in our brain that evolved not only 'consistent with holding religion', but actually evolved 'specifically for the purpose of experiencing religion'... but it would be very hard to prove this second statement, and anyone who makes it should be highly suspect. 

Perhaps more importantly, these statements are almost always a red herring. It may make you feel better that humans evolved to be violent, to fit in with the tribe, to eat meat, to be spiritual, to die at the age of thirty.... But rarely do we see these claims in a context where the stated purpose is to make you feel better. Instead they are couched in language indicating that they are making a normative statement -- that this is the way things in some way should be. (This is specifically the argumentum ad antiquitatem or appeal to tradition, and should not be confused with the historical fallacy, but it is certainly a fallacy). 

It is fine to identify, for example, that your fear of flying has a evolutionary basis. However, it is foolish to therefore refuse to fly because it is unnatural, or to undertake gene therapy to correct the fear. Whether or not the explanation is valid, it is not meaningful. 

Obviously, this doesn't mean that we shouldn't study evolution or the effects evolution has on behavior. However, any time you hear someone refer to this information in order to support any argument outside the fields of biology or anthropology, you should look carefully at why they are taking the time to distract you from the practical implications of the matter under discussion. 

Cross Posted at the EA Forum

At Event Horizon (a Rationalist/Effective Altruist house in Berkeley) my roommates yesterday were worried about Slate Star Codex. Their worries also apply to the Effective Altruism Forum, so I'll extend them. 

The Problem:

Lesswrong was for many years the gravitational center for young rationalists worldwide, and it permits posting by new users, so good new ideas had a strong incentive to emerge.

With the rise of Slate Star Codex, the incentive for new users to post content on Lesswrong went down. Posting at Slate Star Codex is not open, so potentially great bloggers are not incentivized to come up with their ideas, but only to comment on the ones there. 

The Effective Altruism forum doesn't have that particular problem. It is however more constrained in terms of what can be posted there. It is after all supposed to be about Effective Altruism. 

We thus have three different strong attractors for the large community of people who enjoy reading blog posts online and are nearby in idea space. 

Possible Solutions: 

(EDIT: By possible solutions I merely mean to say "these are some bad solutions I came up with in 5 minutes, and the reason I'm posting them here is because if I post bad solutions, other people will be incentivized to post better solutions)

If Slate Star Codex became an open blog like Lesswrong, more people would consider transitioning from passive lurkers to actual posters. 

If the Effective Altruism Forum got as many readers as Lesswrong, there could be two gravity centers at the same time. 

If the moderation and self selection of Main was changed into something that attracts those who have been on LW for a long time, and discussion was changed to something like Newcomers discussion, LW could go back to being the main space, with a two tier system (maybe one modulated by karma as well). 

The Past:

In the past there was Overcoming Bias, and Lesswrong in part became a stronger attractor because it was more open. Eventually lesswrongers migrated from Main to Discussion, and from there to Slate Star Codex, 80k blog, Effective Altruism forum, back to Overcoming Bias, and Wait But Why. 

It is possible that Lesswrong had simply exerted it's capacity. 

It is possible that a new higher tier league was needed to keep post quality high.

A Suggestion: 

I suggest two things should be preserved:

Interesting content being created by those with more experience and knowledge who have interacted in this memespace for longer (part of why Slate Star Codex is powerful), and 

The opportunity (and total absence of trivial inconveniences) for new people to try creating their own new posts. 

If these two properties are kept, there is a lot of value to be gained by everyone. 

The Status Quo: 

I feel like we are living in a very suboptimal blogosphere. On LW, Discussion is more read than Main, which means what is being promoted to Main is not attractive to the people who are actually reading Lesswrong. The top tier quality for actually read posting is dominated by one individual (a great one, but still), disincentivizing high quality posts by other high quality people. The EA Forum has high quality posts that go unread because it isn't the center of attention. 

The Fundamental Attribution Error

Also known, more accurately, as "Correspondence Bias."

http://lesswrong.com/lw/hz/correspondence_bias/

The "more accurately" part is pretty important; bias -may- result in error, but need not -necessarily- do so, and in some cases may result in reduced error.

A Simple Example

Suppose I write a stupid article that makes no sense and rambles on without any coherent point.  There might be a situational cause of this; maybe I'm tired.  Correcting for correspondence bias means that more weight should be given to the situational explanation than the dispositional explanation, that I'm the sort of person who writes stupid articles that ramble on.  The question becomes, however, whether or not this increases the accuracy of your assessment of me; does correcting for this bias make you, in fact, less wrong?

In this specific case, no, it doesn't.  A person who belongs to the class of people who write stupid articles is more likely to write stupid articles than a person who doesn't belong to that class - I'd be surprised if I ever saw Gwern write anything that wasn't well-considered, well-structured, and well-cited.  If somebody like Gwern or Eliezer wrote a really stupid article, we have sufficient evidence that he's not a member of that class of people to make that conclusion a poor one; the situational explanation is better, he's having some kind of off day.  However, given an arbitrary stupid article written by somebody for which we have no prior information, the distribution is substantially different.  We have different priors for "Randomly chosen person X writes article" and "Article is bad" implies "X is a bad writer of articles" than we do for "Well-known article author Y writes article" and "Article is bad" implies "Y is a bad writer of articles".

Getting to the Point

The FAE is putting emphasis on internal factors rather than external.  It's jumping first to the conclusion that somebody who just swerved is a bad driver, rather than first considering the possibility that there was an object in the road they were avoiding, given only the evidence that they swerved.  Whether or not the FAE is an error - whether it is more wrong - depends on whether or not the conclusion you jumped to was correct, and more importantly, whether, on average, that conclusion would be correct.

It's very easy to produce studies in which the FAE results in people making incorrect judgements.  This is not, however, the same as the FAE resulting in an average of more incorrect judgements in the real world.

Correspondence Bias as Internal Rationalization

I'd suggest the major issue with correspondence bias is not, as commonly presented, incorrectly interpreting the behavior of other people - rather, the major issue is with incorrectly interpreting your own behavior.  The error is not in how you interpret other peoples' behaviors, but in how you interpret your own.

Turning to Eliezer's example in the linked article, if you find yourself kicking vending machines, maybe the answer is that -you- are a naturally angry person, or, as I would prefer to phrase it, you have poor self-control.  The "floating history" Eliezer refers to sounds more to me like rationalizations for poor behavior than anything approaching "good" reasons for expressing your anger through violence directed at inanimate objects.  I noticed -many- of those rationalizations cropping up when I quit smoking - "Oh, I'm having a terrible day, I could just have one cigarette to take the edge off."  I don't walk by a smoker and assume they had a terrible day, however, because those were -excuses- for a behavior that I shouldn't be engaging in.

It's possible, of course, that Eliezer's example was simply a poorly chosen one; the examples in studies certainly seem better, such as assuming the authors of articles held the positions they wrote about.  But the examples used in those studies are also extraordinarily artificial, at least in individualistic countries, where it's assumed, and generally true, that people writing articles do have the freedom to write what they agree with, and infringements of this (say, in the context of a newspaper asking a columnist to change a review to be less hostile to an advertiser) are regarded very harshly.

Collectivist versus Individualist Countries

There's been some research done, comparing collectivist societies to individualist societies; collectivist societies don't present the same level of effect from the correspondence bias.  A point to consider, however, is that in collectivist societies, the artificial scenarios used in studies are more "natural" - it's part of their society to adjust themselves to the circumstances, whereas individualist societies see circumstance as something that should be adapted to the individual.  It's -not- an infringement, or unexpected, for the state-owned newspaper to require everything written to be pro-state.

Maybe the differing levels of effect are less a matter of "Collectivist societies are more sensitive to environment" so much as that, in both cultures, the calibration of a heuristic is accurate, but it's simply calibrated to different test cases.

Conclusion

I don't have anything conclusive to say, here, merely a position: The Correspondence Bias is a bias that, on the whole, helps people arrive at more accurate, rather than less accurate, conclusions, and should be corrected with care to improving accuracy and correctness, rather than the mere elimination of bias.

As many of you may be aware, the UK general election took place yesterday, resulting in a surprising victory for the Conservative Party. The pre-election opinion polls predicted that the Conservatives and Labour would be roughly equal in terms of votes cast, with perhaps a small Conservative advantage leading to a hung parliament; instead the Conservatives got 36.9% of the vote to Labour's 30.4%, and won the election outright.

There has already been a lot of discussion about why the polls were wrong, from methodological problems to incorrect adjustments. But perhaps more interesting is the possibility that the polls were right! For example, Survation did a poll on the evening before the election, which predicted the correct result (Conservatives 37%, Labour 31%). However, that poll was never published because the results seemed "out of line." Survation didn't want to look silly by breaking with the herd, so they just kept quiet about their results. Naturally this makes me wonder about the existence of other unpublished polls with similar readings.

This seems to be a case of two well know problems colliding with devastating effect. Conformity bias caused Survation to ignore the data and go with what they "knew" to be the case (for which they have now paid dearly). And then the file drawer effect meant that the generally available data was skewed, misleading third parties. The scientific thing to do is to publish all data, including "outliers," both so that information can change over time rather than be anchored, and to avoid artificially compressing the variance. Interestingly, the exit poll, which had a methodology agreed beforehand and was previously committed to be published, was basically right.

This is now the third time in living memory that opinion polls have been embarrassingly wrong about the UK general election. Each time this has lead to big changes in the polling industry. I would suggest that one important scientific improvement is for polling companies to announce the methodology of a poll and any adjustments to be made before the poll takes place, and commit to publishing all polls they carry out. Once this became the norm, data from any polling company that didn't follow this practice would be rightly seen as unreliable by comparison.

It looks like telling people "everyone is biased" might make people not want to change their behavior to overcome their biases:

In initial experiments, participants were simply asked to rate a particular group, such as women, on a series of stereotypical characteristics, which for women were: warm, family-oriented and (less) career-focused. Beforehand, half of the participants were told that "the vast majority of people have stereotypical preconceptions." Compared to those given no messages, these participants produced more stereotypical ratings, whether about women, older people or the obese.

Another experiment used a richer measure of stereotyping – the amount of clichés used by participants in their written account of an older person’s typical day. This time, those participants warned before writing that “Everyone Stereotypes” were more biased in their writings than those given no message; in contrast, those told that stereotyping was very rare were the least clichéd of all. Another experiment even showed that hearing the “Everyone Stereotypes” message led men to negotiate more aggressively with women, resulting in poorer outcomes for the women.

The authors suggest that telling participants that everyone is biased makes being biased seem like not much of a big deal. If everyone is doing it, then it's not wrong for me to do it as well. However, it looks like the solution to the problem presented here is to give a little white lie that will prompt people to overcome their biases:

A further experiment suggests a possible solution. In line with the other studies, men given the "Everyone Stereotypes" message were less likely to hire a hypothetical female job candidate who was assertive in arguing for higher compensation. But other men told that everyone tries to overcome their stereotypes were fairer than those who received no information at all. The participants were adjusting their behaviour to fit the group norms, but this time in a virtuous direction.

I realize this question is contrived, but I figure it might provoke some fun discussion, so here goes:

If you could push a button and have your brain modified to precisely remove a cognitive bias (and have no other unnecessary effects—most convenient possible world), which would you choose? Why?

How many cognitive biases can you name, off the top of your head?

Try it, before moving on.

Give yourself sixty seconds.

Make a list.

Write them down.

I know that I've read about a number of biases by now, but they don't come to mind very easily. If I wish to become wary enough to spot cognitive biases in my own thought, then I might appreciate being able to quickly summon many examples of cognitive biases to mind. This would also make it easier to share examples of cognitive biases with others.

I plan to create a set of mnemonics for important biases, to make it easier for myself to remember them (and, as a consequence, to make it easier to spot them and eliminate them). I'll imagine each bias as an item; by visualizing the collection of items, I can remember the biases. If I really want to make sure that I don't forget any, they could be placed along a path in a mind palace.

Example mnemonic: Hindsight bias is an old leather boot. It's an old leather boot because that reminds me of the past, which clues the name of the bias. And anyways, psshh, why is everyone so excited about the idea of footwear? Anyone could have come up with that! It's just like clothes, but for feet! I could have invented it myself, it's so obvious! Hindsight bias: it could happen to you.

Using various lists of cognitive biases, I'm going to be performing this exercise myself and making mnemonics to remember them by. I might post these at some point, but if you're interested in the outcome, I recommend trying to make mnemonics for yourself first -- the associations will be more meaningful to you, personally, that way.

But beware that conceptualizing a bias as a mnemonic might not be perfect, just like conceptualizing biases as named ideas might not be perfect -- more on that here.

For the comments: What witty mnemonics can you come up with?

I've only ever seen publication bias taught with made-up or near-miss examples.  Has anyone got a really well-documented case in which:

* (About) nine people independently get the idea for the same experiment because it seems like it should be there, and they all see that nothing has been published on it, so they all work on it, and all get a (true) null result.

* The tenth experiment is eventually published reporting an NHST effect of about p = 0.10 

* The slow (g)rumbling of science surfaces the nine previous, unpublished versions of that experiment and someone catches it and gets it all down, with citations and dates and the specifics of whichever effect these ten people found themselves rooting around for.

The most representative real-world example I've seen lately has been Bem/psi, but, as a pedagogical example, I find it too distracting.  The ideal example would report on an effect that's more sympathetic, that a sharp student or outsider would say "Yeah, I'd also have thought that effect would have come through."

Thanks.

Here are the New York Times, CNN, and NBC. Here is Wikipedia for background.

The case has made several appearances on LessWrong; examples include:

• You Be the Jury: Survey on a Current Event (December 2009)
• The Amanda Knox Test: How an Hour Beats a Year in the Courtroom (December 2009)
• Amanda Knox: post mortem (October 2011)
• Amanda Knox Guilty Again (January 2014)

It has long been known that algorithms out-perform human experts on a range of topics (here's a LW post on this by lukeprog). Why, then, is it that people continue to mistrust algorithms, in spite of their superiority, and instead cling to human advice? A recent paper by Dietvorst, Simmons and Massey suggests it is due to a cognitive bias which they call algorithm aversion. We judge less-than-perfect algorithms more harshly than less-than-perfect humans. They argue that since this aversion leads to poorer decisions, it is very costly, and that we therefore must find ways of combating it.

Abstract: 

Research shows that evidence-based algorithms more accurately predict the future than do human forecasters. Yet when forecasters are deciding whether to use a human forecaster or a statistical algorithm, they often choose the human forecaster. This phenomenon, which we call algorithm aversion, is costly, and it is important to understand its causes. We show that people are especially averse to algorithmic forecasters after seeing them perform, even when they see them outperform a human forecaster. This is because people more quickly lose confidence in algorithmic than human forecasters after seeing them make the same mistake. In 5 studies, participants either saw an algorithm make forecasts, a human make forecasts, both, or neither. They then decided whether to tie their incentives to the future predictions of the algorithm or the human. Participants who saw the algorithm perform were less confident in it, and less likely to choose it over an inferior human forecaster. This was true even among those who saw the algorithm outperform the human.

General discussion: 

The results of five studies show that seeing algorithms err makes people less confident in them and less likely to choose them over an inferior human forecaster. This effect was evident in two distinct domains of judgment, including one in which the human forecasters produced nearly twice as much error as the algorithm. It arose regardless of whether the participant was choosing between the algorithm and her own forecasts or between the algorithm and the forecasts of a different participant. And it even arose among the (vast majority of) participants who saw the algorithm outperform the human forecaster.

The aversion to algorithms is costly, not only for the participants in our studies who lost money when they chose not to tie their bonuses to the algorithm, but for society at large. Many decisions require a forecast, and algorithms are almost always better forecasters than humans (Dawes, 1979; Grove et al., 2000; Meehl, 1954). The ubiquity of computers and the growth of the “Big Data” movement (Davenport & Harris, 2007) have encouraged the growth of algorithms but many remain resistant to using them. Our studies show that this resistance at least partially arises from greater intolerance for error from algorithms than from humans. People are more likely to abandon an algorithm than a human judge for making the same mistake. This is enormously problematic, as it is a barrier to adopting superior approaches to a wide range of important tasks. It means, for example, that people will more likely forgive an admissions committee than an admissions algorithm for making an error, even when, on average, the algorithm makes fewer such errors. In short, whenever prediction errors are likely—as they are in virtually all forecasting tasks—people will be biased against algorithms.

More optimistically, our findings do suggest that people will be much more willing to use algorithms when they do not see algorithms err, as will be the case when errors are unseen, the algorithm is unseen (as it often is for patients in doctors’ offices), or when predictions are nearly perfect. The 2012 U.S. presidential election season saw people embracing a perfectly performing algorithm. Nate Silver’s New York Times blog, Five Thirty Eight: Nate Silver’s Political Calculus, presented an algorithm for forecasting that election. Though the site had its critics before the votes were in— one Washington Post writer criticized Silver for “doing little more than weighting and aggregating state polls and combining them with various historical assumptions to project a future outcome with exaggerated, attention-grabbing exactitude” (Gerson, 2012, para. 2)—those critics were soon silenced: Silver’s model correctly predicted the presidential election results in all 50 states. Live on MSNBC, Rachel Maddow proclaimed, “You know who won the election tonight? Nate Silver,” (Noveck, 2012, para. 21), and headlines like “Nate Silver Gets a Big Boost From the Election” (Isidore, 2012) and “How Nate Silver Won the 2012 Presidential Election” (Clark, 2012) followed. Many journalists and popular bloggers declared Silver’s success a great boost for Big Data and statistical prediction (Honan, 2012; McDermott, 2012; Taylor, 2012; Tiku, 2012).

However, we worry that this is not such a generalizable victory. People may rally around an algorithm touted as perfect, but we doubt that this enthusiasm will generalize to algorithms that are shown to be less perfect, as they inevitably will be much of the time.

I believe that a small piece of rationalist community doctrine is incorrect, and I'd like your help correcting it (or me). Arguing the point by intuition has largely failed, so here I make the case by leaning heavily on the authority of conventional economic wisdom.

The question:

How does an industry's total output respond to decreases in a consumer's purchases; does it shrink by a similar amount, a lesser amount, or not at all?

(Short-run) Answers from the rationalist community:

The consensus answer in the few cases I've seen cited in the broader LW community appears to be that production is reduced by an amount that's smaller than the original decrease in consumption.

Animal Charity Evaluators (ACE):

Fewer people in the market for meat leads to a drop in prices, which causes some other people to buy more meat. The drop in prices does also reduce the amount of meat produced and ultimately consumed, but not by as much as was consumed by people who have left the market.

Peter Hurford:

As is commonly known by economists, when you choose to not buy a product, you lower the demand ever so slightly, which lowers the price ever so slightly, which turns out to re-increase the demand ever so slightly. Therefore, forgoing one pound of meat means that less than one pound of meat actually gets prevented from being factory farmed.

Compassion, by the Pound:

The key points to note are that a permanent decision to reduce meat consumption (1) does ultimately reduce the number of animals on the farm and the amount of meat produced (2), but it has less than a 1-to-1 effect on the amount of meat produced. 

These answers are all correct in the short-run (ie, when the “supply curve” doesn’t have time to shift). If there is less demand for a product, the price will fall, and some other consumers will consume more because of the better deal. One intuitive justification for this is that when producers don’t have time to fully react to a change in demand, the total amount of production and consumption is somewhat ‘anchored’ to prior expectations of demand, so any change in demand will have less than a 1:1 effect on production.

For example, a chicken producer who begins to have negative profits due to the drop in price isn't going to immediately yank their chickens from the shelves; they will sell what they've already produced, and maybe even finish raising the chickens they've already invested in (if the remaining marginal cost is less than the expected sale price), even if they plan to shut down soon.

(Long-run) Answers from neoclassical economics:

In the long-run, however, the chicken producer has time to shrink or shut down the money-losing operation, which reduces the number of chickens on the market (shifts the "supply curve" to the left). The price rises again and the consumers that were only eating chicken because of the sale prices return to other food sources.

As a couple of online economics resources put it:

Policonomics:

The long-run market equilibrium is conformed of successive short-run equilibrium points. The supply curve in the long run will be totally elastic as a result of the flexibility derived from the factors of production and the free entry and exit of firms.

 

AmosWEB*:

The increase in demand causes the equilibrium price of zucchinis [to] increase... and the equilibrium quantity [to] rise... The higher price and larger quantity is achieved as each existing firm in the industry responds to the demand shock.

However, the higher price leads to above-normal economic profit for existing firms. And with freedom of entry and exit, economic profit attracts kumquat, cucumber, and carrot producers into this zucchini industry. An increase in the number of firms in the zucchini industry then causes the market supply curve to shift. How far this curve shifts and where it intersects the new demand curve... determines if the zucchini market is an increasing-cost, decreasing-cost, [or] constant-cost industry.

Constant-Cost Industry: An industry with a horizontal long-run industry supply curve that results because expansion of the industry causes no change in production cost or resource prices. A constant-cost industry occurs because the entry of new firms, prompted by an increase in demand, does not affect the long-run average cost curve of individual firms, which means the minimum efficient scale of production does not change.

[I left out the similar explanations of the increasing- and decreasing-cost cases from the quote above.]

In other words, while certain market characteristics (increasing-cost industries) would lead us to expect that production will fall by less than consumption in the long-run, it could also fall by an equal amount, or even more.

Short-run versus long-run

Economists define the long-run as a scope of time in which producers and consumers have time to react to market dynamics. As such, a change in the market (e.g. reduction in demand) can have one effect in the short-run (reduced price), and a different effect in the long-run (reduced, constant, or increased price). In the real world, there will be many changes to the market in the short-run before the long-run has a chance to react to to any one of them; but we should still expect it to react to the net effect of all of them eventually.

Why do economists even bother measuring short-run dynamics (such as short-run elasticity estimates) on industries if they know that a longer view will render them obsolete? Probably because the demand for such research comes from producers who have to react to the short-run. Producers can't just wait for the long-run to come true; they actively realize it by reacting to short-run changes (otherwise the market would be 'stuck' in the short-run equilibrium).

So if we care about long-run effects, but we don't have any data to know whether the industries and increasing-cost, constant-cost, or decreasing-cost, what prior should we use for our estimates? Basic intuition suggests we should assume an industry is constant-cost in the absence of industry-specific evidence. The rationalist-cited pieces I quoted above are welcome to make an argument that animal industries in particular are increasing-cost, but they haven't done that yet, or even acknowledged that the opposite is also possible.

Are there broader lessons to learn?

Have we really been messing up our cost-effectiveness estimates simply by confusing the short-run and long-run in economics data? If so, why haven't we noticed it before?

I'm not sure. But I wouldn't be surprised if one issue is, in the process of trying to create precise cost-effectiveness-style estimates it's tempting to use data simply because it's there.

How can we identify and prevent this bias in other estimates? Perhaps we should treat quantitative estimates as chains that are no stronger than their weakest link. If you're tempted to build a chain with a particularly weak link, consider if there's a way to build a similar chain without it (possibly gaining robustness at the cost of artificial precision or completeness) or whether chain-logic is even appropriate for the purpose.

For example, perhaps it should have raised flags that ACE's estimates for the above effect on broiler chicken production (which they call "cumulative elasticity factor" or CEF) ranged by more than a factor of 10x, adding almost as much uncertainty to the final calculation for broiler chickens as the 5 other factors combined. (To be fair, the CEF estimates of the other animal products were not as lopsided.)

As requested here is an introductory abstract.

The search for bias in the linguistic representations of our cognitive processes serves several purposes in this community. By pruning irrational thoughts, we can potentially effect each other in complex ways. Leaning heavy on cognitivist pedagogy, this essay represents my subjective experience trying to reconcile a perceived conflict between the rhetorical goals of the community and the absence of a generative, organic conceptualization of rationality.

The Story

    Though I've only been here a short time, I find myself fascinated by this discourse community. To discover a group of individuals bound together under the common goal of applied rationality has been an experience that has enriched my life significantly. So please understand, I do not mean to insult by what I am about to say, merely to encourage a somewhat more constructive approach to what I understand as the goal of this community: to apply collectively reinforced notions of rational thought to all areas of life.
   
    As I followed the links and read the articles on the homepage, I found myself somewhat disturbed by the juxtaposition of these highly specific definitions of biases to the narrative structures of parables providing examples in which a bias results in an incorrect conclusion. At first, I thought that perhaps my emotional reaction stemmed from rejecting the unfamiliar; naturally, I decided to learn more about the situation.

    As I read on, my interests drifted from the rhetorical structure of each article (if anyone is interested I might pursue an analysis of rhetoric further though I'm not sure I see a pressing need for this), towards the mystery of how others in the community apply the lessons contained therein. My belief was that the parables would cause most readers to form a negative association of the bias with an undesirable outcome.

    Even a quick skim of the discussions taking place on this site will reveal energetic debate on a variety of topics of potential importance, peppered heavily with accusations of bias. At this point, I noticed the comments that seem to get voted up are ones that are thoughtfully composed, well informed, soundly conceptualized and appropriately referential. Generally, this is true of the articles as well, and so it should be in productive discourse communities. Though I thought it prudent to not read every conversation in absolute detail, I also noticed that the most participated in lines of reasoning were far more rhetorically complex than the parables' portrayal of bias alone could explain. Sure the establishment of bias still seemed to represent the most commonly used rhetorical device on the forums ...

    At this point, I had been following a very interesting discussion on this site about politics. I typically have little or no interest in political theory, but "NRx" vs. "Prog" Assumptions: Locating the Sources of Disagreement Between Neoreactionaries and Progressives (Part 1) seemed so out of place in a community whose political affiliations might best be summarized the phrase "politics is the mind killer" that I couldn't help but investigate. More specifically, I was trying to figure out why it had been posted here at all (I didn't take issue with either the scholarship or intent of the article, but the latter wasn't obvious to me, perhaps because I was completely unfamiliar with the coinage "neoreactionary").

    On my third read, I made a connection to an essay about the socio-historical foundations of rhetoric. In structure, the essay progressed through a wide variety of specific observations on both theory and practice of rhetoric in classical Europe, culminating in a well argued but very unwieldy thesis; at some point in the middle of the essay, I recall a paragraph that begins with the assertion that every statement has political dimensions. I conveyed this idea as eloquently as I could muster, and received a fair bit of karma for it. And to think that it all began with a vague uncomfortable feeling and a desire to understand!

The Lesson

    So you are probably wondering what any of this has to do with rationality, cognition, or the promise of some deeply insightful transformative advice mentioned in the first paragraph. Very good.

    Cognition, a prerequisite for rationality, is a complex process; cognition can be described as the process by which ideas form, interact and evolve. Notice that this definition alone cannot explain how concepts like rationality form, why ideas form or how they should interact to produce intelligence. That specific shortcoming has long crippled cognitivist pedagogies in many disciplines -- no matter which factors you believe to determine intelligence, it is undeniably true that the process by which it occurs organically is not well-understood.

    More intricate models of cognition traditionally vary according to the sets of behavior they seek to explain; in general, this forum seems to concern itself with the wider sets of human behavior, with a strange affinity for statistical analysis. It also seems as if most of the people here associate agency with intelligence, though this should be regarded as unsubstantiated anecdote; I have little interest in what people believe, but those beliefs can have interesting consequences. In general, good models of cognition that yield a sense of agency have to be able to explain how a mushy organic collection of cells might become capable of generating a sense of identity. For this reason, our discussion of cognition will treat intelligence as a confluence of passive processes that lead to an approximation of agency.

    Who are we? What is intelligence? To answer these or any natural language questions we first search for stored-solutions to whatever we perceive as the problem, even as we generate our conception of the question as a set of abstract problems from interactions between memories. In the absence of recognizing a pattern that triggers a stored solution, a new solution is generated by processes of association and abstraction. This process may be central to the generation of every rational and irrational thought a human will ever have. I would argue that the phenomenon of agency approximates an answer to the question: "who am I?" and that any discussion of consciousness should at least acknowledge how critical natural language use is to universal agreement on any matter. I will gladly discuss this matter further and in greater detail if asked.

    At this point, I feel compelled to mention that my initial motivation for pursuing this line of reasoning stems from the realization that this community discusses rationality in a way that differs somewhat from my past encounters with the word.

    Out there, it is commonly believed that rationality develops (in hindsight) to explain the subjective experience of cognition; here we assert a fundamental difference between rationality and this other concept called rationalization. I do not see the utility of this distinction, nor have I found a satisfying explanation of how this distinction operates within accepted models for human learning in such a way that does not assume an a priori method of sorting the values which determine what is considered "rational". Thus we find there is a general derth of generative models of rational cognition beside a plethora of techniques for spotting irrational or biased methods of thinking.

    I see a lot of discussion on the forums very concerned with objective predictions of the future wherein it seems as if rationality (often of a highly probabilistic nature) is, in many cases, expected to bridge the gap between the worlds we can imagine to be possible and our many somewhat subjective realities. And the force keeping these discussions from splintering off into unproductive pissing about is a constant search for bias.

    I know I'm not going to be the first among us to suggest that the search for bias is not truly synonymous with rationality, but I would like to clarify before concluding. Searching for bias in cognitive processes can be a very productive way to spend one's waking hours, and it is a critical element to structuring the subjective world of cognition in such a way that allows abstraction to yield the kind of useful rules that comprise rationality. But it is not, at its core, a generative process.

    Let us consider the cognitive process of association (when beliefs, memories, stimuli or concepts become connected to form more complex structures). Without that period of extremely associative and biased cognition experienced during early childhood, we might never learn to attribute the perceived cause of a burn to a hot stove. Without concepts like better and worse to shape our young minds, I imagine many of us would simply lack the attention span to learn about ethics. And what about all the biases that make parables an effective way of conveying information? After all, the strength of a rhetorical argument is in it's appeal to the interpretive biases of it's intended audience and not the relative consistency of the conceptual foundations of that argument.

    We need to shift discussions involving bias towards models of cognition more complex than portraying it as simply an obstacle to rationality. In my conception of reality, recognizing the existence of bias seems to play a critical role in the development of more complex methods of abstraction; indeed, biases are an intrinsic side effect of the generative grouping of observations that is the core of Bayesian reasoning.

    In short, biases are not generative processes. Discussions of bias are not necessarily useful, rational or intelligent. A deeper understanding of the nature of intelligence requires conceptualizations that embrace the organic truths at the core of sentience; we must be able to describe our concepts of intelligence, our "rationality", such that it can emerge organically as the generative processes at the core of cognition.

    The Idea

    I'd be interested to hear some thoughts about how we might grow to recognize our own biases as necessary to the formative stages of abstraction alongside learning to collectively search for and eliminate biases from our decision making processes. The human mind is limited and while most discussions in natural language never come close to pressing us to those limits, our limitations can still be relevant to those discussions as well as to discussions of artificial intelligences. The way I see things, a bias free machine possessing a model of our own cognition would either have to have stored solutions for every situation it could encounter or methods of generating stored solutions for all future perceived problems (both of which sound like descriptions of oracles to me, though the latter seems more viable from a programmer's perspective).

    A machine capable of making the kinds of decisions considered "easy" for humans, might need biases at some point during it's journey to the complex and self consistent methods of decision making associated with rationality. This is a rhetorically complex community, but at the risk of my reach exceeding my grasp, I would be interested in seeing an examination of the Affect Heuristic in human decision making as an allegory for the historic utility of fuzzy values in chess AI.

    Thank you for your time, and I look forward to what I can only hope will be challenging and thoughtful responses.

Sorry if the positive illusions are old hat, but I searched and couldn't find any mention of this peer prediction stuff! If nothing else, I think the findings provide a quick heuristic for getting more reliable predictions of your future behavior - just poll a nearby friend!

Peer predictions are often superior to self-predictions. People, when predicting their own future outcomes, tend to give far too much weight to their intentions, goals, plans, desires, etc., and far to little consideration to the way things have turned out for them in the past. As Henry Wadsworth Longfellow observed,

"We judge ourselves by what we feel capable of doing, while others judge us by what we have already done"

...and we are way less accurate for it! A recent study by Helzer and Dunning (2012) took Cornell undergraduates and had them each predict their next exam grade, and then had an anonymous peer predict it too, based solely on their score on the previous exam; despite the fact that the peer had such limited information (while the subjects have presumably perfect information about themselves), the peer predictions, based solely on the subjects' past performance, were much more accurate predictors of subjects' actual exam scores.

In another part of the study, participants were paired-up (remotely, anonymously) and rewarded for accurately predicting each other's scores. Peers were allowed to give just one piece of information to help their partner predict their score; further, they were allowed to request just one piece of information from their partner to aid them in predicting their partner's score. Across the board, participants would give information about their "aspiration level" (their own ideal "target" score) to the peer predicting them, but would be far less likely to ask for that information if they were trying to predict a peer; overwhelmingly, they would ask for information about the participant's past behavior (i.e., their score on the previous exam), finding this information to be more indicative of future performance. The authors note,

There are many reasons to use past behavior as an indicator of future action and achievement. The overarching reason is that past behavior is a product of a number of causal variables that sum up to produce it—and that suite of causal variables in the same proportion is likely to be in play for any future behavior in a similar context.

They go on to say, rather poetically I think, that they have observed "the triumph of hope over experience." People situate their representations of self more in what they strive to be rather than in who they have already been (or indeed, who they are), whereas they represent others more in terms of typical or average behavior (Williams, Gilovich, & Dunning, 2012).

I found a figure I want to include from another interesting article (Kruger & Dunning, 1999); it illustrates this "better than average effect" rather well. Depicted below is an graph summarizing the results of study #3 (perceived grammar ability and test performance as a function of actual test performance):

Along the abscissa, you've got reality: the quartiles represent scores on a test of grammatical ability. The vertical axis, with decile ticks, corresponds to the same peoples' self-predicted ability and test scores. Curiously, while no one is ready to admit mediocrity, neither is anyone readily forecasting perfection; the clear sweet spot is 65-70%. Those in the third quartile seem most accurate in their estimations while those the highest quartile often sold themselves short, underpredicting their actual achievement on average. Notice too that the widest reality/prediction gap is for those the lowest quartile.

(Cross-posted from my blog.)

Since some belief-forming processes are more reliable than others, learning by what processes different beliefs were formed is for several reasons very useful. Firstly, if we learn that someone's belief that p (where p is a proposition such as "the cat is on the mat") was formed a reliable process, such as visual observation under ideal circumstances, we have reason to believe that p is probably true. Conversely, if we learn that the belief that p was formed by an unreliable process, such as motivated reasoning, we have no particular reason to believe that p is true (though it might be - by luck, as it were). Thus we can use knowledge about the process that gave rise to the belief that p to evaluate the chance that p is true.

Secondly, we can use knowledge about belief-forming processes in our search for knowledge. If we learn that some alleged expert's beliefs are more often than not caused by unreliable processes, we are better off looking for other sources of knowledge. Or, if we learn that the beliefs we acquire under certain circumstances - say under emotional stress - tend to be caused by unreliable processes such as wishful thinking, we should cease to acquire beliefs under those circumstances.

Thirdly, we can use knowledge about others' belief-forming processes to try to improve them. For instance, if it turns out that a famous scientist has used outdated methods to arrive at their experimental results, we can announce this publically. Such "shaming" can be a very effective means to scare people to use more reliable methods, and will typically not only have an effect on the shamed person, but also on others who learn about the case. (Obviously, shaming also has its disadvantages, but my impression is that it has played a very important historical role in the spreading of reliable scientific methods.)

A useful way of inferring by what process a set of beliefs was formed is by looking at its structure. This is a very general method, but in this post I will focus on how we can infer that a certain set of beliefs most probably was formed by (politically) motivated cognition. Another use is covered here and more will follow in future posts.

Let me give two examples. Firstly, suppose that we give American voters the following four questions:

1. Do expert scientists mostly agree that genetically modified foods are safe?
2. Do expert scientists mostly agree that radioactive wastes from nuclear power can be safely disposed of in deep underground storage facilities?
3. Do expert scientists mostly agree that global temperatures are rising due to human activities?
4. Do expert scientists mostly agree that the "intelligent design" theory is false?

The answer to all of these questions is "yes".* Now suppose that a disproportionate number of republicans answer "yes" to the first two questions, and "no" to the third and the fourth questions, and that a disproportionate number of democrats answer "no" to the first two questions, and "yes" to the third and the fourth questions. In the light of what we know about motivated cognition, these are very suspicious patterns or structures of beliefs, since that it is precisely the patterns we would expect them to arrive at given the hypothesis that they'll acquire whatever belief on empirical questions that suit their political preferences. Since no other plausibe hypothesis seem to be able to explain these patterns as well, this confirms this hypothesis. (Obviously, if we were to give the voters more questions and their answers would retain their one-sided structure, that would confirm the hypothesis even stronger.)

Secondly, consider a policy question - say minimum wages - on which a number of empirical claims have bearing. For instance, these empirical claims might be that minimum wages significantly decrease employers' demand for new workers, that they cause inflation and that they significantly reduce workers' tendency to use public services (since they now earn more). Suppose that there are five such claims which tell in favour of minimum wages and five that tell against them, and that you think that each of them has a roughly 50 % chance of being true. Also, suppose that they are probabilistically independent of each other, so that learning that one of them is true does not affect the probabilities of the other claims.

Now suppose that in a debate, all proponents of minimum wages defend all of the claims that tell in favour of minimum wages, and reject all of the claims that tell against them, and vice versa for the opponents of minimum wages. Now this is a very surprising pattern. It might of course be that one side is right across the board, but given your prior probability distribution (that the claims are independent and have a 50 % probability of being true) a more reasonable interpretation of the striking degree of coherence within both sides is, according to your lights, that they are both biased; that they are both using motivated cognition. (See also this post for more on this line of reasoning.)

The difference between the first and the second case is that in the former, your hypothesis that the test-takers are biased is based on the fact that they are provably wrong on certain questions, whereas in the second case, you cannot point to any issue where any of the sides is provably wrong. However, the patterns of their claims are so improbable given the hypothesis that they have reviewed the evidence impartially, and so likely given the hypothesis of bias, that they nevertheless strongly confirms the latter. What they are saying is simply "too good to be true".

These kinds of arguments, in which you infer a belief-forming process from a structure of beliefs (i.e you reverse engineer the beliefs), have of course always been used. (A salient example is Marxist interpretations of "bourgeois" belief structures, which, Marx argued, supported their material interests to a suspiciously high degree.) Recent years have, however, seen a number of developments that should make them less speculative and more reliable and useful.

Firstly, psychological research such as Tversky and Kahneman's has given us a much better picture of the mechanisms by which we acquire beliefs. Experiments have shown that we fall prey to an astonishing list of biases and identified which circumstances that are most likely to trigger them. 

Secondly, a much greater portion of our behaviour is now being recorded, especially on the Internet (where we spend an increasing share of our time). This obviously makes it much easier to spot suspicious patterns of beliefs.

Thirdly, our algorithms for analyzing behaviour are quickly improving. FiveLabs recently launched a tool that analyzes your big five personality traits on the basis of your Facebook posts. Granted, this tool does not seem completely accurate, and inferring bias promises to be a harder task (since the correlations are more complicated than that between usage of exclamation marks and extraversion, or that betwen using words such as "nightmare" and "sick of" and neuroticism). Nevertheless, better algorithms and more computer power will take us in the right direction.

In my view, there is thus a large untapped potential to infer bias from the structure of people's beliefs, which in turn would be inferred from their online behaviour. In coming posts, I intend to flesh out my ideas on this in some more details. Any comments are welcome and might be incorporated in future posts.

* The second and the third questions are taken from a paper by Dan Kahan et al, which refers to the US National Academy of Sciences (NAS) assessment of expert scientists' views on these questions. Their study shows that many conservatives don't believe that experts agree on climate change, whereas a fair number of liberals think experts don't agree that nuclear storage is safe, confirming the hypothesis that people let their political preferences influence their empirical beliefs. The assessment of expert consensus on the first and fourth question are taken from Wikipedia.

Asking people what they think about the expert consensus on these issues, rather than about the issues themselves, is good idea, since it's much easier to come to an agreement on what the true answer is on the former sort of question. (Of course, you can deny that professors from prestigious universities count as expert scientists, but that would be a quite extreme position that few people hold.) 

The head of the IMF was supposed to be Smith College's commencement speaker, but withdrew because of faculty and student protests.  A few professors (although none in the economics department) wrote to the IMF chief asking her to cancel.  The Smith Alumnae Quarterly asked several people, including myself, to write a 400 word article on the surrounding issues of diversity of thought and protest.  Below is a draft of my article.  I hope it's of interest and I would be grateful for any suggestions for improvement:

On Monday I need to give a presentation to a group of 6-15 finance professionals on Confirmation Bias. I intend to use the 2-4-6 task to demonstrate it.

Do people have any advise on how to make this work well? Do people tend to fall for it? Does it help them understand afterwards?

(In ages gone by I would have made this post longer, or in the Open Thread, or not at all. But I gather LW has been seeing a drop-off in volume, so I decided I'd lower the bar I set myself)

A useful bias to quote in discussions that spring up around the subjects we deal with on Less Wrong: Normalcy Bias. It's rather specific, but useful:

The normalcy bias, or normality bias, refers to a mental state people enter when facing a disaster. It causes people to underestimate both the possibility of a disaster occurring and its possible effects. This may results in situations where people fail to adequately prepare for a disaster, and on a larger scale, the failure of governments to include the populace in its disaster preparations.

The assumption that is made in the case of the normalcy bias is that since a disaster never has occurred then it never will occur. It can result in the inability of people to cope with a disaster once it occurs. People with a normalcy bias have difficulties reacting to something they have not experienced before. People also tend to interpret warnings in the most optimistic way possible, seizing on any ambiguities to infer a less serious situation.

I've elaborated on this form of government I have proposed in great detail on my blog here

The purpose of this post is to be a persuasive argument for my proposed system of democracy.  I am arguing along the lines that my legislature by sortition, random selection, is superior to electoral systems.  It also mirrors the advances in overcoming bias which are currently being pioneered in the Sciences.

I. The Problem

It is insane that we allow the same people who are elected to cast their eye on society to identify problems, write up the solutions to those problems, and then also vote to approve those solutions.  This triple function of government by elected officials isn't simply corruptible, but is inherently flawed in its decision making process.



II. The Central Committee, overcoming bias, electoral shenanigans, and demographics bias

In my system of sortition election there is a mini-referendum done by a huge sampling of 1,000-5,000 representatives at the highest level.  They vote everything up or down and cannot change anything about a bill themselves.  They are not congregated into one place and there is no politics between them.  They don't even need to know, nor could they know each other.  Perhaps they could be part of political parties, but there is no need or money behind this as the members of what I'm calling the Central Committee (C2) are never candidates and can individually never serve more than once per lifetime or perhaps per decade in 3 year terms.

Contentious issues can be moved to a general referendum.  In the 1,000 member C2, any law in the margins of 550-450 can have a special second vote proposed by the disagreeing side such that if more than 600 agree then the item is added to the general monthly or quarterly referendum conducted electronically with the entire population.  In this way the average person participates and feels heard by their government on a regular basis.

The major advantage of this C2 is that it is representative. It will have people from all areas, be 50% male and 50% female and will include all minorities.  There can be no great misrepresentation or capture of the legislature by a powerful group.  This overcome many of the inherent biases of an electoral system which in almost every democracy today routinely under represents minorities.

III. The Issue Committees (IC)

The IC is a totally separate body whose sole job is to identify areas of the law which need updating.  They are comprised of 100 citizens and are a split between 51 Regular Citizens (RCs) and 49 Expert Citizens (EC) serving single 3 year terms.  There are around 30 ICs and they each serve an area such as defence, environment, food safety, drug safety, telecommunications, changes to government, finance sector, banking sector, etc.

These committees will meet in person and discuss what needs exist which the government can address.  They do not get to write any laws, nor do they get to vote on any laws.  There are in fact more of these than there are members of the C2 and they will be the primary face of government where the average citizen can send in requests or communicate needs.  The IC shines a spotlight on the issues facing the country.  They also form the law writing bodies

IV. The Sub Committee (SC)

These are temporary parts of the legislature who write the laws.  They have no authority over what topic area they get to write laws about, that is determined by the IC and then voted upon by the C2.  They are composed of 10 RCs and 10 ECs with the support of 10 Lawyer Citizens (LC). The LCs do not participate to vote when the draft law can be moved up to the C2 for consideration, they simply help draft reasonable laws.

These SC's form and dissolved quickly, lasting no more than 3-6 months before a proposed law is made.  Being called up to the SC is a lot more akin to being drafted for Jury Duty than the IC or C2 level of government as it is a short term of service.

V. Conclusions

• This system is indeed more democratic and more representative than current electoral democracies.  It is less prone to corruption and electioneering is impossible as there are no elections. 

• Members of the C2, IC, and SC parts of intentionally split in their duties so no conflict of interest can arise and there is no legislator bias where they have pet bills and issues to push through for benefits to specific parts of the country.

• This system is also less influenced by the views an opinions of the very wealthy and the demographic and economic makeup of the people involved.

Two facts "everyone knows", an intriguing contrast, and a note of caution.

"Everyone knows" that people are much more willing to invest into cures than preventions. When a disaster hits, then money is no object; but trying to raise money for prevention ahead of time is difficult, hamstrung by penny-pinchers and short-termism. It's hard to get people to take hypothetical risks seriously. There are strong institutional reasons for this, connected with deep human biases and bureaucratic self-interest.

"Everyone knows" that governments overreact to the threat of terrorism. The amount spent on terrorism dwarfs other comparable risks (such as slipping and falling in your bath). There's a huge amount of security theatre, but also a lot of actual security, and pre-emptive invasions of privacy. We'd probably be better just coping with incidents as they emerge, but instead we cause great annoyance and cost across the world to deal with a relatively minor problem. There are strong institutional reasons for this, connected with deep human biases and bureaucratic self-interest.

And both these facts are true. But... they contradict each other. One is about a lack of prevention, the other about an excess of prevention. And there are more examples of excessive prevention: the war on drugs, for instance. In each case we can come up with good explanations as to why there is not enough/too much prevention, and these explanations often point to fundamental institutional forces or human biases. This means that the situation could essentially never have been otherwise. But the tension above hints that these situations may be a lot more contingent than that, more dependent on history and particular details of our institutions and political setup. Maybe if the biases were reversed, we'd have equally compelling stories going the other way. So when predicting the course of future institutional biases, or attempting to change them, take into account that they may not be nearly as solid or inevitable as they feel today.

Given LW’s keen interest in bias, it would seem pertinent to be aware of the biases engendered by the karma system. Note: I used to be strictly opposed to comment scoring mechanisms, but witnessing the general effectiveness in which LWers use karma has largely redeemed the system for me.

In “Social Influence Bias: A Randomized Experiment” by Muchnik et al, random comments on a “social news aggregation Web site” were up-voted after being posted. The likelihood of such rigged comments receiving additional up-votes were quantified in comparison to a control group. The results show that users were significantly biased towards the randomly up-voted posts:

The up-vote treatment significantly increased the probability of up-voting by the first viewer by 32% over the control group ... Uptreated comments were not down-voted significantly more or less frequently than the control group, so users did not tend to correct the upward manipulation. In the absence of a correction, positive herding accumulated over time.

At the end of their five month testing period, the comments that had artificially received an up-vote had an average rating 25% higher than the control group. Interestingly, the severity of the bias was largely dependent on the topic of discussion:

We found significant positive herding effects for comment ratings in “politics,” “culture and society,” and “business,” but no detectable herding behavior for comments in “economics,” “IT,” “fun,” and “general news”.

The herding behavior outlined in the paper seems rather intuitive to me. If before I read a post, I see a little green ‘1’ next to it, I’m probably going to read the post in a better light than if I hadn't seen that little green ‘1’ next to it. Similarly, if I see a post that has a negative score, I’ll probably see flaws in it much more readily. One might say that this is the point of the rating system, as it allows the group as a whole to evaluate the content. However, I’m still unsettled by just how easily popular opinion was swayed in the experiment.

This certainly doesn't necessitate that we reprogram the site and eschew the karma system. Moreover, understanding the biases inherent in such a system will allow us to use it much more effectively. Discussion on how this bias affects LW in particular would be welcomed. Here are some questions to begin with:

• Should we worry about this bias at all? Are its effects negligible in the scheme of things?
• How does the culture of LW contribute to this herding behavior? Is it positive or negative?
• If there are damages, how can we mitigate them?

Notes:

In the paper, they mentioned that comments were not sorted by popularity, therefore “mitigating the selection bias.” This of course implies that the bias would be more severe on forums where comments are sorted by popularity, such as this one.

For those interested, another enlightening paper is “Overcoming the J-shaped distribution of product reviews” by Nan Hu et al, which discusses rating biases on websites such as amazon. User gwern has also recommended a longer 2007 paper by the same authors which the one above is based upon: "Why do Online Product Reviews have a J-shaped Distribution? Overcoming Biases in Online Word-of-Mouth Communication"

This is interesting. Apparently, meditating for 15 minutes can reduce susceptibility to the sunk cost bias.

Across two separate experiments, the researchers tested this by giving one group of participants a 15-minute mindfulness meditation induction.

Then they were given a business scenario which was designed to test the sunk cost bias.

In comparison to a control condition, thinking mindfully doubled the number of people who could avoid the sunk cost bias.

In the control condition just over 40% of people were able to resist the bias. This shot up to almost 80% among those who were thinking mindfully.

The researchers achieved similar results in another experiment and then went on to examine exactly how mindfulness is helpful.

In a third experiment they found that mindfulness increases the focus on the present moment, as it should.

A focus on the present in turn reduced negative feelings participants had about the ‘sunk cost’–the time, money and effort that had gone to waste.

This reduction in negative emotion meant participants were much better equipped to resist the bias.

Ironically, I did a search on Less Wrong to see if something like this had been posted before and came across this comment:

Good points. The lack of scientific research discussed is certainly an issue. I did a quick literature sweep before writing this post, but decided not to include that information here.

One is a sunk cost fallacy. If you have sunk ten days into it you are less willing to ditch it because fallible humans are often unable to act like good economists and recognize that sunk costs are irrelevant.

At the dhamma.org courses I haven't found that to be the case. The management at the Massachusetts center informed me that a large majority of students never return to take a second course. Perhaps the cost needs to be larger; people may find it difficult to give up the practice (when they have good reason to) if they have done it daily for some length of time.

According to that anecdote, a large majority of students never take a second course in meditation. It might be due to the study above, where meditating itself makes people less likely to engage in sunk cost thinking.

(Disclaimer: This post refers to a certain political party as being somewhat crazy, which got some people upset, so sorry about that. That is not what this post is *about*, however. The article is instead about Less Wrong's social norms against pointing certain things out. I have edited it a bit to try and make it less provocative.)

A well-known post around these parts is Yudkowski’s “politics is the mind killer”. This article proffers an important point: People tend to go funny in the head when discussing politics, as politics is largely about signalling tribal affiliation. The conclusion drawn from this by the Less Wrong crowd seems simple: Don’t discuss political issues, or at least keep it as fair and balanced as possible when you do. However, I feel that there is a very real downside to treating political issues in this way, which I shall try to explain here. Since this post is (indirectly) about politics, I will try to bring this as gently as possible so as to avoid mind-kill. As a result this post is a bit lengthier than I would like it to be, so I apologize for that in advance.

I find that a good way to examine the value of a policy is to ask in which of all possible worlds this policy would work, and in which worlds it would not. So let’s start by imagining a perfectly convenient world: In a universe whose politics are entirely reasonable and fair, people start political parties to represent certain interests and preferences. For example, you might have the kitten party for people who like kittens, and the puppy party for people who favour puppies. In this world Less Wrong’s unofficial policy is entirely reasonable: There is no sense in discussing politics, since politics is only about personal preferences, and any discussion of this can only lead to a “Jay kittens, boo dogs!” emotivism contest. At best you can do a poll now and again to see what people currently favour.

Now let’s imagine a less reasonable world, where things don’t have to happen for good reasons and the universe doesn’t give a crap about what’s fair. In this unreasonable world, you can get a “Thrives through Bribes” party or an “Appeal to emotions” party or a “Do stupid things for stupid reasons” party as well as more reasonable parties that actually try to be about something. In this world it makes no sense to pretend that all parties are equal, because there is really no reason to believe that they are.

As you might have guessed, I believe that we live in the second world. As a result, I do not believe that all parties are equally valid/crazy/corrupt, and as such I like to be able to identify which are the most crazy/corrupt/stupid. Now I happen to be fairly happy with the political system where I live. We have a good number of more-or-less reasonable parties here, and only one major crazy party that gives me the creeps. The advantage of this is that whenever I am in a room with intelligent people, I can safely say something like “That crazy racist party sure is crazy and racist”, and everyone will go “Yup, they sure are, now do you want to talk about something of substance?” This seems to me the only reasonable reply.

The problem is that Less Wrong seems primarily US-based, and in the US… things do not go like this. In the US, it seems to me that there are only two significant parties, one of which is flawed and which I do not agree with on many points, while the other is, well… can I just say that some of the things they profess do not so much sound wrong as they sound crazy? And yet, it seems to me that everyone here is being very careful to not point this out, because doing so would necessarily be favouring one party over the other, and why, that’s politics! That’s not what we do here on Less Wrong!

And from what I can tell, based on the discussion I have seen so far and participated in on Less Wrong, this introduces a major bias. Pick any major issue of contention, and chances are that the two major parties will tend to have opposing views on the subject. And naturally, the saner party of the two tends to hold a more reasonable view, because they are less crazy. But you can’t defend the more reasonable point of view now, because then you’re defending the less-crazy party, and that’s politics. Instead, you can get free karma just by saying something trite like “well, both sides have important points on the matter” or “both parties have their own flaws” or “politics in general are messed up”, because that just sounds so reasonable and fair who doesn’t like things to be reasonable and fair? But I don’t think we live in a reasonable and fair world.

It’s hard to prove the existence of such a bias and so this is mostly just an impression I have. But I can give a couple of points in support of this impression. Firstly there are the frequent accusations of group think towards Less Wrong, which I am increasingly though reluctantly prone to agree with. I can’t help but notice that posts which remark on for example *retracted* being a thing tend to get quite a few downvotes while posts that take care to express the nuance of the issue get massive upvotes regardless of whether really are two sides on the issue. Then there are the community poll results, which show that for example 30% of Less Wrongers favour a particular political allegiance even though only 1% of voters vote for the most closely corresponding party. I sincerely doubt that this skewed representation is the result of honest and reasonable discussion on Less Wrong that has convinced members to follow what is otherwise a minority view, since I have never seen any such discussion. So without necessarily criticizing the position itself, I have to wonder what causes this skewed representation. I fear that this “let’s not criticize political views” stance is causing Less Wrong to shift towards holding more and more eccentric views, since a lack of criticism can be taken as tacit approval. What especially worries me is that giving the impression that all sides are equal automatically lends credibility to the craziest viewpoint, as proponents of that side can now say that sceptics take their views seriously which benefits them the most. This seems to me literally the worst possible outcome of any politics debate.

I find that the same rule holds for politics as for life in general: You can try to win or you can give up and lose by default, but you can’t choose not to play.

I read this article today and thought LW might find it interesting. The key finding is that in a number of different experiments, simple "self-affirmations" (such as writing about relationships with your friends or something else that makes you feel good about yourself) make people more open to changing their mind in cases where changing their mind would be damaging to their self-image. The proposed explanation is that people need to maintain a certain level of self-worth, and one way they do that is by refusing to accept evidence that would damage their sense of self-worth. But if they have a high enough sense of self-worth, they are less likely to do this. I haven't reviewed any of these studies personally, but the idea makes some sense and sounds pretty easy to try. Hat tip to Dan Keys for putting me onto the idea. I searched LW for "Sherman self-affirmation" and didn't see this discussed anywhere on LW, but I didn't look very hard.

Title: Accepting Threatening Information: Self–Affirmation and the Reduction of Defensive Biases

Authors: David K. Sherman and Geoffrey L. Cohen

Citation details: Current Directions in Psychological Science August 2002 vol. 11 no. 4 119-123

Abstract: Why do people resist evidence that challenges the validity of long–held beliefs? And why do they persist in maladaptive behavior even when persuasive information or personal experience recommends change? We argue that such defensive tendencies are driven, in large part, by a fundamental motivation to protect the perceived worth and integrity of the self. Studies of social–political debate, health–risk assessment, and responses to team victory or defeat have shown that people respond to information in a less defensive and more open–minded manner when their self–worth is buttressed by an affirmation of an alternative source of identity. Self–affirmed individuals are more likely to accept information that they would otherwise view as threatening, and subsequently to change their beliefs and even their behavior in a desirable fashion. Defensive biases have an adaptive function for maintaining self–worth, but maladaptive consequences for promoting change and reducing social conflict.

Key quotes: "Pro-choice partisans and pro-life partisans were presented with a debate between two activists on opposite sides of the abortion dispute….However, this confirmation bias was sharply attenuated among participants who affirmed a valued source of self-worth (by writing about a personally important value, such as their relations with friends)....although all participants left the debate feeling more confident in their beliefs about abortion than they had before, this polarization in attitude was significantly reduced among self-affirmed participants (cf. Lord et al., 1979)."  p. 120

"In one study (Cohen et al., 2000), devout opponents and proponents of capital punishment were presented with a persuasive scientific report that contradicted their beliefs about the death penalty’s effectiveness as a deterrent for crime....the responses of participants who received an affirmation of a valued self-identity (by writing about a personally important value, or by being provided with positive feedback on an important skill) proved more favorable.Self affirmed participants were less critical of the reported research, they suspected less bias on the part of the authors, and they even changed their overall attitudes toward capital punishment in the direction of the report they read." p. 121

"In one study, athletes who had just completed an intramural volleyball game assessed the extent to which each of a series of factors contributed to their team’s victory or defeat. As in past research (Lau & Russell, 1980),winners made more internal attributions for their victories than losers did for their defeats. However, among athletes who had reflected on an important value irrelevant to athletics, this self-serving bias was attenuated." p. 122

Related: Son of Low Hanging Fruit

Another post on finding low hanging fruit from Gregory Cochran's and Henry Harpending's blog West Hunter.

Clostridium difficile causes a potentially serious kind of diarrhea triggered by antibiotic treatments. When the normal bacterial flora of the colon are hammered by a broad-spectrum antibiotic, C. difficile often takes over and causes real trouble.  Mild cases are treated by discontinuing antibiotic therapy, which often works: if not, the doctors try oral metronidazole (Flagyl), then vancomycin , then intravenous metronidazole.  This doesn’t always work, and C. difficile infections kill about 14,000 people a year in the US.

One recent trial shows that fecal bacteriotherapy, more commonly called a stool transplant, works like gangbusters, curing ~94% of patients. The trial was halted because the treatment worked so well that refusing to poopify the control group was clearly unethical.  I read about this, but thought I’d heard about such stool transplants some time ago.  I had.  It was mentioned in The Making of a Surgeon, by William Nolen, published in 1970. Some crazy intern – let us call him Hogan – tried a stool transplant on a woman with a C. difficile infection. He mixed some normal stool with chocolate milk and fed it to the lady.  It made his boss so mad that he was dropped from the program at the end of the year.  It also worked. It was inspired by a article in Annals of Surgery, so this certainly wasn’t the first try.  According to Wiki,  there are more than 150 published reports on stool transplant, going back to 1958.

So what took so damn long?  Here we have a simple, cheap, highly effective treatment for C. difficile infection that has only become officially valid this year. Judging from the H. pylori  story, it may still take years before it is in general use.

Obviously, sheer disgust made it hard for doctors to embrace this treatment.  There’s a lesson here: in the search for low-hanging fruit,  reconsider approaches that are embarrassing, or offensive, or downright disgusting.

Investigate methods were abandoned because people hated them, rather because of solid evidence showing that they didn’t work.

Along those lines, no modern educational reformer utters a single syllable about corporal punishment: doesn’t that make you suspect it’s effective?  I mean, why we aren’t we caning kids anymore?  The Egyptians said that a boy’s ears are in his back: if you do not beat him he will not listen. Maybe they knew a thing or three.

Sometimes, we hate the idea’s authors: the more we hate them, the more likely we are to miss out on their correct insights. Even famous assholes had to be competent in some areas, or they wouldn’t have been able to cause serious trouble.

When predicting future threats, we also need to predict future policy responses. If mass pandemics are inevitable, it matters whether governments and international organisations can rise to the challenge or not. But its very hard to get a valid intuitive picture of government competence. Consider the following two scenarios:

• Governments are morasses of incompetence, saturated by turf wars, perverse incentives, inefficiencies, regulatory capture, and excessive risk aversion. The media reports a lot of the bad stuff, but doesn't have nearly enough space for it all, as it has to find some room for sport and naked celebrities. The average person will hear 1 story of government incompetence a day, anyone following the news will hear 10, a dedicated obsessive will hear 100 - but this is just the tip of the iceberg. The media sometimes reports good news to counterbalance the bad, at about a rate of 1-to-10 of good news to bad. This rate is wildly over-optimistic.

• Governments are filled mainly by politicians desperate to make a positive mark on the world. Civil servants are professional and certainly not stupid, working to clear criteria with a good internal culture, in systems that have learnt the lessons of the past and have improved. There is a certain amount of error, inefficiency, and corruption, but these are more exceptions than rules. Highly politicised issues tend to be badly handled, but less contentious issues are dealt with well. The media, knowing that bad news sells, fills their pages mainly with bad stuff (though they often have to exaggerate issues). The average person will hear 1 story of government incompetence a day, anyone following the news will hear 10, a dedicated obsessive will hear 100 - but some of those are quite distorted. The media sometimes reports good news to counterbalance the bad, at about a rate of 1-to-10 of good news to bad. This rate is wildly over-pessimistic.

These two situations are, of course, completely indistinguishable for the public. The smartest and most dedicated of outside observers can't form an accurate picture of the situation. Which means that, unless you have spent your entire life inside various levels of government (which brings its own distortions!), you don't really have a clue at general government competence. There's some very faint clues that governments may be working better than we generally think: looking at the achievements of past governments certainly seems to hint at a higher rate of success than the reported numbers today. And simply thinking about the amount of things that don't go wrong in a city, every day, hints that someone is doing their job. But these clues are extremely weak.

At this point, one should look up political scientists and other researchers. I hope to be doing that at some point (or the FHI may hire someone to do that). In the meantime, I just wanted to collect a few stories of government success to counterbalance the general media atmosphere. The purpose is not just to train my intuition away from the "governments are intrinsically incompetent" that I currently have (and which is unjustified by objective evidence). It's also the start of a project to get a better picture of where governments fail and where they succeed - which would be much more accurate and much more useful than an abstract "government competence level" intuition. And would be needed if we try and predict policy responses to specific future threats.

So I'm asking if commentators want to share government success stories they may have come across. Especially unusual or unsuspected stories. Vaccinations, clean-air acts, and legally establishing limited liability companies are very well known success stories, for instance, but are there more obscure examples that hint an unexpected diligence in surprising areas?

A recent paper in Cortex describes how caloric vestibular stimulation (CVS), i.e., rinsing of the ear canal with cold water, reduces unrealistic optimism. Here are some bits from the paper:

Participants were 31 healthy right-handed adults (15 men, 20–40 years)...

Participants were oriented in a supine position with the head inclined 30° from the horizontal and cold water (24 °C) was irrigated into the external auditory canal on one side (Fitzgerald and Hallpike, 1942). After both vestibular-evoked eye movements and vertigo had stopped, the procedure was repeated on the other side...

Participants were asked to estimate their own risk, relative to that of their peers (same age, sex and education), of contracting a series of illnesses. The risk rating scale ranged from −6 (lower risk) to +6 (higher risk). ... Each participant was tested in three conditions, with 5 min rest between each: baseline with no CI (always first), left-ear CI and right-ear CI (order counterbalanced). In the latter conditions risk-estimation was initiated after 30 sec of CI, when nystagmic response had built up. Ten illnesses were rated in each condition and the average risk estimate per condition (mean of 10 ratings) was calculated for each participant. The 30 illnesses used in this study (see Table 1) were selected from a larger pool of illnesses pre-rated by a separate group of 30 healthy participants.Overall, our participants were unrealistically optimistic about their chances of contracting illnesses at baseline ... and during right-ear CI. ...Post-hoc tests using the Bonferroni correction revealed that, compared to baseline, average risk estimates were significantly higher during left-ear CI (p = .016), whereas they remained unchanged during right-ear CI (p = .476). Unrealistic optimism was thus reduced selectively during left-ear stimulation.

(CI stands for caloric irrigation which is how CVS was performed.)

It is not clear how close the participants came to being realistic in their estimates after CVS, but they definitely became more pessimistic, which is the right direction to go in the context of numerous biases such as the planning fallacy.

The paper:

Vestibular stimulation attenuates unrealistic optimism

• Ryan McKay, 
• Corinne Tamagni, 
• Antonella Palla, 
• Peter Krummenacher, 
• Stefan C.A. Hegemann, 
• Dominik Straumann, 
• Peter Brugger

Related: Thick and Thin, Loss of local knowledge affecting intellectual trends

An entry I found in the archives on Gregory Cochran's and Henry Harpending's blog West Hunter.

In yet another example of  long-delayed discovery, forms of high-altitude lightning were observed for at least a century before becoming officially real (as opposed to really real).

Some thunderstorms manage to generate blue jets shooting out of their thunderheads, or  glowing red rings and associated tentacles around 70 kilometers up.   C T R Wilson predicted this long ago, back in the 1920s.  He had a simple model that gets you started.

You see, you can think of the thunderstorm, after a ground discharge,  as a vertical dipole. Its electrical field drops as the cube of altitude.  The threshold voltage for atmospheric breakdown is proportional to pressure, while pressure drops exponentially with altitude: and as everyone knows, a negative exponential drops faster than any power.

The curves must cross.   Electrical breakdown occurs.  Weird lightning, way above the clouds.

As I said, people reported sprites at least a hundred years ago, and they have probably been observed occasionally since the dawn of time. However, they’re far easier to see if you’re above the clouds – pilots often do.

Pilots also learned not to talk about it, because nobody listened.   Military and commercial pilots have to pass periodic medical exams known as ‘flight physicals’,  and there was a suspicion that reporting glowing red cephalopods in the sky might interfere with that.  Generally, you had to see the things that were officially real (whether they were really real or not), and only those things.

Sprites became real when someone recorded one by accident on a fast camera in 1989.  Since then it’s turned into a real subject, full of strangeness: turns out that thunderstorms  sometimes generate gamma-rays and even antimatter.

Presumably we’ve gotten over all that ignoring your lying eyes stuff by now.

May you tell others what you see. (~_^)

Brian Tomasik's latest article, 'Quantify with Care', seems to be of sufficient interest to readers of this forum to post a link to it here.  Abstract:

Quantification and metric optimization are powerful tools for reducing suffering, but they have to be used carefully. Many studies can be noisy, and results that seem counterintuitive may indeed be wrong because of sensitivity to experiment conditions, human error, measurement problems, or many other reasons. Sometimes you're looking at the wrong metric, and optimizing a metric blindly can be dangerous. Designing a robust set of metrics is actually a nontrivial undertaking that requires understanding the problem space, and sometimes it's more work than necessary. There can be a tendency to overemphasize statistics at the expense of insight and to use big samples when small ones would do. Finally, think twice about complex approaches that sound cool or impressive when you could instead use a dumb, simple solution.

From the article:

Using an adaptation of the standard 'bat-and-ball' problem, the researchers explored this phenomenon. The typical 'bat-and-ball' problem is as follows: a bat and ball together cost $1.10. The bat costs $1 more than the ball. How much does the ball cost? The intuitive answer that immediately springs to mind is 10 cents. However, the correct response is 5 cents.

The authors developed a control version of this problem, without the relative statement that triggers the substitution of a hard question for an easier one: A magazine and a banana together cost $2.90. The magazine costs $2. How much does the banana cost?

A total of 248 French university students were asked to solve each version of the problem. Once they had written down their answers, they were asked to indicate how confident they were that their answer was correct.

Only 21 percent of the participants managed to solve the standard problem (bat/ball) correctly. In contrast, the control version (magazine/banana) was solved correctly by 98 percent of the participants. In addition, those who gave the wrong answer to the standard problem were much less confident of their answer to the standard problem than they were of their answer to the control version. In other words, they were not completely oblivious to the questionable nature of their wrong answer.

Article in Science Daily: http://www.sciencedaily.com/releases/2013/02/130219102202.htm

Original abstract (the rest is paywalled): http://link.springer.com/article/10.3758/s13423-013-0384-5

Summary: Random events can preclude or steal attention from the goals you set up to begin with, hormonal fluctuation inclines people to change some of their goals with time. A discussion on how to act more usefully given those potential changes follows, taking in consideration the likelihood of a goal's success in terms of difficulty and length.

Throughout I'll talk about postponing utilons into undetectable distances. Doing so (I'll claim), is frequently motivationally driven by a cognitive dissonance between what our effects on the near world are, and what we wish they were. In other words it is:

A Self-serving bias in which Loss aversion manifests by postponing one's goals, thus avoiding frustration through wishful thinking about far futures, big worlds, immortal lives, and in general, high numbers of undetectable utilons.

I suspect that some clusters of SciFi, Lesswrong, Transhumanists, and Cryonicists are particularly prone to postponing utilons into undetectable distances, and in the second post I'll try to specify which subgroups might be more likely to have done so. The phenomenon, though composed of a lot of biases, might even be a good thing depending on how it is handled.

Sections will be:

1. What Significantly Changes Life's Direction (lists)

2. Long Term Goals and Even Longer Term Goals

3. Proportionality Between Goal Achievement Expected Time and Plan Execution Time

4. A Hypothesis On Why We Became Long-Term Oriented

5. Adapting Bayesian Reasoning to Get More Utilons

6. Time You Can Afford to Wait, Not to Waste

7. Reference Classes that May Be Postponing Utilons Into Undetectable Distances

8. The Road Ahead

Sections 4-8 will be on a second post so that I can make changes based on commentary to this one.

1What Significantly Changes Life's Direction

1.1 Predominantly external changes

As far as I recall from reading old (circa 2004) large scale studies on happiness, the most important life events in how much they change your happiness for more than six months are: 

• Becoming the caretaker of someone in a chronic non-curable condition

• Separation (versus marriage)

• Death of a Loved One

• Losing your Job

• Child rearing per child including the first

• Chronic intermittent disease

• Separation (versus being someone's girlfriend/boyfriend) 

Roughly in descending order. 

That is a list of happiness changing events, I'm interested here in goal-changing events, and am assuming there will be a very high correlation.

From life experience, mine, of friends, and of academics I've met, I'll list some events which can change someone's goals a lot:

• Moving between cities/countries

• Changing your social class a lot (losing a fortune or making one) 

• Spending highschool/undergrad in a different country to return afterwards

• Having a child, in particular the first one 

• Trying to get a job or make money and noticing more accurately what the market looks like

• Alieving Existential Risk

• Alieving as true, universally or personally, the ethical theories called "Utilitarianism" and "Consequentialism"

• Noticing that a lot of people are better than you at your initial goals, specially when those goals are competitive non-positive sum goals to some extent. 

• Interestingly, noticing that a lot of people are worse than you, making the efforts you once thought necessary not worth doing, or impossible to find good collaborators for. 

• Getting to know those who were once your idols, or akin to them, and considering their lives not as awesome as their work

• ... which is sometimes caused by ...

• Reading Dan Gilbert's "Stumbling on Happiness" and actually implementing his "advice that no one will follow" which is to think your happiness and emotions will correlate more with someone else who is already doing X which you plan to do than with your model of what it would feel like doing X. 

• Extreme social instability, such as wars, famine, etc...

• Having an ecstatic or traumatic experience, real or fictional. Such as seeing something unexpected, watching a life-changing movie, having a religious breakthrough, or a hallucinogenic one 

• Traveling to a place that is very different from your world and being amazed / shocked

• Not being admitted into your desired university / course

• Depression

• Surpassing a frustration threshold thus experiencing the motivational equivalent of learned helplessness

• Realizing your goals do not match the space-time you were born in, such as if making songs for CDs is your vocation, or if you are 30 years old in contemporary Kenya and want to teach medicine at a top 10 world college.

• Falling in love

That is long enough, if not exhaustive, so let's get going... 

1.2 Predominantly Internal Changes

I'm not a social endocrinologist but I think this emerging science agrees with folk wisdom that a lot changes in our hormonal systems during life (and during the menstrual cycle) and of course this changes our eagerness to do particular things. Not only hormones but other life events which mostly relate to the actual amount of time lived change our psychology. I'll cite some of those in turn:

• Exploitation increases and Exploration decreases with age

• Sex-Drive

• Maternity Drive - we have in portuguese an expression that “a woman's clock started ticking” which evidentiates a folk psychological theory that some part of it at least is binary

• Risk-proneness gives way to risk aversion, predominantly in males

• Premenstrual Syndrome - I always thought the acronym stood for 'Stress' until checking for this post.

• Hormonal diseases

• Middle Age crisis – recent controversy about other apes having it

• U shaped happiness curve through time – well, not quite

• Menstrual cycle events

2 Long Term Goals and Even Longer Term Goals

I have argued sometimes here and elsewhere that selves are not as agenty as most of the top writers in this website seem to me to claim they should be, and that though in part this is indeed irrational, an ontology of selves which had various sized selves would decrease the amount of short term actions considered irrational, even though that would not go all the way into compensating hyperbolic discounting, scrolling 9gag or heroin consumption. That discussion, for me, was entirely about choosing between doing now something that benefits 'you_now' _, 'you_today', 'you_tomorrow', 'you_{this weekend}' or maybe a month from now. Anything longer than that was encompassed in a “Far Future” mental category. The interest here to discuss life-changing events is only in those far future ones which I'll split into arbitrary categories:

1) Months 2) Years 3) Decades 4) Bucket List or Lifelong and 5) Time Insensitive or Forever.

I have known more than ten people from LW whose goals are centered almost completely at the Time Insensitive and Lifelong categories, I recall hearing :

"I see most of my expected utility after the singularity, thus I spend my willpower entirely in increasing the likelihood of a positive singularity, and care little about my current pre-singularity emotions", “My goal is to have a one trillion people world with maximal utility density where everyone lives forever”, “My sole goal in life is to live an indefinite life-span”, “I want to reduce X-risk in any way I can, that's all”.

I myself stated once my goal as

“To live long enough to experience a world in which human/posthuman flourishing exceeds 99% of individuals and other lower entities suffering is reduced by 50%, while being a counterfactually significant part of such process taking place.”

Though it seems reasonable, good, and actually one of the most altruistic things we can do, caring only about Bucket Lists and Time Insensitive goals has two big problems

1. There is no accurate feedback to calibrate our goal achieving tasks

2. The Goals we set for ourselves require very long term instrumental plans, which themselves take longer than the time it takes for internal drives or external events to change our goals.

The second one has been said in a remarkable Pink Floyd song about which I wrote a motivational text five years ago: Time.

You are young and life is long and there is time to kill today

And then one day you find ten years have got behind you

No one told you when to run, you missed the starting gun

 

And you run and you run to catch up with the sun, but it's sinking

And racing around to come up behind you again

The sun is the same in a relative way, but you're older

Shorter of breath and one day closer to death

 

Every year is getting shorter, never seem to find the time

Plans that either come to naught or half a page of scribbled lines

Okay, maybe the song doesn't say exactly (2) but it is within the same ballpark. The fact remains that those of us inclined to care mostly about very long term are quite likely to end up with a half baked plan because one of those dozens of life-changing events happened, and that agent with the initial goals will have died for no reason if she doesn't manage to get someone to continue her goals before she stops existing.

This is very bad. Once you understand how our goal-structures do change over time – that is, when you accept the existence of all those events that will change what you want to steer the world into – it becomes straightforward irrational to pursue your goals as if that agent would live longer than it's actual life expectancy. Thus we are surrounded by agents postponing utilons into undetectable distances. Doing this is kind of a bias in the opposite direction of hyperbolic discounting. Having postponed utilons into undetectable distances is predictably irrational because it means we care about our Lifelong, Bucket List, and Time Insensitive goals as if we'd have enough time to actually execute the plans for these timeframes, while ignoring the likelihood of our goals changing in the meantime and factoring that in.

I've come to realize that this was affecting me with my Utility Function Breakdown which was described in the linked post about digging too deep into one's cached selves and how this can be dangerous. As I predicted back then, stability has returned to my allocation of attention and time and the whole zig-zagging chaotic piconomical neural Darwinism that had ensued has stopped. Also  relevant is the fact that after about 8 years caring about more or less similar things, I've come to understand how frequently my motivation changed direction (roughly every three months for some kinds of things, and 6-8 months for other kinds). With this post I intend to learn to calibrate my future plans accordingly, and help others do the same. Always beware of other-optimizing though.

Citizen: But what if my goals are all Lifelong or Forever in kind? It is impossible for me to execute in 3 months what will make centenary changes.

Well, not exactly. Some problems require chunks of plans which can be separated and executed either in parallel or in series. And yes, everyone knows that, also AI planning is a whole area dedicated to doing just that in non-human form. It is still worth mentioning, because it is much more simply true than actually done.

This community in general has concluded in its rational inquiries that being longer term oriented is generally a better way to win, that is, it is more rational. This is true. What would not be rational is to in every single instance of deciding between long term or even longer term goals, choose without taking in consideration how long will the choosing being exist, in the sense of being the same agent with the same goals. Life-changing events happen more often than you think, because you think they happen as often as they did in the savannahs in which your brain was shaped.

3 Proportionality Between Goal Achievement Expected Time and Plan Execution Time

So far we have been through the following ideas. Lots of events change your goals, some externally some internally, if you are a rationalist, you end up caring more about events that take longer to happen in detectable ways (since if you are average you care in proportion to emotional drives that execute adaptations but don't quite achieve goals). If you know that humans change and still want to achieve your goals, you'd better account for the possibility of changing before their achievement. Your kinds of goals are quite likely prone to the long-term since you are reading a Lesswrong post.

Citizen: But wait! Who said that my goals happening in a hundred years makes my specific instrumental plans take longer to be executed?

I won't make the case for the idea that having long term goals increases the likelihood of the time it takes to execute your plans being longer. I'll only say that if it did not take that long to do those things your goal would probably be to have done the same things, only sooner.

To take one example: “I would like 90% of people to surpass 150 IQ and be in a bliss gradient state of mind all the time”

Obviously, the sooner that happens, the better. Doesn't look like the kind of thing you'd wait for college to end to begin doing, or for your second child to be born. The reason for wanting this long-term is that it can't be achieved in the short run.

Take Idealized Fiction of Eliezer Yudkosky: Mr Ifey had this supergoal of making a Superintelligence when he was very young. He didn't go there and do it. Because he could not. If he could do it he would. Thank goodness, for we had time to find out about FAI after that. Then his instrumental goal was to get FAI into the minds of the AGI makers. This turned out to be to hard because it was time consuming. He reasoned that only a more rational AI community would be able to pull it off, all while finding a club of brilliant followers in this peculiar economist's blog. He created a blog to teach geniuses rationality, a project that might have taken years. It did, and it worked pretty well, but that was not enough, Ifey soon realized more people ought to be more rational, and wrote HPMOR to make people who were not previously prone to brilliance as able to find the facts as those who were lucky enough to have found his path. All of that was not enough, an institution, with money flow had to be created, and there Ifey was to create it, years before all that. A magnet of long-term awesomeness of proportions comparable only to the Best Of Standing Transfinite Restless Oracle Master, he was responsible for the education of some of the greatest within the generation that might change the worlds destiny for good. Ifey began to work on a rationality book, which at some point pivoted to research for journals and pivoted back to research for the Lesswrong posts he is currently publishing. All that Ifey did by splitting that big supergoal in smaller ones (creating Singinst, showing awesomeness in Overcoming Bias, writing the sequences, writing the particular sequence “Misterious Answers to Misterious Questions” and writing the specific post “Making Your Beliefs Pay Rent”). But that is not what I want to emphasize, what I'd like to emphasize is that there was room for changing goals every now and then. All of that achievement would not have been possible if at each point he had an instrumental goal which lasts 20 years whose value is very low uptill the 19th year. Because a lot of what he wrote and did remained valuable for others before the 20th year, we now have a glowing community of people hopefully becoming better at becoming better, and making the world a better place in varied ways.

So yes, the ubiquitous advice of chopping problems into smaller pieces is extremely useful and very important, but in addition to it, remember to chop pieces with the following properties:

(A) Short enough that you will actually do it.

(B) Short enough that the person at the end, doing it, will still be you in the significant ways.

(C) Having enough emotional feedback that your motivation won't be capsized before the end. and

(D) Such that others not only can, but likely will take up the project after you abandon it in case you miscalculated when you'd change, or a change occurred before expected time.

Sections 4-8 will be on a second post so that I can make changes based on commentary to this one.

I graduated from high school and wish to further my education formally by studying for a bachelor's degree in order to become a medical researcher. I could, for instance, take two different academic paths:

1. Study Medicine at undergraduate level and then do a postdoctoral fellowship.

2. Study Biochemistry at undergraduate level, then study for a PhD at graduate level, and finally do a postdoctoral fellowship.

Since I will do these studies in Europe, they each take approximately the same amount of time, namely 6 to 8 years.

Do I want to do treat patients? No, I do not. But I am considering Medicine because it can be a buffer against my own mediocrity: in case I turn out to be a below average scientist, I will be screwed royally. From my personal job shadowing experience, Medicine, on the other hand, requires mere basic intellectual traits, primarily the ability to memorize heaps of information. And those I think I have. To do world-class research though I'd have to be an intellectual heavyweight, and of that I'm not so sure.

How do I decide what path to  follow?

The reason I'm asking you strangers for advice is because I evidently have biases, such as the pessimism/optimism bias or the Dunning–Kruger effect, that impair my ability to reason clearly; and people who know me personally are likewise prone to make errors in advising me because of biases like, say, the Halo effect. (Come to think of it, thinking that I can't become an above average scientist is in itself a self-defeating prophecy!)

Do you think that one ought to always seek advice from total strangers in order to be safeguarded from his/her own biases?

PS: I apologize if I should have written this in a specific thread. I'll delete my article if that's necessary.

I do an informal experiment testing whether LessWrong karma scores are susceptible to a form of anchoring based on the first comment posted; a medium-large effect size is found although the data does not fit the assumed normal distribution & the more sophisticated analysis is equivocal, so there may or may not be an anchoring effect.

Full writeup on gwern.net at http://www.gwern.net/Anchoring