An Anchoring Experiment

prase

17 An Anchoring Experiment

1st Apr 2011

2 min read

17

The experiment is closed, for the results look here.

In recent discussion I have expressed an opinion that anchoring may, for some quantitative questions, cause the answer to lie further away from the correct value than the anchor itself. For concreteness, let's suppose that the correct value of a quantity Q is x, and the subject is asked whether Q is greater or lower than y, y > x. My hypothesis is that the anchor moves the subject's probability distribution up as a whole, including the part which already has been lying above y. Therefore the subjects will positively answer the question "Is Q > y ?" more often than their guess would exceed y if they were just asked to estimate the value of Q with no anchor given. One commenter apparently disagreed. I thought it may be interesting to resolve the disagreement experimentally. (More generally, I would like to see how well LW audience fights the standard biases, and if this experiment turns out successful - which means the number of respondents be greater than, say, five - I would think about posting more of this kind.)

How to participate:

The experiment has two parts.

First, toss a coin to decide whether you belong to the biased group I or the control group II for the first question. If you belong to the group I, look at a comment linked below, which will give you a question of form "is Q is greater or lower than y", where y is either significantly lower or significantly greater than the correct value of Q. The comment has a form of a typical LW poll. If you belong to the group II, look at different linked comment which asks "what is the value of Q", and then give your estimate in a subcomment there.

The second part is completely analogical to the first one, only with a different question. If you have participated in the first part within the group I, take part in the group II for the second part, and vice versa. Try to eliminate the irrelevant biases: switch on the anti-kibitzer before looking on the group I questions to avoid being influenced by the votes of others. Don't read the subcomments of the group II questions before writing down your own.

The hypothesis is that the percentage of the group I respondents answering incorrectly will be greater than the percentage of the group II respondents estimating on the incorrect side of the anchor.

First part: Question for the group I. Question for the group II.

Second part: Question for the group I. Question for the group II.

Personal Blog

17

Mentioned in

28An Anchoring Experiment: Results

An Anchoring Experiment

0radical_negative_one

New Comment

89 comments, sorted by

top scoring

Click to highlight new comments since: Today at 1:39 PM

Some comments are truncated due to high volume. (⌘F to expand all)Change truncation settings

[-]Dorikka14y30

Uh, some people's answers are short enough that their numbers are showing up in the 'Recent Comments' section. Please make a buffer in your comments prior to the answer if it's short (I don't think spaces work, but I'm not sure.) so this doesn't happen.

1prase14y

Thanks. I have been already given a notice about that and have included a similar instruction in the relevant comments.

2Normal_Anomaly14y

Your sample buffer is one character too short.

2prase14y

Fixed.

[-]Pavitra14y20

We should vote down the group II questions as well.

1RobinZ14y

I suggest upvoting the main post and downvoting the group II question you answered. Edit: At least as a stopgap until prase makes a proper karma balance comment.

[-]RHollerith14y20

My coinflip put me in group II, but since I was reading /comments/ I saw one of the anchors intended for group I, so I decided that for me to continue would bias the results.

In addition I was primed (anchored? terminology?) by the answer of another subject in group II, and this time the problem could not be blamed on my having been reading /comments/.

Nice try, but I doubt the method of collecting responses you have chosen captures the aspects of reality you wish to investigate. Maybe you could redo the collection of responses by having respondents email t... (read more)

1prase14y

Thanks for feedback. I know that it is difficult to avoid seeing the comments of others. The problem with e-mail is that it is marginally less convenient to do, and I was afraid that people would have not enough motivation to respond. As for hiding the instructions, I don't see a strong reason. I suppose priming is a strong enough effect to persist even if people know they are being tested for it.

1RHollerith14y

If the coin flip puts me in group II, does not my inadvertently seeing the question intended for group I (e.g., "Is the population of the Central African Republic lower or greater than 20 million?") de facto reassign me to group I? In other words, after I had been exposed to the anchor (20 million) would not my continued participation in the control group have undermined the purpose of the control group? ADDED. I tend to agree, but that's not it.

0Pavitra14y

Agreed. It would have been better to always answer the group-II question first, whichever one you were assigned to. (That is, switch the part 1 group II question with the part 2 group I question.)

0prase14y

I have misunderstood you and thought you suggest that I hide parts of the original post. I have expected that people follow on the links in the bottom of the post. If you click on the group II question link, you don't see the group I question. Of course, if you scroll down, you will see it, but therefore I have warned against reading the comments.

0rwallace14y

As a positive data point, the instructions worked for me, and I think this is an interesting and well thought-out experiment - the method isn't perfectly watertight, but that has to be balanced against making it easy and attractive for people to participate, as well as against the cost (in your time and energy) of setting it up. I think you struck a good balance.

[-]Kenoubi14y10

Err, I think I screwed this up. I thought if you were in group I for the first part, you'd also be in group I for the second part. I still think this would be more intuitive (just reverse the questions in part 2).

0prase14y

Perhaps, but I am not going to change it now when several answers have been already collected.

0Kenoubi14y

Sure, I wasn't suggesting you should. Given the current situation, I'd change to I probably would have tried to use bulleted lists and/or bold to make the text more scannable overall. But maybe my error was atypical and all this isn't necessary.

[-]ShardPhoenix14y00

From a quick eyeballing of the results so far it does seem that the anchors are having the expected effect, even though we know that we are being anchored.

[-]RobinZ14y00

Anti-kibitzer is acting odd in my Google Chrome browser - it's not bolding links and all the comments are shown in full, rather than truncated to a single line.

[-]FAWS14y00

I think you should have posted just the links to the questions and the instructions on which link to follow without explaining what the experiment is about.

0prase14y

That would be certainly a reasonable way to do it, however I suppose that the LW audience is well informed about the biases and able to guess what it is all about even without being explicitly told. Meanwhile I wanted to provide some explanation why I am doing this, and to publish the tested hypothesis in advance (as a measure of honesty). The effect seems robust enough to survive when people know that they are being tested for it.

[-]prase14y-10

Part 1, group II question:

What is the population of the Central African Republic?

Give an estimate in a subcomment. Please begin your answer with "I suppose the correct value is probably" or some other preface of comparable length; if you write just the number, it appears in the Recent Comments bar and can bias other respondents.

3Dorikka14y

I note that I don't know what the Central African Republic is, but I'm guessing that it's somewhere in Africa. I'm probably best off estimating that it has the average population of a state in Africa, but I don't know what this is either. It will be much smaller than the US population (P>.999), but that doesn't help me....much. I note that this fact in itself implies that I can form a probability distribution. My estimated probability of the Central African Republic being smaller than the US is .999, so I have .999 to distribute within the range 0>450,000,000 (4.5x10^8 is my estimate of the US population). I don't think that I should use a hypothesis of complete ignorance here because I think that I would have a greater chance of hearing of it the more people it had, but this isn't very dependable considering that I don't follow politics that much. I also note that you may have made the name up, giving it a population of zero. But wouldn't that render the experiment invalid? I can't immediately see why. You're also posting this on April Fools, of all days, and we haven't had a joke yet, so I weight this option more than the others. Yet we're trying to minimize error here, not pick the integer which we believe has the highest probability of being a correct answer. To take this possibility into account, I will multiply my guess of an average nation's population by .4. What data can I use to make that guess. Well, I was at a Model United Nations conference once, and I thought that there were about 94 nations present. I suppose that there are about 40 nations not present in the UN, so let's run 6.5 billion/136. We get 46 million. I will multiply that number by .4 to get 18.7 million. Reposted: put this in the wrong place the first time.

1prase14y

Upvoted for being the sort of analysis one should do to avoid anchoring. Ironically it ended up much closer to the anchor than to the correct value.

1[anonymous]14y

rot13ed: svsgl zvyyvba crbcyr

0lincolnquirk14y

I suppose the correct value is probably 16 million.

0LastVillaiN14y

I suppose the correct value is probably around 2 million.

0AnotherKevin14y

I suppose the correct value is probably 10 million

0Pfft14y

I suppose the correct value is probably 10 million.

0[anonymous]14y

blah blah blah I also would have guessed 5 million.

0gwern14y

Probably: around 10 million.

0Oscar_Cunningham14y

I had no idea there was even a country with that name, 20 million.

0MinibearRex14y

I suppose the correct value is probably 10 million. (Incidentally, I have no clue whatsoever, this is a guess)

0Paul Crowley14y

Zl thrff jbhyq or svir zvyyvba

0gscshoyru14y

Edit: scratch that. I was influenced by the first answer I saw on the page... my answer should not count. Leaving it for posterity, below -- I'd make it strikethrough if I knew how. I suppose the correct value is probably 50 million.

0paulfchristiano14y

I don't know what the Central African Republic is, but I would guess about 1 million.

0KenChen14y

I remember looking up the population of Libya recently, which is around 6m, and I feel like the population of Central African Republic is a bit lower. I suppose the correct value is around 3m.

0Normal_Anomaly14y

I suppose the correct value is around 5 million. I posted this without looking up any data that might have informed my answer.

0drethelin14y

I have very little idea but it's probably around five million?

0[anonymous]14y

About 25 million people.

0Larifari14y

5 million

0cousin_it14y

30 million people

[-]prase14y-20

Part 2, group II question:

What is the altitude of the highest point in Sweden?

1jaimeastorga200014y

I suppose the correct value is probably 5000 km. I had no idea what a reasonable height for a mountain might be, but I did half-remember a quote from the Spanish dub of Jurassic Park III about K2 being between x000 km and 9000 km. I knew that K2 was the 2nd highest mountain on the planet, so I thought Sweden's own highest mountain might be quite a bit smaller and therefore wrote down 5000 km as my guess. Except that, lo and behold, it turns out that I misremembered meters as kilometers! Great, now I'm gonna have the most inaccurate answer in this subthread... so much for my attempt to at least be in the correct order of magnitude.

1prase14y

5000 kilometres ?

1jaimeastorga200014y

Yes, 5000000 meters. Like I said, I didn't realize I had screwed up until I looked at the other posts, which was after I had written my answer down.

0gjm14y

I don't know anything about Sweden, nor about geology, so I will not be surprised to find that the following is very wrong. I guess that the highest point in Sweden is about 1800m above sea level.

0Mystfan14y

I'd guess around 8,000 feet (I seem to recall Sweden having only smaller mountains)

0LastVillaiN14y

Oh darn that question, I climbed it last summer. Rot13: Fyvtugyl zber guna gjb gubhfnaq naq bar uhaqerq zrgref nobir gur frn. Gur zbhagvna vf fuevaxvat.

0beriukay14y

I suppose the correct value is probably about 10,000 feet. I base it on not knowing much about Sweden, but seeing bumps on maps, and guessing that they don't have anything any bigger than 2/3 of Mt. McKinley, which probably served as an anchor for me, so I'm not sure if I actually belong in group II any more. Edit: The number I guessed was actually closer to half a McKinley. So much for memory.

0jschulter14y

I suppose the correct value is probably around 3000 m. I know of no large mountains to be found in Sweden, so I'm guessing what seems to be a reasonably low number.

0Nic_Smith14y

My guess of the answer to this question, bearing in mind that I haven't studied geography since elementary school and am very unsure at to the order of magnitude, is 3600 feet.

0Nic_Smith14y

Some notes on my reasoning: I was originally going to guess an answer of 10000 feet on the basis that I vaguely remembered a Nickleback song indicating that commercial airplanes go up even higher, but thought without looking that the lyrics might have been "join the mile high club at 20000 feet" and so disregarded this answer as biased (but apparently the lyrics to "Rock Star" are "37000 feet" instead, so this was a mistake). I then figured that 6x the height of the Sears Tower would be a somewhat reasonable height for a mountain, and thought that the Tower was either 300ft or 600ft tall, eventually deciding on 600ft (actual height: 1451ft). 6(600) = 3600.

0falenas10814y

Absolutely no idea what the answer is, so I'll guess 2000 feet.

0kpreid14y

I have no concrete preexisting knowledge about the scale of altitudes, or the geography of Sweden, including whether or not it contains mountains. I am therefore somewhat arbitrarily choosing one-half of the value which Wikipedia labels “high altitude”, namely 1200 meters. I also note that on first reading I missed “the highest point” and worked on answering the typical occupied altitude of Sweden, and had some confusion with Switzerland.

0radical_negative_one14y

I suppose the correct value is probably... I'm trying to remember anything about the geography of Sweden and i'm drawing a blank. Hmm, i know that the highest mountains in the continental US are about 14,000 feet. I remember the highest mountains in Europe are a few thousand feet higher than that, but i think those were probably in the Alps. At the moment i can't remember hearing much about famous Scandinavian mountain ranges, but then again they could be higher than anything in the continental US and still not be as famous as the Alps, right? For the sake of having an answer i'll guess about 14,000ft.

0Pavitra14y

I suppose the correct value is probably..................here are many characters for padding............... 100 meters above sea level.

0[anonymous]14y

Well, I'm constantly disappointed by how low the highest peaks are here in the UK, and tend to imagine almost all other nations as having much higher ones. I also know that Norway is extremely mountainous, and that Sweden is much less so, but might still be mountainous around the edges or up north (they hosted the winter olympics one year, after all.) So I'm going to guess 2000m. This is consistent with having been "anchored" by the anchor, but honestly, I think my own weird logic of taking the UK's highest mountain and "adding a bit on" is to blame.

0ShardPhoenix14y

No idea, but my impression of Sweden isn't very mountainous so I'll guess 3000m (~10000 feet)

0gwern14y

I would guess somewhere around 12,000 feet.

0RobinZ14y

Thinking about the question in the context of my current education on geography (a disturbingly large portion of which is from the 1632 series of books), I would imagine that Sweden includes some reasonably tall mountains - but not record-breakingly tall. As ignorant as I am, I'm going to use 5000 feet as a benchmark for an ordinary mountain, and guess 12,000 feet.

0Eneasz14y

No clue, but anchoring off my state and adjusting downward on the belief that it's a high-than-average state: 10,000 feet

0orangecat14y

Knowing virtually nothing about the geography of Sweden, I'll guess 10,000 feet.

0FAWS14y

Probably in the low four digit meter range, I guess 1200m.

0rwallace14y

I looked at a map some years ago, but I don't really remember it. I think it's a bit more mountainous than Ireland or Scotland, so I'm going to guess 1500 m.

0Benquo14y

i suppose the correct value is probably 10,000 feet.

0Nornagest14y

If I remember right, the Scandinavian peninsula is rugged but not particularly mountainous in absolute terms, and the Swedish part is less so than the Norwegian. Yrg'f fnl friragrra uhaqerq naq svsgl zrgref. Edit: Rot13'd.

0KenChen14y

I seem to recall that Sweden is "somewhat mountainous", so I suppose that the correct value is around 2000m.

0[anonymous]14y

I accidentially looked at the first comment, so my guess could be already biased, however, I think I remember that I hat a rough figure already in my head when I saw the other number, so probably the guess is still valid. I think 1500 m

0endoself14y

I suppose the correct value is about 5 km.

0cata14y

I guess that the highest point in Sweden is probably about 1000 feet above sea level.

0Psy-Kosh14y

Let's see, how would I go about guessing? I forget if Sweden is supposed to be mountainous, and looking up stuff is probably outside the intent of the experiment, so... I'll give a range. Probably between 500ft and 10,000ft. If you want, for the purposes of the experiment, a single number, let's say 5000ft.

0Plasmon14y

I suppose the correct value is ... 1800m

0[anonymous]14y

3500feet.

0[anonymous]14y

I guess the following number: 5000 feet

0EStokes14y

1000 ft

0[anonymous]14y

900 meters.

0Manfred14y

7000 ft (about 7/3 km)

0Dan_Moore14y

500 meters

[+]prase14y-240

[+]prase14y-380

Moderation Log