= 27d567bc2d48d9f40e9ccc20ea370b15)
Stefan_Schubert comments on Open thread, Nov. 10 - Nov. 16, 2014 - Less Wrong Discussion
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (194)
I am constructing a political bias quiz together with Spencer Greenberg, who runs the site Clearer Thinking and wonder if people could help me coming up with questions. The quiz will work like this. First, you'll respond to a number of questions regarding your political views: e.g., republican or democrat, pro-life or pro-choice, pro- or anti-immigration, etc. Then you'll be given a number of factual questions. On the basis of your answers, you'll be given two scores:
1) The number of correct answers - your degree of political knowledge. 2) Your degree of political bias.
The assigmment of political bias will be based on the following reasoning. Suppose you're a hard-core environemntalist, and are consistently right about the questions where hard-core environemntalist like the true answer (e.g. climate change) but consistenly wrong about the questions where they are not (e.g. GMOs). Now this suggests that you have not reviewed these questions impartially, but that you acquire whatever factual beliefs suit your political opinions - i.e. that you're biased. Hence, the higher the ratio between the correct answers you like and the correct answers you dislike is, the more biased you are.
(The argument is slightly more complicated, but this should suffice for the present purposes. Also, the test shouldn't be taken too seriously - the main purpose is to make people think more about political bias as a problem).
The questions are intended for an American audience. I have come up with the following questions so far:
1) Which of the following statements best describes expert scientists’ views of the claim that global temperatures are rising due to human activities (this question is taken from a great paper by Dan Kahan )? (Most agree it's true, divided, most agree it's false)
2) Which of the following statements best describes expert scientists’ views of the claim that genetically modified foods are safe? (Same possible answers)
3) Which of the following statements best describes expert scientists’ views of the claim that humans are causing mass extinction of species at a rate that is at least 100 times the natural rate? (Same possible answers)
4) Which of the following statements best describes expert scientists’ views of the claim that radioactive wastes from nuclear power can be safely disposed of in deep underground storage facilities? (Same possible answers)
5) Which of the following statements best describes expert scientists’ views of the claim that humankind evolved from other species through natural selection. (Same possible answers)
6) Which of the following statements best describes expert scientists’ views of the claim that the death penalty increases homicide rates. (Same possible answers)
7) Studies show that on spatial reasoning tests, male mean scores are higher than females', whereas the converse is true of emotional intelligence tests. (True/false)
8-10) (These are taken from Bryan Caplan's excellent The Myth of the Rational Voter ) Expert economists were given the following possible explanations for why the economy isn't doing better. For each one, please indicate whether they thought it is a major reason the economy is not doing better than it is, a minor reason, or not a reason at all:
8) “Taxes are too high”
9) “Foreign aid spending is too high"
10) “Top executives are paid too much”
11) How much does the US spend on foreign aid, as a share of GDP (0-1 %, 1-3 %, 3+ %).
I need perhaps 10-15 additional questions. The questions need to have the following features:
1) The answer needs to be provable. Hence why ask what expert scientists believe about P – on which there are surveys I can point to – rather than P itself in many of the questions. However, you can also have questions about P itself if you can point to reliable sources such as government statistics, as I do in question 11.
2) They should be “baits” for biased people; i.e. such that biased people should be expected to give the wrong answer if they don’t like the true answer, and the true answer if they like it.
3) The questions shouldn’t be very difficult. If you give people questions on, e.g, numbers, you have to give fairly large intervals, as I do in question 11. Also you cannot ask too outlandish questions (e.g., questions on small parts of the federal budget).
At present I seem to have more questions where the liberal answer is the true one, so “pro-conservative” questions are particularly welcome.
Any suggestions of questions or other forms of input is highly appreciated! :)
I would rather rephrase "X is too high" as "X should be reduced" if that's what you want to ask. Otherwise it seems to shift the perspective from policy-making to emotional evaluation.
I see no reason to bundle those claims.
True. I'll split them. Thanks!
How about a questions about the average IQ in some subsaharan country?
Hm, good idea. Could be very controversial, though, and I'm not sure of whether it would be sufficiently provable. But yes, a question where the true answer is some negative fact about an African country is a good idea. Thanks!
If you don't want to go into the IQ area, personal values are a good topic.
The World Value Survey seems a good source.
In some African countries more Muslims believe that homosexuality should be punishable by death than most Western liberals would like.
A number of the above questions are not asking, "Is X true?" but rather "Do group Y believe that X is true?"
But once you get into asking "Do group Y believe that X should be done?" you're not talking about respondents' model of others' factual "is" beliefs, but respondents' model of others' moral "ought" beliefs.
That might be a very different thing.
Excellent! Yes those sorts of questions are even better.
Without knowing anything about the extinction of species, I could guess that the answer is "most scientists agree".
If the correct number is not 100 but is large, the question would incorrectly conclude that some people who are not biased are biased (since someone who falsely thinks the number is 100 when it's really 75 or 125 is not biased, but would incorrectly answer "scientists agree").
If the correct number is not 100 but is small, the question would incorrectly conclude that some people who are biased are not biased (since someone who falsely thinks the number is 75 or 125 when it's really 1 is biased, but this bias would be undetectable since he would correctly answer the question with "scientists disagree")
Therefore the correct number is 100.
This question should be phrased using words like "many", not using the exact number 100.
For pro-conservative questions, one could be: In a recent poll of 15000 police officers polled, a large majority thought assault weapon bans are effective, a small majority thought assault weapon bans are effective, they were about equal, a small majority thought assault weapon bans are ineffective, a large majority thought assault weapon bans are ineffective. http://www.policeone.com/Gun-Legislation-Law-Enforcement/articles/6183787-PoliceOnes-Gun-Control-Survey-11-key-lessons-from-officers-perspectives/
Generally, however, people should be suspicious of such questions because in real political discourse, these questions are used to set the terms of the debate.
-- Does it matter whether the death penalty increases homicide rates?
-- Does it matter that humans cause climate change regardless of the size of the change?
-- I would think that any truly expert economist would say "we have no way to know the answers to these questions to the same degree as we know physics or chemistry answers. I could give you my educated opinion, but there's still a lot of disagreement within economics".
Hm, you're right that question 3) is not formulated rightly. Great comment!
Thanks for the poll data. One worry is, though, that the police officers might not be seen as proper "experts" in the same sense as climate scientists are. I need to think about that.
The data are very interesting, though. US police officers seem to very conservative indeed.
Thanks!
The point of the exercise is detecting bias. As such it's not important whether the answer to the question is important.
It doesn't work that way.
Imagine that you don't know much about homeopathy, but you do know that experts oppose it. Then someone asks you the question "The number of homeopathic cures of all types rejected by the FDA for not being effective is (much less than) (less than) (equal to) (greater than) (much greater than) the number of allopathic cancer cures."
If you approached this question out of context, you would think "I know that experts believe homeopathy isn't effective. The FDA uses experts. So experts probably rejected lots of homeopathic remedies."
If you approached this question in context, however, you would reason "I know that experts believe homeopathy isn't effective. But given the way this question is phrased, it's being asked by a homeopath. He's probably asking this question because it makes homeopathy look good, so this must be an unusual situation where experts' belief on homeopathy doesn't affect the answer, and he's falsely trying to imply that it does. So the FDA probably rejected few homeopathic remedies for being ineffective, but for some reason this doesn't reflect the belief of experts."
For instance, if most homeopathic treatments are not submitted to the FDA, they would not have a chance to reject them.
Actually, one of the sponsors of the act that created the FDA was a homeopath and he wrote in an exception for homeopathy, so homeopathic treatments don't have to prove they are safe and effective.
Also, keep note of who this question would falsely mark as biased. Someone who opposes homeopathy and correctly knows that experts also oppose homeopathy, who tries to reason the first way, would be marked down as biased, because he answered in a way favorable to his own position but contrary to the facts. Yet answering the first way doesn't mean bias, it just means he ignored the agenda of the person asking the question.
What do economists think of the American Reinvestment and Recovery Act of 2009?
Or, the following based on http://ew-econ.typepad.fr/articleAEAsurvey.pdf. (I've bolded the answers I think are supported, but you should check my work!)
Great!
On your question 1, I would rephrase it to say that human activities tend to cause global temperatures to rise. Or that human activities have caused global temperatures to rise. Otherwise, you get stuck in the whole issue about the "pause," which might show that temperatures are not currently rising for reasons that are not fully understood and are subject to much debate. The paper you cite was from early 2010, and was based on research before that, so the pause had not become much-discussed by then.
One thing that I think will be interesting if you run the quiz is to identify a group who resist polarization and are between the extremes. For example, I think there are plenty of people that agree that carbon dioxide causes temperatures to rise (all else being equal) but believe that the feedback loop is not significantly positive. People from each extreme tend to lump them middle-grounders in with the people at the other extreme: "You're an alarmist!" "You're a denier!" etc.
Good point - I'll change the formulation.
For the true/false(/divided) questions, it'd be wise to aim for an even split in true/false(/divided) answers to minimize acquiescence bias. At the moment disproportionately few answers for questions 1-7 are "false", so someone who just likes agreeing with things has an unfair advantage there!
Haha! Good point!
You could have a question about the scientific consensus on whether abortion can cause breast cancer (to catch biased pro-lifers). For bias on the other side, perhaps there is some human characteristic the fetus develops earlier than the average uninformed pro-choicer would guess? There seems to be no consensus on fetus pain, but maybe some uncontroversial-yet-surprising fact about nervous system development? I couldn't find anything too surprising on a quick Wiki read, but maybe there is something.
I would expect that even as a fairly squishy pro-abortion Westerner (incredibly discomforted with the procedure but even more discomforted by the actions necessary to ban it), I'm likely to underestimate the health risks of even contragestives, and significantly underestimate the health risks of abortion procedures. Discussion in these circles also overstates the effectiveness of conventional contraception and often underestimates the number of abortions performed yearly. The last number is probably the easiest to support through evidence, although I'd weakly expect it to 'fool' smaller numbers of people than qualitative assessments.
I'm also pretty sure that most pro-choice individuals drastically overestimate its support by women in general -- this may not be what you're looking for, but the intervals (40% real versus 20% expected for women who identify as "pro-life") are large enough that they should show up pretty clearly.
These are good ideas. You've got it quite right - these are exactly the kinds of questions I'm looking for. Possibly the health risks questions are the best ones - I'll see what evidence I can find on those issues. Thanks!
It wouldn't surprise me if people generally overestimate the safety and effectiveness of drugs and medical procedures-- would you want to compare the accuracy of people's evaluation of contraceptives and abortions to their evaluation of medicine in general?
It also wouldn't surprise me if there's a minority who drasitically underestimate the safety and effectiveness of medicine.
If you want to catch the other side on the global warming debate as well, there are a bunch of claims where I suspect the average liberals is overconfident. Maybe something like "Hurricane X wouldn't have happened without global warming". The IPCC report shows their confidence for various claims and it's likely something there to catch liberals.
Yes, something like that could probably catch some liberals, that's true.
Not unlikely at all. Try “There would have be many fewer hurricanes in the past 10 years without global warming” instead.
I said something like because I just want to illustrate an idea and not the suggestion in that form. It makes sense to a claim directly from the IPCC report instead of making up your own claim.
Some domestic questions would be nice. Opinions about school choice for example.
Or homeschooling. Possibilities:
"Studies show that home-schooled children score worse on tests related to socialization than conventionally educated children." This is false according to the first paragraph under "Socialization" on en.wikipedia.org/wiki/Homeschooling in that always true resource, Wikipedia.
"The most cited reason for parents to choose homeschooling over public schools is the public schools' (a) the lack of religious or moral instruction, (b) social environment, or (c) quality of instruction." The actual answer is (b), with (a) taking second place and (c) taking third. See http://nces.ed.gov/pubs2006/homeschool/parentsreasons.asp.
I was homeschooled and hated every minute of it! But I think it can be alright in a few cases. I came out pretty good.
Did you also attend public school? If so, which did you dislike more? If you didn't, which do you think you would have disliked more?
I'm also curious if you don't mind me asking: what did you hate about it?
I went to private schools, a montessori and a private christian school. I hated the isolation, the lack of intellectual curiosity and my browbeating perfectionist mother who never let me learn and just expected perfection at all times in all subjects. It led to a lot of abuse in my family, especially being an only child.