Great job, you two! Don't forget to give your elephant and rider some time to "discuss" the findings internally before making the final judgment. I find that my elephant will slowly come around unless there's something important I've overlooked, which is a major risk when doing explicit calculations. For instance, I notice there's no representation of location, which tends to be a very important factor in deciding where to live.
Vegetarianism Ideological Turing Test Results
Back in August I ran a Caplan Test (or more commonly an "Ideological Turing Test") both on Less Wrong and in my local rationality meetup. The topic was diet, specifically: Vegetarian or Omnivore?
If you're not familiar with Caplan Tests, I suggest reading Palladias' post on the subject or reading Wikipedia. The test I ran was pretty standard; thirteen blurbs were presented to the judges, selected by the toss of a coin to either be from a vegetarian or from an omnivore, and also randomly selected to be genuine or an impostor trying to pass themselves off as the alternative. My main contribution, which I haven't seen in previous tests, was using credence/probability instead of a simple "I think they're X".
I originally chose vegetarianism because I felt like it's an issue which splits our community (and particularly my local community) pretty well. A third of test participants were vegetarians, and according to the 2014 census, only 56% of LWers identify as omnivores.
Before you see the results of the test, please take a moment to say aloud how well you think you can do at predicting whether someone participating in the test was genuine or a fake.
.
.
.
.
.
.
.
.
.
.
.
.
.
If you think you can do better than chance you're probably fooling yourself. If you think you can do significantly better than chance you're almost certainly wrong. Here are some statistics to back that claim up.
I got 53 people to judge the test. 43 were from LessWrong, and 10 were from my local group. Averaging across the entire group, 51.1% of judgments were correct. If my Chi^2 math is correct, the p-value for the null hypothesis is 57% on this data. (Note that this includes people who judged an entry as 50%. If we don't include those folks the success rate drops to 49.4%.)
In retrospect, this seemed rather obvious to me. Vegetarians aren't significantly different from omnivores. Unlike a religion or a political party there aren't many cultural centerpieces to diet. Vegetarian judges did no better than omnivore judges, even when judging vegetarian entries. In other words, in this instance the minority doesn't possess any special powers for detecting other members of the in-group. This test shows null results; the thing that distinguishes vegetarians from omnivores is not familiarity with the other sides' arguments or culture, at least not to the degree that we can distinguish at a glance.
More interesting, in my opinion, than the null results were the results I got on the calibration of the judges. Back when I asked you to say aloud how good you'd be, what did you say? Did the last three paragraphs seem obvious? Would it surprise you to learn that not a single one of the 53 judges held their guesses to a confidence band of 40%-60%? In other words, every single judge thought themselves decently able to discern genuine writing from fakery. The numbers suggest that every single judge was wrong.
(The flip-side to this is, of course, that every entrant to the test won! Congratulations rationalists: signs point to you being able to pass as vegetarians/omnivores when you try, even if you're not in that category. The average credibility of an impostor entry was 59%, while the average credibility of a genuine response was 55%. No impostors got an average credibility below 49%.)
Using the logarithmic scoring rule for the calibration game we can measure the error of the community. The average judge got a score of -543. For comparison, a judge that answered 50% ("I don't know") to all questions would've gotten a score of 0. Only eight judges got a positive score, and only one had a score higher than 100 (consistent with random chance). This is actually one area where Less Wrong should feel good. We're not at all calibrated... but for this test at least, the judges from the website were much better calibrated than my local community (who mostly just lurk). If we separate the two groups we see that the average score for my community was -949, while LW had an average of -448. Given that I restricted the choices to multiples of 10, a random selection of credences gives an average score of -921.
In short, the LW community didn't prove to be any better at discerning fact from fiction, but it was significantly less overconfident. More de-biasing needs to be done, however! The next time you think of a probability to reflect your credence, ask yourself "Is this the sort of thing that anyone would know? Is this the sort of thing I would know?" That answer will probably be "no" a lot more than it feels like from the inside.
Full data (minus contact info) can be found here.
Those of you who submitted a piece of writing that I used, or who judged the test and left their contact information: I will be sending out personal scores very soon (probably by this weekend). Deep apologies regarding the delay on this post. I had a vacation in late August and it threw off my attention to this project.
EDIT: Here's a histogram of the identification accuracy.

EDIT 2: For reference, here are the entries that were judged.
I found these very difficult (with one exception -- one of the entries was mine). Perhaps I'm just bad at spotting the relevant cues, but I think actually vegetarianism and non-vegetarianism are probably easier to fake than (e.g.) liberalism/conservatism or theism/atheism. Being a vegetarian isn't a matter of all-encompassing worldview; you can turn an omnivore into a vegetarian by adjusting one or two opinions (e.g. on what sorts of diet are healthiest) or preferences (e.g., does meat taste delicious or gross) or values (e.g., is a chicken's suffering 0%, 1% or 50% as important as a human's), and it's probably not super-difficult for most thoughtful people to imagine themselves into the opposite position.
I pretty much agree with you. I think it'll be interesting to get the data out of this and see how competent the judges are compared to Leah's Christianity tests. A few people in my local group thought this would be a good topic.
I love the idea though it got hard to keep going with the number of items. I don't think I read page 2 as carefully as page 1.
I didn't include my username but will you be posting the full breakdown of which were and were not real?
Yes. I'll be providing the answer key in the stats post.
Vegetarian/Omnivore Ideological Turing Test Judging Round!
Come one, come all! Test your prediction skills in my Caplan Test (more commonly called an Ideological Turing Test). To read more about such tests, check out palladias' post here.
The Test: http://goo.gl/forms/7f4pQfxB8I
In the test, you will be asked to read responses written by rationalists from LessWrong (and the Columbus Ohio LW group). These responses are either from a vegetarian or omnivore (as decided by a coin flip) and are either their genuine response or a fake response where they pretend to be a member of the other group (also decided by coin flip). If you'd like to participate (and the more, the merrier) you'll be asked to distinguish fake from real by assigning a credence to the proposition that a given response is genuine.
I'll be posting general statistics on how people did at a later date (probably early September). Please use the comments on this thread to discuss or ask questions. Do not make predictions in the comments. I got more entries than would be reasonable to ask people to judge, so if your entry didn't make it into the test, I'm sorry. We might be able to run a second round of judging. If you're interested in judging more entries, send me a PM or leave a comment. I tended to favor the first entries I got, when selecting who got in.
Really, I think it's pretty obvious which side AndreInfante is on. I can PM you if you're interested.
They already did. I encourage you to make your prediction, however (the full judging round will start on Monday or Tuesday depending on my schedule).
For Omnivores:
- Do you think the level of meat consumption in America is healthy for individuals? Do you think it's healthy for the planet?
America has an obesity crisis, but I don't see any reason to think that meat specifically is a major part of it. I'm far more worried about the sugar consumption. If, as part of a general reduction in calorific intake, the meat consumption fell, that would be a good thing, but I worry more about people too poor to afford steak.
Regarding planetary "health" - the damage seems to me to be caused by a lack of property rights and tragedy-of-the-commons situations (see e.g. global warming, fish stock depletion, etc) than meat consumption per se. If reliable property rights could be established, we could all happily consume far more meat than we currently do - and without those reliable property rights, everyone going vegan will make little difference.
- How do you feel about factory farming? Would you pay twice as much money for meat raised in a less efficient (but "more natural") way?
I would (and do) pay a little more for humanely raised meat, but not twice as much. Regarding factory farming - what alternative am I comparing it to? It's worse than happy animals frolicking in the fields, it's better than those animals not existing. As in my judgement, the realistic alternative is much closer to the latter, I am an enthusiast for factory farming, but not in an absolute sense.
- Are there any animals you would (without significantly changing your mind) never say it was okay to hunt/farm and eat? If so, what distinguishes these animals from the animals which are currently being hunted/farmed?
It's context-dependent. If I'm starving on a lifeboat, in extremis I'd eat anything that came to hand - even human! - but I wouldn't resort to cannibalism just out of idle curiosity. There are no animals I'd never eat.
- If all your friends were vegetarians, and you had to go out of your way to find meat in a similar way to how vegans must go out of their way right now, do you think you'd still be an omnivore?
Yes. Meat is delicious and I would seek it out.
For Vegetarians:
- If there was a way to grow meat in a lab that was indistinguishable from normal meat, and the lab-meat had never been connected to a brain, do you expect you would eat it? Why/why not?
I'd need to know more about the way the meat was grown, and why, but no, I don't expect I would eat it. Partly because the fact that this product is classified as "meat" tells me a great deal about the people it's aimed at, and the sensibilities of the corporation behind it. Partly because this kind of artificial creation is highly suspect. But mostly because it's so unnecessary - when you can live a happy and healthy lifestyle without having any truck with meat, why would you want to go down that route?
- Indigenous hunter gatherers across the world get around 30 percent of their annual calories from meat. Chimpanzees, our closest non-human relatives, eat meat. There are arguments that humans evolved to eat meat and that it's natural to do so. Would you disagree? Elaborate.
Humans didn't evolve "to" do anything. That's the naturalistic fallacy. We just evolved. And sure, we have certain adaptations that make it easier for us to eat meat, but we have other adaptations (the shrivelled appendix, for example) that make it harder for us to do so. Unlike most other omnivores, humans can't safely eat raw meat - presumably because we have adapted to early technology such as fire. That makes it very misleading to talk about what's "natural" for humans, because there is no natural human state apart from our technology. But sure, there is a sense in which it's "natural" for humans to eat meat - the same sense in which it's natural for humans to murder our stepchildren, which chimpanzees and most other mammals do. The word I prefer is "barbaric."
- Do you think it's any of your business what other people eat? Have you ever tried (more than just suggesting it or leading by example) to get someone to become a vegetarian or vegan?
As long as governments are pumping billions into agricultural subsidies, as long as corporations are distorting the democratic process and subverting the First Amendment with ag-gag laws, as long as chemicals and pollutants are flooding our rivers, as long as innocent animals are being tortured so you can eat a roast, then of course it's everyone's business. What you put into your mouth may seem like a private decision, but your ability to eat meat rests on a massive industry and supply chain that affects all of us, whether we like it or not. Externalities matter.
Yes, I have encouraged vegetarianism. I have volunteered for Viva in the past. However, there is a time and a place for such conversations. It's better to build trust first, rather than hectoring strangers.
- What do you think is the primary health risk of eating meat (if any)?
The biggest risk is heart disease and related cardiovascular problems. However, the increased cancer risk is almost as bad.
I really like this entry. Don't forget to PM me your actual opinion so I can give feedback to the judges and see how you do. ^_^
For Omnivores:
- Do you think the level of meat consumption in America is healthy for individuals? Do you think it's healthy for the planet?
Meat is obviously healthy for individuals. We evolved to eat as much of it as we could get. Many nutrients seem to be very difficult to obtain in sufficient, bio-available form from an all-vegetable diet. I just suspect most observant vegans are substantially malnourished.
On the planet side of things, meat is an environmental disaster. The methane emissions are horrifying, as is the destruction of rainforest. Hopefully, lab-grown meat allows us to switch to an eco-friendly alternative.
- How do you feel about factory farming? Would you pay twice as much money for meat raised in a less efficient (but "more natural") way?
Factory farming is necessary to continue to feed the world. I don't care about "natural", but I'd pay extra for food from animals that had been genetically engineered to be happy and extremely stupid/near-comatose, to reduce total suffering-per-calorie. This would be more effective and less costly than switching to free-range.
- Are there any animals you would (without significantly changing your mind) never say it was okay to hunt/farm and eat? If so, what distinguishes these animals from the animals which are currently being hunted/farmed?
Great apes. cetaceans, and a few birds. The range of animal intelligence is extremely broad. I find it extremely unlikely that chickens have anything recognizable as a human-like perception of the world. I think the odds are better than not that dolphins, chimps, and parrots do.
If you're interested, the animal I'm most on the fence about is pigs.
- If all your friends were vegetarians, and you had to go out of your way to find meat in a similar way to how vegans must go out of their way right now, do you think you'd still be an omnivore?
Yes. I cook most of my own meals, and my meat consumption would continue even in the absence of social eating.
For Vegetarians:
- If there was a way to grow meat in a lab that was indistinguishable from normal meat, and the lab-meat had never been connected to a brain, do you expect you would eat it? Why/why not?
I obviously no moral problem with that. That would be fantastic. However, I probably wouldn't eat the lab meat. I find the texture / mouth-feel of most meat pretty gross, and lab-grown meat would be significantly more expensive than my current diet. Since microbiome acclimation means that resuming eating meat could make me very sick for a while, I'm not sure I see the profit in it.
I am very interested in synthetic milk, cheese, and eggs, however.
- Indigenous hunter gatherers across the world get around 30 percent of their annual calories from meat. Chimpanzees, our closest non-human relatives, eat meat. There are arguments that humans evolved to eat meat and that it's natural to do so. Would you disagree? Elaborate.
Obviously, humans evolved to be omnivorous. However, the paleo people are lunatics if they think we ate as much meat as they do (much less of the hyper-fatty livestock we've bred over the last couple of millenia). Meat was a most likely a rare supplement to the largely-vegetarian diets of ancestral peoples.
Regardless, none of this is the point. Today, it's perfectly possible to eat a vegan diet and be healthy (see: Soylent). You can't avoid the obvious moral horror of eating the flesh of semi-sentient animals like pigs by shouting the word 'natural' and running away.
- Do you think it's any of your business what other people eat? Have you ever tried (more than just suggesting it or leading by example) to get someone to become a vegetarian or vegan?
Only if they bring it up first. I do think we have a moral obligation to try to reduce animal suffering, but harassing my friends isn't actually helping the cause in any way, and might be hurting. I do try to corrupt my meat-eating friends who are having seconds thoughts about it, but, you know, in a friendly way.
- What do you think is the primary health risk of eating meat (if any)?
Parasites probably. Meat in moderation clearly isn't especially bad for you. It's just, you know, wrong.
Yikes. If all responses are this good, I'm sure the judges will have a rough time! Thanks so much for your words. At some point you'll need to PM me with a description of your actual beliefs so I can give feedback to the judges and see how you do.
Vegetarianism Ideological Turing Test!
I'm running an Ideological Turing Test (or Caplan Test) in my local rationality group on the topic of vegetarianism. (Based on a survey, it's one that splits my community pretty evenly.) If anyone here is interested, you're welcome to participate! I'll be posting the responses I get on LW for judging, and I'm hoping to get responses from a couple people here that I could use for my local group. After I get responses and the community judges them, I'll post here to share the statistics.
You can PM me or rot13 your entry if you're concerned about information leakage, but I'll also accept plaintext comments. The (soft) deadline for submission is the evening of the 15th. If I don't have enough responses by then (for LW or my local group) I'll extend it.
If you're interested in participating, please read on:
First, please write a paragraph or two about what your general position is on vegetarianism. Please make it clear which way you lean for the purposes of answering/judging. This text will be public knowledge (used for the reveal) so include your name if you want to be known, and keep it anonymous if you don't.
Once you've described yourself, please write a paragraph (or two) to briefly answer each of the following questions. If you do not identify as an omnivore, answer the omnivore questions by pretending to be an (aspiring rationalist) omnivore. If you are not a vegetarian, answer the vegetarian questions by pretending to be an (aspiring rationality) vegetarian. When writing responses to the prompts, do not include information that makes it clear who you are (for example: I shouldn't say "Because I am very tall....")
For Omnivores:
* Do you think the level of meat consumption in America is healthy for individuals? Do you think it's healthy for the planet?
* How do you feel about factory farming? Would you pay twice as much money for meat raised in a less efficient (but "more natural") way?
* Are there any animals you would (without significantly changing your mind) never say it was okay to hunt/farm and eat? If so, what distinguishes these animals from the animals which are currently being hunted/farmed?
* If all your friends were vegetarians, and you had to go out of your way to find meat in a similar way to how vegans must go out of their way right now, do you think you'd still be an omnivore?
For Vegetarians:
* If there was a way to grow meat in a lab that was indistinguishable from normal meat, and the lab-meat had never been connected to a brain, do you expect you would eat it? Why/why not?
* Indigenous hunter gatherers across the world get around 30 percent of their annual calories from meat. Chimpanzees, our closest non-human relatives, eat meat. There are arguments that humans evolved to eat meat and that it's natural to do so. Would you disagree? Elaborate.
* Do you think it's any of your business what other people eat? Have you ever tried (more than just suggesting it or leading by example) to get someone to become a vegetarian or vegan?
* What do you think is the primary health risk of eating meat (if any)?
Subscribe to RSS Feed
= f037147d6e6c911a85753b9abdedda8d)
Possible confounding variable: people who don't think they can tell the difference might be less likely to do the test. I remember Brienne doing something similar on facebook. I read a couple of entries, spent a while agonizing about whether I thought they were slightly more likely to be genuine or fake, got bored/frustrated and gave up.
This is a very good point, and I ought to have mentioned it in the post. The point remains about overconfidence, however. Those who did decide to try (even given that it was hard) didn't have the mental red-flag that perhaps their best try should be saying "I don't know" with or without walking away.