Back in August I ran a Caplan Test (or more commonly an "Ideological Turing Test") both on Less Wrong and in my local rationality meetup. The topic was diet, specifically: Vegetarian or Omnivore?

If you're not familiar with Caplan Tests, I suggest reading Palladias' post on the subject or reading Wikipedia. The test I ran was pretty standard; thirteen blurbs were presented to the judges, selected by the toss of a coin to either be from a vegetarian or from an omnivore, and also randomly selected to be genuine or an impostor trying to pass themselves off as the alternative. My main contribution, which I haven't seen in previous tests, was using credence/probability instead of a simple "I think they're X".

I originally chose vegetarianism because I felt like it's an issue which splits our community (and particularly my local community) pretty well. A third of test participants were vegetarians, and according to the 2014 census, only 56% of LWers identify as omnivores.

Before you see the results of the test, please take a moment to say aloud how well you think you can do at predicting whether someone participating in the test was genuine or a fake.

.

.

.

.

.

.

.

.

.

.

.

.

.

If you think you can do better than chance you're probably fooling yourself. If you think you can do significantly better than chance you're almost certainly wrong. Here are some statistics to back that claim up.

I got 53 people to judge the test. 43 were from LessWrong, and 10 were from my local group. Averaging across the entire group, 51.1% of judgments were correct. If my Chi^2 math is correct, the p-value for the null hypothesis is 57% on this data. (Note that this includes people who judged an entry as 50%. If we don't include those folks the success rate drops to 49.4%.)

In retrospect, this seemed rather obvious to me. Vegetarians aren't significantly different from omnivores. Unlike a religion or a political party there aren't many cultural centerpieces to diet. Vegetarian judges did no better than omnivore judges, even when judging vegetarian entries. In other words, in this instance the minority doesn't possess any special powers for detecting other members of the in-group. This test shows null results; the thing that distinguishes vegetarians from omnivores is not familiarity with the other sides' arguments or culture, at least not to the degree that we can distinguish at a glance.

More interesting, in my opinion, than the null results were the results I got on the calibration of the judges. Back when I asked you to say aloud how good you'd be, what did you say? Did the last three paragraphs seem obvious? Would it surprise you to learn that not a single one of the 53 judges held their guesses to a confidence band of 40%-60%? In other words, every single judge thought themselves decently able to discern genuine writing from fakery. The numbers suggest that every single judge was wrong.

(The flip-side to this is, of course, that every entrant to the test won! Congratulations rationalists: signs point to you being able to pass as vegetarians/omnivores when you try, even if you're not in that category. The average credibility of an impostor entry was 59%, while the average credibility of a genuine response was 55%. No impostors got an average credibility below 49%.)

Using the logarithmic scoring rule for the calibration game we can measure the error of the community. The average judge got a score of -543. For comparison, a judge that answered 50% ("I don't know") to all questions would've gotten a score of 0. Only eight judges got a positive score, and only one had a score higher than 100 (consistent with random chance). This is actually one area where Less Wrong should feel good. We're not at all calibrated... but for this test at least, the judges from the website were much better calibrated than my local community (who mostly just lurk). If we separate the two groups we see that the average score for my community was -949, while LW had an average of -448. Given that I restricted the choices to multiples of 10, a random selection of credences gives an average score of -921.

In short, the LW community didn't prove to be any better at discerning fact from fiction, but it was significantly less overconfident. More de-biasing needs to be done, however! The next time you think of a probability to reflect your credence, ask yourself "Is this the sort of thing that anyone would know? Is this the sort of thing I would know?" That answer will probably be "no" a lot more than it feels like from the inside.

Full data (minus contact info) can be found here.

Those of you who submitted a piece of writing that I used, or who judged the test and left their contact information: I will be sending out personal scores very soon (probably by this weekend). Deep apologies regarding the delay on this post. I had a vacation in late August and it threw off my attention to this project.

EDIT: Here's a histogram of the identification accuracy. 

EDIT 2: For reference, here are the entries that were judged.

Come one, come all! Test your prediction skills in my Caplan Test (more commonly called an Ideological Turing Test). To read more about such tests, check out palladias' post here.

The Test: http://goo.gl/forms/7f4pQfxB8I

In the test, you will be asked to read responses written by rationalists from LessWrong (and the Columbus Ohio LW group). These responses are either from a vegetarian or omnivore (as decided by a coin flip) and are either their genuine response or a fake response where they pretend to be a member of the other group (also decided by coin flip). If you'd like to participate (and the more, the merrier) you'll be asked to distinguish fake from real by assigning a credence to the proposition that a given response is genuine.

I'll be posting general statistics on how people did at a later date (probably early September). Please use the comments on this thread to discuss or ask questions. Do not make predictions in the comments. I got more entries than would be reasonable to ask people to judge, so if your entry didn't make it into the test, I'm sorry. We might be able to run a second round of judging. If you're interested in judging more entries, send me a PM or leave a comment. I tended to favor the first entries I got, when selecting who got in.

I'm running an Ideological Turing Test (or Caplan Test) in my local rationality group on the topic of vegetarianism. (Based on a survey, it's one that splits my community pretty evenly.) If anyone here is interested, you're welcome to participate! I'll be posting the responses I get on LW for judging, and I'm hoping to get responses from a couple people here that I could use for my local group. After I get responses and the community judges them, I'll post here to share the statistics.

You can PM me or rot13 your entry if you're concerned about information leakage, but I'll also accept plaintext comments. The (soft) deadline for submission is the evening of the 15th. If I don't have enough responses by then (for LW or my local group) I'll extend it.

If you're interested in participating, please read on:

First, please write a paragraph or two about what your general position is on vegetarianism. Please make it clear which way you lean for the purposes of answering/judging. This text will be public knowledge (used for the reveal) so include your name if you want to be known, and keep it anonymous if you don't.

Once you've described yourself, please write a paragraph (or two) to briefly answer each of the following questions. If you do not identify as an omnivore, answer the omnivore questions by pretending to be an (aspiring rationalist) omnivore. If you are not a vegetarian, answer the vegetarian questions by pretending to be an (aspiring rationality) vegetarian. When writing responses to the prompts, do not include information that makes it clear who you are (for example: I shouldn't say "Because I am very tall....")

For Omnivores:

 * Do you think the level of meat consumption in America is healthy for individuals? Do you think it's healthy for the planet?

 * How do you feel about factory farming? Would you pay twice as much money for meat raised in a less efficient (but "more natural") way?

 * Are there any animals you would (without significantly changing your mind) never say it was okay to hunt/farm and eat? If so, what distinguishes these animals from the animals which are currently being hunted/farmed?

 * If all your friends were vegetarians, and you had to go out of your way to find meat in a similar way to how vegans must go out of their way right now, do you think you'd still be an omnivore?

For Vegetarians:

 * If there was a way to grow meat in a lab that was indistinguishable from normal meat, and the lab-meat had never been connected to a brain, do you expect you would eat it? Why/why not?

 * Indigenous hunter gatherers across the world get around 30 percent of their annual calories from meat. Chimpanzees, our closest non-human relatives, eat meat. There are arguments that humans evolved to eat meat and that it's natural to do so. Would you disagree? Elaborate.

 * Do you think it's any of your business what other people eat? Have you ever tried (more than just suggesting it or leading by example) to get someone to become a vegetarian or vegan?

 * What do you think is the primary health risk of eating meat (if any)?

Hello! I'm running an Ideological Turing Test for my local rationality group, and I'm wondering what ideology to use (and what prompts to use for that ideology). Palladias has previously run a number of tests on Christianity, but ideally I'd find something that was a good 50/50 split for my community, and I don't expect to find many Christians in my local group. The original test was proposed for politics, which seems like a reasonable first-guess, but I also worry that my group has too many liberals and not enough conservatives to make that work well.

What I plan to do is email the participants who have agreed to write entries asking how they stand on a number of issues (politics, religion, etc) and then use the issue that is most divisive within the population. To do that, however, I'll need a number of possible issues. Do any of you have good ideas for ITT domains other than religion or politics, particularly for rationalists?

(Side questions:

I've been leaning towards using the name "Caplan Test" instead of "Ideological Turing Test". I think the current name is too unwieldy and gives the wrong impression. Does the ITT name seem worth keeping?

Also, would anyone on here be interested in submitting entries to my test and/or seeing results?)