Yeah, I had much the same problem. And since allegedly each person was randomly labelled either "vegetarian" or "omnivore", clearly that indicates that we're failing to tell which is which. I made some (insufficient) attempt to adjust my probability estimates downward a little (effectively, updating on the knowledge that randomized answers usually look credible to me, so that my naive credibility judgements must be overgenerous), so that almost all my estimates were 40%-60%.
I fear that this one (at least with the sample of contestants we've got -- LW posters are an intelligent lot) is just too difficult.
I am prepared to be embarrassed if it turns out that some people were able to identify fakers much better than chance...
Come one, come all! Test your prediction skills in my Caplan Test (more commonly called an Ideological Turing Test). To read more about such tests, check out palladias' post here.
The Test: http://goo.gl/forms/7f4pQfxB8I
In the test, you will be asked to read responses written by rationalists from LessWrong (and the Columbus Ohio LW group). These responses are either from a vegetarian or omnivore (as decided by a coin flip) and are either their genuine response or a fake response where they pretend to be a member of the other group (also decided by coin flip). If you'd like to participate (and the more, the merrier) you'll be asked to distinguish fake from real by assigning a credence to the proposition that a given response is genuine.
I'll be posting general statistics on how people did at a later date (probably early September). Please use the comments on this thread to discuss or ask questions. Do not make predictions in the comments. I got more entries than would be reasonable to ask people to judge, so if your entry didn't make it into the test, I'm sorry. We might be able to run a second round of judging. If you're interested in judging more entries, send me a PM or leave a comment. I tended to favor the first entries I got, when selecting who got in.