The Ideological Turing Test

35 Post author: ciphergoth 25 June 2011 10:17PM

Bryan Caplan:

Mill states it well: "He who knows only his own side of the case knows little of that."  If someone can correctly explain a position but continue to disagree with it, that position is less likely to be correct.  And if ability to correctly explain a position leads almost automatically to agreement with it, that position is more likely to be correct.

[...] Put me and five random liberal social science Ph.D.s in a chat room.  Let liberal readers ask questions for an hour, then vote on who isn't really a liberal.  Then put Krugman and five random libertarian social science Ph.D.s in a chat room.  Let libertarian readers ask questions for an hour, then vote on who isn't really a libertarian.  Simple as that.

My challenge: Nail down the logistics, and I'll happily bet money that I fool more voters than Krugman.

Leah at Unequally Yoked:

Just like Caplan, I'd like to put my money where my mouth is and play in an ideological Turing Test against a Christian blogger.

UPDATE: Two Christians have contacted me to tell me they're interested.  Please suggest format ideas for us to talk over and let me know if you'd like to join in!

update to clarify: When the panel is made up of mostly genuine Blues and one or a few Greens pretending to be Blue, then the judges are all Blue.

Comments (29)

Comment author: CronoDAS 26 June 2011 07:31:20PM 13 points [-]

A while back, David Brin wrote up a proposal for "disputation arenas".

An interesting part of the competition/argument process:

Phase Four: The Paraphrasing Challenge

Here is a unique, but crucial phase. After both manifestos are declared logically usable, a distinct period -- say a month -- will be given to each side so they may paraphrase the other side's position.

This step aims to ensure that each party has actually read and understood where the other one stands, so they aren't simply shouting past each other at chimeric caricatures. Paraphrasing is hard to do when you've spent years demonizing the opposition, calling them venal or stupid and dismissing their concerns. Success at paraphrasing will be seen as a way of winning credibility. It means "I do understand my opponents... so my disagreement with them is well-informed."

Each side may lace their formal paraphrasing with asterisks and footnotes asserting that the statements they are describing are idiotic. That's fine. But if they fail to depict their opponents' point of view in a manner that the jury finds at least generally accurate, they will be disqualified.

Comment author: [deleted] 26 June 2011 06:43:13AM *  10 points [-]

Indeed, I'll happily bet that any libertarian with a Ph.D. from a top-10 social science program can fool more voters than Krugman. We learn his worldview as part of the curriculum. He learns ours in his spare time - if he chooses to spare it.

If we take this as a test, It should be noted that it seems to be biased against popular/conventional/establishment/dominant schools of thought. Historically speaking religious minorities often know a lot about the dominant religion, though the slant may be biased.

However depending on which other tests we use in our investigation this might be a feature not a bug.

Comment author: sixes_and_sevens 25 June 2011 11:32:02PM 10 points [-]

This seems like a very fun game, but I'm not sure how much merit it has as a method of determining which side in a debate is the most informed.

Much like the actual Turing Test, it also seems vulnerable to hard-ball inquisitorial tactics. There are probably quite a few questions you could ask which lie outside the ostensible scope of the debate, but which would correlate strongly with a user's true beliefs, or require faculties the respondent simply doesn't have in order to answer satisfactorily. I'm pretty sure if I asked a fake-atheist to name ten atheist-related figures that inspired them, they'd come up with a predictable, odd, or obviously Googled list. Equally, if I was posing as a Christian I'd struggle to name ten inspiring Bible passages that weren't either predictable, odd, or obviously pulled off Google.

Comment author: Desrtopa 26 June 2011 12:20:40AM 10 points [-]

I'm pretty sure if I asked a fake-atheist to name ten atheist-related figures that inspired them, they'd come up with a predictable, odd, or obviously Googled list.

I've been an atheist for most of my life, and I'm pretty sure I don't have that many atheism-related figures who've inspired me. I'd have a hard time coming up with a list even if I interpreted the question as broadly as possible.

Comment author: sixes_and_sevens 26 June 2011 12:26:44AM 2 points [-]

What if it was just "name ten atheists who you find inspiring"?

Comment author: Desrtopa 26 June 2011 12:29:16AM 11 points [-]

I'd probably have to do some googling to make sure the list really is composed solely of atheists.

Comment author: Emile 26 June 2011 09:03:47AM 1 point [-]

Same here. So if I had to pretend to be a Christian, I'd waffle around a bit just like I do if you ask me that about atheists.

Comment author: ciphergoth 26 June 2011 09:31:59AM 3 points [-]

Yes, I think this is the biggest drawback with the proposal. Lots of us understand pretty well the arguments that Christians make for their position, but we wouldn't be able to produce the shibboleths that Christians would look for.

Comment author: Kaj_Sotala 28 June 2011 09:14:20AM 1 point [-]

You could just forbid tactics violating the spirit of the test, such as these ones.

Comment author: ciphergoth 26 June 2011 12:01:12PM *  6 points [-]

I think there are several big problems with this that make it a lot less useful than you might hope as a way of telling which side better understands the other side's arguments - leaving aside the big problem with picking a "side" in an argument. Nonetheless, I think it's likely to be informative and worth a go!

As stated, it sounds like everyone is expected to be on the same IRC channel. That doesn't seem like such a good idea - the imitators can for the most part simply listen to the genuines to work out what to say. What you really want is a way to ask the panel a question and have their answers revealed only once they have all been submitted. However, I can't immediately think of an existing software platform that makes this convenient.

I don't know how best to minimise the shibboleth problem. At the very least, we could remind panellists and judges that shibboleths may be more easily imitated than real winning arguments. Bear in mind that the judges are effectively all Blues pretending to be Greens, in that they want to test how well the panel understand the arguments against Green positions.

Comment author: Manfred 25 June 2011 11:29:41PM 6 points [-]

Troublesome because you have an uncontrolled variable - the investigative ability of the people you're trying to fool.

Comment author: rhollerith_dot_com 25 June 2011 11:53:49PM 0 points [-]

That's why you make it so that there are a lot of them.

Comment author: Manfred 26 June 2011 12:10:38AM *  4 points [-]

Not noise - I really mean a variable. For example, say you want to tax cigarettes, and someone else wants to ban cigarettes. To find out who knows the other person's position better you trade places and try to fool some large group of their peers. The taxer chats with some banners for a while, and the banner chats with some taxers for a while. Then the two groups write down how knowledgeable they thought people were.

But what if taxers tend to be introverts, and banners tend to be police officers who are masters at interrogation? Or, more plausibly, what if there are group-specific things that will impact peoples' judgement of your knowledge other than the substance of the issue? There are all sorts of things that could cause groups to grade transplants differently, and these factors are left uncontrolled if you just do the swap as suggested.

Comment author: rhollerith_dot_com 26 June 2011 12:20:54AM 1 point [-]

Ah, OK then. (Selection bias.)

Comment author: Bo102010 26 June 2011 03:01:31AM 3 points [-]

I asked Leah about participating in her test. I think I will find it challenging the emulate a Christian's viewpoint, instead of mocking it.

That is, normally when I've examined Christian thought, I've approached it from the pretense "let me see where this fails."

I'm interested to see if I can construct an argument that I myself as an atheist would find at least superficially compelling - that is, I want to avoid doing what pretty much every popular Christianity book does.

Comment author: ciphergoth 26 June 2011 01:05:15PM 6 points [-]

If you try to present radically better arguments for Christianity than you think most Christians present, you will sound out of step with the rest of the panel to an audience of Christian judges.

Comment author: jfm 27 June 2011 06:39:15PM 1 point [-]

I think this is correct. If you want to successfully pose as a Christian, you might be well advised to read a bunch of C. S. Lewis, and then imitate his arguments and style. I say this because I think his books constitute the most accessible body of reasonable-but-still-wrong arguments in favour of Christian orthodoxy. If you can quote him, all the better, because being able to quote C. S. Lewis is a high-status marker among people who have both a self-identity as Christians, and a self-identity as intellectuals.

Comment author: Eugine_Nier 25 June 2011 11:34:47PM *  6 points [-]

I suspect the results would depend heavily on the jury. Specifically a blue pretending to be a green, could probably fool fellow blues much more easily then he could fool actual greens.

Comment author: ciphergoth 26 June 2011 09:32:59AM 11 points [-]

In this proposal, whenever a Blue is pretending to be a Green he is trying to fool Green judges.

Comment author: CronoDAS 25 June 2011 11:35:24PM *  9 points [-]

"Things that try to look like things often look more like things than things. Well known fact."

—Granny Weatherwax, Wyrd Sisters (by Terry Pratchett)

In other words, you win this game not by trying to imitate actual people who hold a position, but by imitating the stereotype of people that hold that position.

Comment author: ciphergoth 26 June 2011 09:30:10AM *  9 points [-]

If the judges were random disinterested people, maybe, but the judges are people sympathetic to the stated position of the panel; they're less likely to be fooled by someone parroting a stereotype of themselves. I think that to cryonicists, someone trying to fulfill a stereotype of a cryonicists among a panel of genuine cryonicists would stick out like a sore thumb.

Comment author: mutterc 26 June 2011 09:14:41PM 5 points [-]

Depends on the position. Remember Poe's Law - there are some positions where it's impossible to distinguish a parody from the real thing.

Comment author: satt 26 June 2011 04:13:19AM 6 points [-]

There are people who do this for free. They're usually known as Internet trolls.

Comment author: MixedNuts 27 June 2011 08:44:20AM 2 points [-]

No, they can't fool supporters of their fake position (e.g. trolls on Alternet pretending to be atheists are quite different from trolls who are sincere about atheism and trolling about another topic).

Comment author: satt 27 June 2011 11:10:35AM 3 points [-]
Comment author: Bo102010 18 July 2011 01:33:31AM *  2 points [-]

Voting for the Christian / "Christian" half is open - see Leah's site.

I had fun figuring these out. Once it's done I'd be interested to see what criteria LW users used to determine someone's real beliefs...

Comment author: CharlieSheen 26 June 2011 06:45:01AM *  2 points [-]

If its carreid out it should be entertaining. Some comparisons among other ideological constructs would be interesitng as well. I wonder how say someone like Paul Gottfried or John Derbyshire would do immitating left or Neoconservative positions.

Comment author: Zack_M_Davis 09 July 2011 05:35:03PM 1 point [-]

First round (Christians pretending to be atheists, atheists being sincere) voting is open.

Comment author: Emile 27 June 2011 07:31:49AM *  0 points [-]

Sounds like a neat idea! I've been looking for these kinds of "meta" approaches to political discussion, and this looks like a pretty good one I hadn't thought of!

An improvement might be to restrict the discussion to a topic narrower than "libertarians vs. liberals" or "christians versus atheists" - for example "labour laws", "immigration", "the documentary hypothesis", "morality of homsexuality", etc.

This looks like it'd have a much better likelihood-I-change-my-mind / invested-time ratio than an "ordinary" poltical debate.