What decision rests on the outcome of your test?

I think there's a problem with your thinking on this - people can spot patterns of good and bad reasoning. Depending on the argument, they may or may not notice a flaw in the reasoning for a wide variety of reasons. Someone who is pretty smart probably notices the most common fallacies naturally - they could probably spot at least a few while watching the news or listening to talk shows.

People who study philosophy are going to have been exposed to many more diverse examples of poor reasoning, and will have had practice identifying weak points and exploiting them to attack an argument. This increases your overall ability to dissolve or decompose arguments by increasing your exposure and by equipping you with a trick bag of heuristics. People who argue on well moderated forums or take part in discussions on a regular basis will likely also pick up some tricks of this sort.

However, there are going to be people who can dissolve one problem, but not another because they have been exposed to something sufficiently similar to one (and are thus probably have some cached details relevant to solving it) but not so for the other:

E.g. a student of logic will probably make the correct choice in the Wason Selection Task and may be able to avoid making the conjunction fallacy, but they may not two box because they fall into the CDT reasoning trap. However, a student of the sciences or statistics may slip up in the selection task but one box, by following the EDT logic.

So if you're using this approach as an intelligence test, I'd worry about committing the fundamental attribution error pretty often. However, I doubt you're carrying out this test in isolation. In practice, it probably is reasonable to engage people you know or meet in challenging discussions if you're looking for people that are sharp and enjoy that sort of thing. I do it every time I meet someone who seems like they might have some inclination toward that sort of thing.

It might help if you provide some context though - who are you asking and how do you know them? Are you accosting strangers with tricky problems or are you probing acquaintances and friends?

If there is no consensus, how do you know what answer is "right"? Surely if it was a simple matter of computation or logic, there would be a consensus.

I'd settle for a well-thought-out answer, even if it's not the one I agree with.

Yes, that's exactly what I mean. The argument itself is terrible. But it invites so many reasonable challenges that it is still very useful as a test of clear thinking. So, awful argument; great test case.

On a related note, I remember the day when I found out my PhD advisor (a computability theorist!) revealed that he believed the argument against AI from Gödel's incompleteness theorem. It was not reassuring.

I'm not sure I understand the Library of Congress bit, but the footnote is exactly right. Even so, that is only one way of resisting Searle's argument. The point for me is that we can measure cleverness to some tolerance by how many ways one finds to fault the argument. For example:

a. The architecture is completely wrong. People don't work by simple look-up tables.

b. Failure of imagination. We are asked to imagine something that passes the Turing test. Anyone convinced by the argument is probably not imagining that premiss vividly enough.

c. The argument depends on a fallacy of division/composition. Searle argues that the system does not understand Chinese since none of its parts understand Chinese. But some humans understand Chinese, and it is implausible that any individual human cell understands Chinese. So, the argument is logically flawed.

d. In order to have an interactive conversation, the room needs to have something like a memory or history. Understanding isn't just about translation but about connecting language to other parts of life.

e. Similarly to (d), the room is not embodied in any interesting way. The room has no perceptual apparatus and no motor functions. Understanding is partly about connecting language to the world. Intelligence is partly about successful navigation in the world. Connect the room to a robot body and then present the case again.

...

Further challenges could be given, I think. But you get the idea.

I bestow a higher likelihood of long-term closeness on persons who "avoid just icking away from the subject."

Oh, I apologize. I entirely misread what you were doing, I think.

I sorta think you can't possibly disagree with this, or you wouldn't be here.

Um... kind of? I guess it depends on what sort of contrarian opinions you were sharing and what sort of setting you were doing it in.

The latter part assumed you were mainly replying to the second question I asked. I apologize for the bluntness of those questions, also. However, I would like to clarify my first question slightly.

When I see the phrase "skill at philosophy" it makes me think of professional philosophers. You probably are not trying to test for the kinds of skills which are found in professional philosophers, because most of these skills cannot be tested through informal questioning. I now realize that you were trying to test for, I think, the ability to think logically about philosophical topics and openness to unpopular ideas. Sorry for the misinterpretation.

Beyond the obvious signaling opportunity of saying that creationists are the worst people ever, I'm not having an easy time figuring out which way the ranking should go between a celebrity who appears to be totally apathetic towards philosophy and a creationist apologist who is enthusiastically doing very bad philosophy.

I also wonder how much agreement there would be if we tried to establish the ranking between Richard Dawkins and Jerry Fodor.