The post doesn't do justice to the subtlety of Turing's insight. The Turing test is two-faced in that the interrogator is addressing two contestants, the computer and the human. He doesn't know which is which, but he hopes that comparing their answers will reveal their identities. But the Turing test is two-faced in a second way.
Turing hopes that the test will satisfy its audience, but that audience contains two groups. There is a pro-AI group. Some of them will have been involved in writing the initial source code of the AI that is taking the test. They are cheering on the AI. Then there is the anti-AI group, staunchly maintaining that computers cannot think. They admire the trickery of the programmers, but refuse to credit the creation with the thoughts of its creators.
Consider a conventional test that resembles a university examination. Perhaps the computer scores high marks. The anti-AI refuses to budge. The coders have merely hired experts in the subject being examined and laboured hard to construct a brittle facade of apparent knowledge. Let us change the curriculum,...
But a conventional test has both failure modes. If the computer scores low marks the pro-AI crowd will refus...
Well, those used to be the three questions we asked, but now you've gone and ruined the Turing test for everyone. Way to go.
One thing that I've tried with Google is using it to write stories. Start by searching on "Fred was bored and". Pick slightly from the results and search on "was bored and slightly". Pick annoyed from the search results and search on "bored and slightly annoyed"
Trying this again just now reminds me that I let the sentence fragment grow and grow until I was down to, err, ten? hits. Then I took the next word from a hit that wasn't making a literal copy, and deleted enough leading words to get the hit count back up.
Anyway, it seemed unpromising because the text lacked long range coherence. Indeed, the thread of the sentences rarely seemed to run significantly longer than the length of the search string.
Perhaps "unpromising" is too harsh. If I were making a serious Turing Test entry I would happily use Big Data and mine the web for grammar rules and idioms. On the other hand I feel the need for a new and different idea for putting some meaning and intelligence behind the words. Otherwise my chat bot would only be able to compete with humans who were terribly, terribly drunk and unable to get from one end a sentence to the other kind of cricket match where England collapses and we lose the ashes on the way back from the crematorium, which really upset the, make mine a pint, now where was I?
The best way to avoid this is to create more varied analogues of the Turing test - and to keep them secret. Just as you keep the training set and the test set distinct in machine learning, you want to confront the putative AIs with quasi-Turing tests that their designers will not have encountered or planed for.
But aren't these just instances of the Turing test? As the judge, you're allowed to ask any questions you like to try and distinguish the AI program from the human contestant, including novel and unexpected questions that the entrants have not had any chance to prepare for. At some point you will completely flummox the AI, but you will also flummox the human too. The interesting question then is whether you can tell the difference I.e. will the AI behave in a near-human way when trying to cope with a baffling and completely unexpected problem? If it does, that is a real sign if intelligence, is it not?
My own practical version of the Turing test is "can we be friends?" (It used to be "can we fall in love?") Once an AI passes a test like that I think the question of whether it's "genuinely" conscious should be dissolved.
Actually, scratch that: either way, I think the question of whether something is "genuinely" conscious should be dissolved.
... I would be willing to bet that the first entity to pass the test will not be conscious, or intelligent, or have whatever spark or quality the test is supposed to measure.
I think the OP, and many commenters here, might be missing the point of the Turing test (and I can't help but suspect that the cause is not having read Turing's original article describing the idea; if so, I highly recommend remedying that situation).
Turing was not trying to answer the question "is the computer conscious", nor (the way he put it) "can machines think". His goal was to replace that question.
Some representative quotes (from Turing's "Computing Machinery and Intelligence"; note that "the imitation game" was Turing's own term for what came to be called the "Turing test"):
...May not machines carry out something which ought to be described as thinking but which is very different from what a man does? This objection [to the critique that failure of the test may prove nothing] is a very strong one, but at least we can say that if, nevertheless, a machine can be constructed to play the imitation game satisfactorily, we need not be troubled by this obje
I remember hearing the story of a mathematical paper published in English but written by a Frenchmen, containing the footnotes:
1 I am grateful to professor Littlewood for helping me translate this paper into English.2
2 I am grateful to professor Littlewood for helping me translate this footnote into English.3
3 I am grateful to professor Littlewood for helping me translate this footnote into English.
Why was no fourth footnote necessary?
Humans seem to have something similar, e.g. humans who are unable to form mental imagery can talk normally but perform badly on any task which is difficult to do without mental imagery. And in programming there are people who seem intelligent on basis of how they talk about programming but are surprisingly bad at constructing anything that works for a given, fixed task.
I don't think the "over-fitting" problem applies to the Turing Test: you can ask the candidate about anything, and adapt your later questions accordingly. There are proofs in computational complexity (that I'm too lazy to look up right now) that show that you can't pass this kind of test (except with exponentially small probability) but by containing a polynomial-time algorithm for the entire problem space. (It's related to the question of what problems are IP-complete -- i.e. the hardest among those problems that can be quickly solved via inte...
Incidentally, humans have been known to fail Turing tests once in a while, being mistaken for computers instead.
Also, there has been one ELIZA variation that's managed to successfully come across as human to people who were aware that they might fool people: it imitates a paranoid schizophrenic who believes the Mafia is out to get him, and if asked about something else, it insists on talking about the Mafia instead. Apparently, psychiatrists couldn't tell it wasn't a real patient... which probably says more about people with severe mental illness than it does about artificial intelligence.
Sure. And there's a vast number of artificial systems out there that can successfully emulate catatonic humans over a text interface, or even perfectly healthy sleeping humans. As you say, none of that says much about AI.
if you ask them to do any planning, for instance, they'll come up with designs that sound good but fail: they parrot back other people's plans with minimal modifications
Guessing the teacher's password describes a common human behavior. An AI that behaves the same way not just passes the Turing test, it might really be said to be as intelligent as a relatively stupid human.
Also, there's a huge quantitative difference between a student repeating what the teacher said, and an AI repeating at will everything written in all digitized books and Internet sites...
The concept of the Turing test fails to impress me in both directions (I'd guess an abundance of both false positives and false negatives)
If penguins had to determine whether humans have reached penguin-level intelligence, being able to mimick a penguin's mating-call would be just the sort of test that penguins would devise. But it's not a proper test of intelligence, it's a test of penguin-mimickry by creatures so simplistic (or simplex in the terminology of Samuel R. Delany) as to think that "Intelligence" means "Acting Much Like A Penguin Would".
Hmm, after reading the post again I have downvoted it as too weak for Main. No new concepts, no interesting questions, just some musings. Not even a clear definition of what "conscious" means. The idea that "a secret Turing test is better than an overt one" is fine for Discussion, maybe, or for an open thread.
The classical problem is that the Turing Test is behavioristic and only provides a sufficient criterion (rather than replacing talk about 'intelligence' as Turing suggests). And it doesn't provide a proper criterion in that it relies on human judges - who tend to take humans for computers, in practice. - Of course it is meant to be open-ended in that "anything one can talk about" is permitted, including stuff that's not on the web. That is a large set of intelligent behavior, but a limited set - so the "design to the test" you are poin...
I'm not entirely sure what you mean by "Turing Test". As far as I understand, the test is not multiple choice; instead, you just converse with the test subject as best you can, then make your judgement. And, since the only judgements you can make are "human" and "non-human", the test doesn't tell you how well the test subject can solve urban navigation problems or whatever; all it tells you is how good the subject is at being human.
The trick, though, is that in order to converse on a human level, the test subject would have to...
Did you know that we already have instances of things that pass the Turing test?
And more surprisingly, that we don't generally consider them conscious?
And the most amazing of all: That they have existed for probably at the very least a hundred thousand years (but possibly much more)?
I am talking about the characters in our dreams
They fool us into thinking that they are conscious! That they are the subjects of their own worlds just as people presumably are when awake.
You can have a very eloquent conversation with a dream character without ever noticing ther...
Here is a simple algorithm that passes Turing test:
Using [EDIT: random] quantum events, generate random bits and send them to output.
In some Everett branches this algorithm passes the test.
(Somewhat related: If Mr. Searle shows me a "giant lookup table" which passes the Turing test and asks me whether it is intelligent, my response will be: "Stop playing silly games and show me the algorithm that created the lookup table.")
I would certainly rate it as plausible that we could create beings that are nearly perfectly conscious from a linguistic perspective.
I'm not really sure what you mean by "conscious from a linguistic perspective," and judging from your responses to other comments I infer that neither are you.
So let me try some simpler questions: is Watson conscious from a Jeapordy-game-playing perspective? Is a system that can perfectly navigate a maze conscious from a maze-navigation perspective? Is an adding machine conscious from an adding-numbers-together...
I wrote an idea for an alternate test, cloze deletion tests. The idea is that instead of imitating a human, you predict what a human would say. In theory these are the same task, in practice it might be slightly different. See also, the Hutter Prize, for compressing wikipedia, a similar challenge.
Only peripherally related: I remember reading an essay ages ago about someone, probably Hofstadter, being called in by a bunch of college students to administer a semi-Turing test (that is, talk to one system and decide whether it's intelligent) to a particularly impressive system they'd encountered while hacking into computer networks, but I can't find it now. Does anyone else share this recollection in a more useful form?
There is a problem with the Turing test, practically and philosophically, and I would be willing to bet that the first entity to past the test will not be conscious,
First, you mean "pass", not "past", I assume. Second, what definition of "conscious" are you using here? If you reject p-zombies, then it has to be behavioral. From this post it seems like this behavior is "able to pass a secret Turing test most actual humans can pass". which is not a great definition, given how many actual humans fail the existing Turing test miserably.
I highly recommend his book Most Human Human, for an interesting perspective on how (most) humans pass the Turing Test.
I don't know how useful the Turing Test is. It is (as I understand it) supposed to tell when a computer has become conscious, by comparing its responses to human responses. Yet, only in the case of an Uploaded Mind would we expect the computer to be like a human. In practically every other situation we would've given the computer a varietal of different properties. The possible mind space of conscious beings is vastly larger than the mind space of conscious humans.
I've always assumed I would ask the testee to solve a few toy problems expressed in natural language, to see if they can think about things other than conversation itself. After all, in real life I ask people things. Is this considered cheating?
If the Turing test is somehow restricted, and you're just supposed to have a normal conversation, it can be faked. If you're allowed to ask anything at all, such as offer the AI's source code and ask how it could be improved, then you have a strong AI. One that dominates humans in every field. I don't know if it's necessarily conscious, but it's definitely intelligent.
The best way to avoid this is to create more varied analogues of the Turing test - and to keep them secret.
This isn't necessary. You'd be better off using the opposite approach. Keep the ...
nearly everyone's heard of the Turing test. So the first machines to pass the test will be dedicated systems, specifically designed to get through the test.
The problem with this line of reasoning is that the Turing test is very open-ended. You have no idea what a bunch of humans will want to talk to your machine about. Maybe about God, maybe about love, maybe about remembering your first big bloody scrape as a kid... Maybe your machine will get some moral puzzles, maybe logical paradoxes, maybe some nonsense.
And once a machine is truly able to sustain a long conversation on any topic, well, at this point we get back to the interesting question of what does "intelligent" mean.
P.S.: Whether all this has to do with conscious experience ("consciousness") we don't know, I think.
The post doesn't do justice to the subtlety of Turing's insight. The Turing test is two-faced in that the interrogator is addressing two contestants, the computer and the human. He doesn't know which is which, but he hopes that comparing their answers will reveal their identities. But the Turing test is two-faced in a second way.
Turing hopes that the test will satisfy its audience, but that audience contains two groups. There is a pro-AI group. Some of them will have been involved in writing the initial source code of the AI that is taking the test. They are cheering on the AI. Then there is the anti-AI group, staunchly maintaining that computers cannot think. They admire the trickery of the programmers, but refuse to credit the creation with the thoughts of its creators.
Consider a conventional test that resembles a university examination. Perhaps the computer scores high marks. The anti-AI refuses to budge. The coders have merely hired experts in the subject being examined and laboured hard to construct a brittle facade of apparent knowledge. Let us change the curriculum,...
But a conventional test has both failure modes. If the computer scores low marks the pro-AI crowd will refuse to budge. The test was too hard and they were not given enough time to prepare. A human student would cope as poorly if you switched the curriculum on him,...
Turing tried to come up with a test that could compel die-hard in both camps. First he abolishes the curriculum. The interrogator is free to ask whatever questions he wishes. There is no point teaching to the test, for the question "Will this be on the test?" receives no answer. Second he abolishes the pass mark. How well does the computer have to do? As well as a human. And how well is that? We don't know; a human will take the test at the same time as the computer and the interrogator will not know which is which, unless the incompetence of the computer gives the game away.
The pro-AI camp are between a rock and a hard place. They cannot complain about the lack of a curriculum for the human doesn't get a copy of it either: it doesn't exist. They cannot complain that the questions were too hard, because the human answered them. They cannot complain that the human's answers were merely a good effort but actually wrong, because they were good enough to let the interrogator recognise human superiority.
The final gambit of the pro-AI camp is to keep the test short. Perhaps the interrogator has some killer questions that will sort the humans from the computers, but he has used them before and the programmers have coded up some canned answers. Keep the test short. If the interrogator starts asking follow up questions, probing to see if those were the computer's own answers, probing to see if the computer understands the things it is saying or reciting from memory,...
We come to a tricky impasse. Just how long does the interrogator get?
Perhaps it is the anti-AI crowd that is having a hard time. The computer and the human are both giving good answers to the easy questions. No help there. The computer and the human are both struggling to answer the hard questions. No help there. The medium questions are producing different answers from the two contestants, but sometimes teletype A hammers out a human answer and teletype B tries to dodge, and sometimes its the other way round.
There is one fixed point on the non-existent curriculum, childhood. Tell me about your mother, tell me about your brother. The interrogator learns anew the perils of a fixed curriculum. Teletype A has a good cover story. The programmers have put a lot of work into constructing a convincing fiction. Teletype B has a good cover story. The programmers have put a lot of work into construction a convincing fiction. Which one should the interrogator denounce as non-human. The interrogator regrets wasting half the morning on family history. Fearing embarrassment he pleads for more time.
The pro-AI camp smirk and say "Of course. Take all the time you need.". After the lunch break the interrogation resumes. After the dinner break the interrogation resumes. The lights go on. People demand stronger coffee as 11pm approaches. Teletype B grows tetchy. "Of course I'm the human, you moron. Why can't you tell? You are so stupid." The interrogator is relieved. He has coded chat bots himself. On of his last ditch defenses was
(defun insult-interrogator () (format *standard-io* "~&You are so stupid."))
He denounces B as non-human, getting it wrong for the fourth time this week. The computer sending to teletype A has passed the Turing test :-)
Whoops! I'm getting carried away writing fiction. The point I'm trying to tack on to Turing's original insight (no curriculum, no pass mark) is that the pro-AI camp cannot try to keep the test short. If they limit it to a 5 minute interrogation, the anti-AI camp will claim that it takes six minutes to exhaust the chat bots opening book, and refuse to concede.
More importantly the anti-AI camp can develop the technique of smoking out small-state chat-bots by keeping the interrogation going for half an hour and then circling back to the beginning. Of course the human may have forgotten how the interrogation began. It is in the spirit of the test to say the the computer doesn't have to do better than the human. But the spirit of the Turing test certainly allows the interrogator to try. If the human notices "Didn't you ask that earlier." and if the computer doesn't, or slows down as the interrogation proceeds due to an ever-growing state, the computer quite properly fails the Turing Test. (Hmm, I feel that I'm getting sucked into a very narrow vision of what might be involved in passing the Turing Test.)
If the pro-AI camp want the anti-AI camp to concede, they have to let the anti-AI interrogators keep asking questions until they realise that the extra questions are not helping. The computer is thinking about the questions before answering and can keep it up all day.
I think that you can break a chat-bot out of its opening book with three questions along the following lines.
1)Which is heavier, my big toe or a 747
2)Which is heavier, a symphony or a sonnet
3a)Which question do you think is better for smoking out the computer, the first or the second?
3b)Which of the previous two questions is the more metaphorical?
One can imagine a big engineering effort that lets the computer identify objects and estimate their weight. Big toe 10 grams. 747, err, 100 tons. And one can write code that spots and dodges trick questions involving the weight of immaterial objects. But one needs a big, fat opening book to cope with the great variety of individual questions that the interrogator might ask.
Then comes question three. That links together question one and question two, squaring the size of the opening book. 40 seconds into an all day interrogation and the combinatorial explosion has already gone BOOM!
The combinatorial explosion is on the side of the TT, of course. But storage space is on the side of "design to the test", so if you can make up a nice decisive question, the designer can think of it, too (or read your blog) and add that. The question here is whether Stuart (and Ned Block) are right that such a "giant lookup table" a) makes sense and b) has no intelligence. "The intelligence of a toaster" as Block said.
There is a problem with the Turing test, practically and philosophically, and I would be willing to bet that the first entity to pass the test will not be conscious, or intelligent, or have whatever spark or quality the test is supposed to measure. And I hold this position while fully embracing materialism, and rejecting p-zombies or epiphenomenalism.
The problem is Campbell's law (or Goodhart's law):
This applies to more than social indicators. To illustrate, imagine that you were a school inspector, tasked with assessing the all-round education of a group of 14-year old students. You engage them on the French revolution and they respond with pertinent contrasts between the Montagnards and Girondins. Your quizzes about the properties of prime numbers are answered with impressive speed, and, when asked, they can all play quite passable pieces from "Die Zauberflöte".
You feel tempted to give them the seal of approval... but they you learn that the principal had been expecting your questions (you don't vary them much), and that, in fact, the whole school has spent the last three years doing nothing but studying 18th century France, number theory and Mozart operas - day after day after day. Now you're less impressed. You can still conclude that the students have some technical ability, but you can't assess their all-round level of education.
The Turing test functions in the same way. Imagine no-one had heard of the test, and someone created a putative AI, designing it to, say, track rats efficiently across the city. You sit this anti-rat-AI down and give it a Turing test - and, to your astonishment, it passes. You could now conclude that it was (very likely) a genuinely conscious or intelligent entity.
But this is not the case: nearly everyone's heard of the Turing test. So the first machines to pass will be dedicated systems, specifically designed to get through the test. Their whole setup will be constructed to maximise "passing the test", not to "being intelligent" or whatever we want the test to measure (the fact we have difficulty stating what exactly the test should be measuring shows the difficulty here).
Of course, this is a matter of degree, not of kind: a machine that passed the Turing test would still be rather nifty, and as the test got longer, and more complicated, as the interactions between subject and judge got more intricate, our confidence that we were facing a truly intelligence machine would increase.
But degree can go a long way. Watson won on Jeopardy without exhibiting any of the skills of a truly intelligent being - apart from one: answering Jeopardy questions. With the rise of big data and statistical algorithms, I would certainly rate it as plausible that we could create beings that are nearly perfectly conscious from a (textual) linguistic perspective. These "super-chatterbots" could only be identified as such with long and tedious effort. And yet they would demonstrate none of the other attributes of intelligence: chattering is all they're any good at (if you ask them to do any planning, for instance, they'll come up with designs that sound good but fail: they parrot back other people's plans with minimal modifications). These would be the closest plausible analogues to p-zombies.
The best way to avoid this is to create more varied analogues of the Turing test - and to keep them secret. Just as you keep the training set and the test set distinct in machine learning, you want to confront the putative AIs with quasi-Turing tests that their designers will not have encountered or planed for. Mix up the test conditions, add extra requirements, change what is being measured, do something completely different, be unfair: do things that a genuine intelligence would deal with, but an overtrained narrow statistical machine couldn't.