The Turing test doesn’t matter

by Massimo Pigliucci

You probably heard the news: a supercomputer has become sentient and has passed the Turing test (i.e., has managed to fool a human being into thinking he was talking to another human being [1,2])! Surely the Singularity is around the corner and humanity is either doomed or will soon become god-like.

Except, of course, that little of the above is true, and it matters even less. First, let’s get the facts straight: what actually happened [3] was that a chatterbot (i.e., a computer script), not a computer, has passed the Turing test at a competition organized at the Royal Society in London. Second, there is no reason whatsoever to think that the chatterbot in question, named “Eugene Goostman” and designed by Vladimir Veselov, is sentient, or even particularly intelligent. It’s little more than a (clever) parlor trick. Third, this was actually the second time that a chatterbot passed the Turing test, the other one was Cleverbot, back in 2011 [4]. Fourth, Eugene only squeaked by, technically convincing “at least 30% of the judges” (a pretty low bar) for a mere five minutes. Fifth, Veseloy cheated somewhat, by giving Eugene the “personality” of a 13-yr old Ukrainian boy, which thereby somewhat insulated the chatterbot from potential problems caused by its poor English or its inept handling of some questions. As you can see, the whole thing was definitely hyped in the press.

Competitions to pass the Turing test have become fashionable entertainment for the AI crowd, and Brian Christian — who participated in one such competition as a human decoy — wrote a fascinating book about it [5], which provides interesting insights into why and how people do these things. But the very idea of the Turing test is becoming more and more obviously irrelevant, ironically in part precisely because of the “successes” of computer scripts like Cleverbot and Eugene.

Turing proposed his famous test back in 1951, calling it “the imitation game.” The idea stemmed out of his famous work on what is now known as the Church-Turing hypothesis [6], the idea that “computers” (very broadly defined) can carry out any task that can be encoded by an algorithm. Turing was interested in the question of whether machines can think, and he was likely influenced by the then cutting edge research approach in psychology, behaviorism [7], whose rejection of the idea of internal mental states as either fictional or not accessible scientifically led psychologists for a while to study human behavior from a strictly externalist standpoint. Since the question of machine thought seemed to be even more daunting than the issue of how to study human thought, Turing’s choice made perfect sense at the time. This, of course, was well before many of the modern developments in computer science, philosophy of mind, neurobiology and cognitive science.

It didn’t take long to realize that it was not that difficult to write short computer scripts that were remarkably successful at fooling human beings into thinking they were dealings with humans rather than computers, at least in specific domains of application. Perhaps the most famous one was Eliza, which simulates a Rogerian psychotherapist [8], and which was invented by Joseph Weizenbaum in the mid ‘60s. Of course, Eliza is far more primitive than Cleverbot or Eugene, and its domain specificity means that it technically wouldn’t pass the Turing test. Still, try playing with it for a while (or, better yet, get a friend who doesn’t know about it to play with it) and you can’t avoid being spooked.

That’s in large part because human beings have a strong instinctual tendency to project agency whenever they see patterns, something that probably also explains why a belief in supernatural entities is so widespread in our species. But precisely because we know of this agency-projection bias, we should be even more careful before accepting any purely behavioristic “test” for the detection of such agency, especially in novel situations where we do not have a proper basis for comparison. After all, the Turing test is trying to solve the problem of other minds [9] (as in, how do we know that other people think like we do?) in the specific case of computers. The difference is that a reasonable argument for concluding that people that look like me and behave like me (and who are internally constituted in the same way as I am, when we are able to check) indeed also think like me is precisely that they look, behave and are internally constituted in the same fashion as I am. In the case of computers, the first and third criteria fail, so we are left with the behavioristic approach of the Turing test, with all the pitfalls of behaviorism, augmented by its application to non biological devices.

But there are deeper reasons why we should abandon the Turing test and find some other way to determine whether an AI is, well, that’s the problem, is what, exactly? There are several attributes that get thrown into the mix whenever this topic comes up, attributes that are not necessarily functionally linked to each other, and that are certainly not synonyms, even though too often they get casually used in just that manner.

Here are a number of things we should test for in order to answer Turing’s original question: can machines think? Each entry is accompanied by a standard dictionary definition, just to take a first stab at clarifying the issue:

Intelligence: The ability to acquire and apply knowledge and skills.

Computing power: The power to calculate.

Self-awareness: The conscious knowledge of one’s own character, feelings, motives, and desires.

Sentience: The ability to perceive or feel things.

Memory: The faculty of storing and retrieving information.

It should be obvious that human beings are characterized by all of the above: we have memory, are sentient, self-aware, can compute, and are intelligent (well, some of us, at any rate). But it’s also obvious that these are distinct, if in some ways related, attributes of the human mind. Some are a subset of others: there is no way for someone to be self-aware and yet not sentient; yet plenty of animals are presumably the latter but likely not the former (it’s hard to tell, really). It should further be clear that some of these attributes have little to do with some of the others: one can imagine more and more powerful computing machines which nonetheless are neither intelligent nor self-aware (my iPhone, for instance). One can also agree that memory is necessary for intelligence and self-awareness, but at the same time realize that human memory is nothing like computer memory: our brains don’t work like hard drives where information is stored and reliably retrieved from. In fact, memories are really best thought of as continuous re-interpretations of past events, whose verisimilitude varies according to a number of factors, not the least of which is emotional affect.

So, when we talk about “AI,” do we mean intelligence (as the “I” deceptively seems to stand for), computation, self-awareness, all of the above? Without first agreeing at the least on what it is we are trying to do we cannot possibly even conceive of a test to see whether we’ve gotten there.

Now, which of the above — if any — does the Turing test in particular actually test for? I would argue, none. Eugene passed the test, but it certainly lacks both sentience and, a fortiori, self-awareness. Right there it seems to me therefore that its much trumpeted achievement has precious little interesting to say to anyone who is concerned with consciousness, philosophy of mind, and the like.

If I understand correctly what a chatterbox is, Eugene doesn’t even have memory per se (though it often does rely on a database of keywords), not in the sense in which a computer has memory, and certainly not in the way a human does. Does it have computing power? Well, yes, sort of, depending on the computing power of its host machine, but not in any interesting sense that should get anyone outside of the AI community excited.

Finally, is it intelligent? Again, no. Vladimir Veselov, the human who designed Eugene, is intelligent (and sentient, self-aware, capable of computation and endowed with memory), while Eugene itself is just a (very) clever trick, nothing more.

And that’s why we need to retire the Turing test once and for all. It doesn’t tell us anything we actually want to know about machine thinking. This isn’t Turing’s fault, of course. At the time, it seemed like a good idea. But so were epicycles in the time of Ptolemy, or luminiferous aether before the Michelson–Morley experiment.

What are we going to replace it with? I’m not sure. Aside from the necessary clarification of what it is that we are aiming for (intelligence? Self-awareness? Computational power? All of the above?),  we are left with an extreme version of the above mentioned problem of other minds. And that problem is already very difficult when it comes to the prima facie easier case of non-human animals. For instance, it’s reasonable to infer that closely related primates have some degree of self-awareness (let’s focus on that aspect, for the sake of discussion), but how much? Unlike most human beings, they can’t communicate to us about their perceptions of their own character, feelings, motives, and desires. What about other animals with complex brains that are more distant from us phylogenetically, and hence more structurally different, like octopi? Again, possibly, to a degree. But I’d wager that ants, for instance, have no self-awareness, and neither does the majority of other invertebrate species, and possibly even a good number of vertebrates (fish? Reptiles?).

When we talk about entirely artificial entities, such as computers (or computer programs), much of the commonsense information on the basis of which we can reasonably infer other minds — biological kinship, known functional complexity of specific areas of the brain, etc. — obviously doesn’t apply. This is a serious problem, and it requires an approach a lot more sophisticated than the Turing test. Indeed, it is dumbfounding how anyone can still think that the Turing test is even remotely informative on the matter. We are in need first of all of clarifying quite a bit of conceptual confusion, and then of some really smart (in the all-of-the-above sense) human being coming up with a new proposal. Anyone wish to give it a shot?

P.S.: the Colbert Report just put out a video that includes my latest and most cutting edge thoughts on black lesbian robotic invasions. Thought you might be interested…

_____

Massimo Pigliucci is a biologist and philosopher at the City University of New York. His main interests are in the philosophy of science and pseudoscience. He is the editor-in-chief of Scientia Salon, and his latest book (co-edited with Maarten Boudry) is Philosophy of Pseudoscience: Reconsidering the Demarcation Problem (Chicago Press).

[1] For an in-depth discussion see: The Turing Test, by Graham Oppy and David Dowe, Stanford Encyclopedia of Philosophy.

[2] Incidentally, and for the sake of giving credit where credit is due, perhaps this should be called the Descartes test. In The Meditations, Descartes wrote: “If there were machines which bore a resemblance to our bodies and imitated our actions as closely as possible for all practical purposes, we should still have two very certain means of recognizing that they were not real men. The first is that they could never use words, or put together signs, as we do in order to declare our thoughts to others. For we can certainly conceive of a machine so constructed that it utters words, and even utters words that correspond to bodily actions causing a change in its organs. … But it is not conceivable that such a machine should produce different arrangements of words so as to give an appropriately meaningful answer to whatever is said in its presence, as the dullest of men can do. Secondly, even though some machines might do some things as well as we do them, or perhaps even better, they would inevitably fail in others, which would reveal that they are acting not from understanding, but only from the disposition of their organs. For whereas reason is a universal instrument, which can be used in all kinds of situations, these organs need some particular action; hence it is for all practical purposes impossible for a machine to have enough different organs to make it act in all the contingencies of life in the way in which our reason makes us act.” Of course, Descartes was a skeptic about machine intelligence, but the basic idea is the same.

[3] A Chatbot Has ‘Passed’ The Turing Test For The First Time, by Robert T. Gonzalez and George Dvorsky, io9, 8 June 2014.

[4] Why The Turing Test Is Bullshit, by George Dvorsky, io9, 9 June 2014.

[5] The Most Human Human: What Talking with Computers Teaches Us About What It Means to Be Alive, by Brian Christian, Doubleday, 2011.

[6] The Church-Turing Thesis, by B. Jack Copeland, Stanford Encyclopedia of Philosophy.

[7] Behaviorism, by George Graham, Stanford Encyclopedia of Philosophy.

[8] If you’d like to play with Eliza, here is a java version of the script.

[9] Other Minds, by Alec Hyslop, Stanford Encyclopedia of Philosophy.