[UPDATE: Other computer scientists are saying that the computer actually failed the test, and badly.]
People who have interacted with Siri, the helpful guide on the iPhone, are usually impressed with her ability to carry on what seems like a normal conversation. But it is not hard to discover that you are talking to a computer. How good would ‘she’ have to be to completely fool you?
The Turing test was first proposed by computer science pioneer Alan Turing in 1950 to address this question and there are many variations of it. It provides a benchmark in the development of artificial intelligence. It essentially consists of whether someone engaging in natural language conversations with an unseen entity can distinguish whether that entity is a human being or a machine. To pass the test, more than 30% of human interrogators should be unable to tell the difference after a series of five-minute conversations.
Via Machines Like Us, I learned that scientists at the University of reading report that for the first time, a machine has passed one version of the test by convincing 33% of the judges that it was a 13-year old Ukrainian boy named Eugene Gootsman.
The test was passed on the 60th anniversary of the death of Turing who committed suicide as a result of the awful treatment of him by the British government and legal system because he was homosexual.
So what does this mean for AI, that fascinating field whose advocates have claimed, for the past half-century, that human-like intelligence was just 10 years away from being achieved? I don’t know. It seems to me that the test results will depend on the quality of the judges and their ability to conceive of conversational initiatives that will elicit revealing responses. Ideally one would like to have some kind of standardized judges in order to monitor quality but I don’t know how one does that.
The 30% cut-off is also just a starting point. Passing it consistently with good judges will likely result in raising the bar from 30% to higher and higher levels until it gets close to 100%.
It means nothing. Were talking hand-picked judges primed to expect responses from someone who doesn’t use English as a first language, a low threshold, and possibly most important of all, Kevin Warwick. Who is just a bit cranky. These are bad claims reported by the terrifically credulous pool of journalists we all know and love.
It’s a chatbot.
The judges were minor celebrities and the computer’s persona was a young teenager of foreign extraction, giving it an excuse for making odd statements or being hard to relate to. Color me unimpressed.
It doesn’t seem to be a universal conclusion.
According to Professor Murray Shanahan of the Department of Computing at Imperial College London,
<
>
He went on to say
and
The professor also stated it was simply a chatbot, not a computer, and that “the rules were bent from the beginning, since the character was designed as 13-year-old Ukrainian, sending the message of limited communication skills”.
It will be interesting to see whether the professor’s opinions are supported by others, or whether the original story will stand.
character was designed as 13-year-old Ukrainian
I love this; it’s a wonderful demonstration of how language skills and intelligence are often connected inaccurately. Put differently, if I was a 13 year-old Ukrainian I’d be screaming “FUCK YOU!” at the researchers for assuming that 13 year-old Ukrainians have inferior communications skills. Or would that be, “Excuse me, professor, but go fuck yourself?”
Yea, that’s a pretty big stretch to say that passes the Turing Test.
I didn’t realize that Turing explained the test with the first version being a man and woman. The idea being a judge has a conversation with a man and a woman. Can the judge tell which one is the man, even while the woman is pretending to be a man? I wonder if people could effectively tell the difference. That sounds like a just a fun test in general (and of course, flipping the genders because maybe one gender is better at imitating the other).
http://psych.utoronto.ca/users/reingold/courses/ai/turing.html